[00:17:04] [bz] (8NEW - created by: 2Addshore, priority: 4High - 6enhancement) [Bug 48894] Include pagecounts dumps in datasets - https://bugzilla.wikimedia.org/show_bug.cgi?id=48894 [03:24:28] !reboot [03:24:41] !reboot is https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation/wm-bot [03:24:41] Key was added [03:57:45] Ryan_Lane: hi [04:03:57] legoktm: hi, are you there? [04:48:27] Hi all [05:49:12] Can't anything be done to stop these spikes? [05:50:03] NFS? [05:50:13] they're on it. Coren had no power for 4 days, which kinda put a stop on things [05:51:10] YuviPanda, actually Coren said its people abusing crontabs. [05:51:28] And not jsubbing the scripts. [05:51:32] the last explanation I saw was the controller (hardware) itself being messed up [05:51:39] and yes, it got less bad after the crons were kicked out [06:28:53] YuviPanda, Cyberpower678: yep controller issues [06:30:30] Ryan_Lane: did dell respond? [06:30:37] no clue [07:39:14] !ping [07:39:14] ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" [07:39:18] wtf [07:40:34] Isn't that Helpmebot's things? [07:41:31] !ping [07:41:31] ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" [07:41:44] !ping [07:41:44] ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" [07:41:59] !ping is pong [07:41:59] ?? [07:41:59] petan, ^ [07:41:59] omg [07:41:59] wait [07:42:00] This key already exist - remove it, if you want to change it [07:43:04] XDD [07:43:11] petan, nice ping [07:44:54] Hi petan [07:45:10] hey [07:45:30] I just want to say hello to you [07:45:42] have a nice time :) [07:45:57] thanks [07:46:53] wm-bot is the new Helpmebot [07:47:40] !trout is goes fishing, catches a trout, and smack $1 with it, over and over. [07:47:40] This key already exist - remove it, if you want to change it [07:47:46] !trout petan [07:47:46] I am taking a 10-pound rainbow trout out of a river, and smacks petan with it over and over. [07:48:04] !trout del [07:48:04] Successfully removed trout [07:48:18] !trout is goes fishing, catches a trout, and smack $1 with it, over and over. [07:48:19] Key was added [07:48:22] #wikimedia-labs-offtopic for this [07:48:28] !trout petan [07:48:28] goes fishing, catches a trout, and smack petan with it, over and over. [07:48:38] petan, I'm done. [08:57:58] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 51700] https://login.wikimedia.beta.wmflabs.org/ trapped in an infinite self-redirect - https://bugzilla.wikimedia.org/show_bug.cgi?id=51700 [09:03:38] !log deployment-prep restarted varnish text cache [09:03:41] Logged the message, Master [09:28:29] [bz] (8NEW - created by: 2Addshore, priority: 4High - 6enhancement) [Bug 48894] Include pagecounts dumps in datasets - https://bugzilla.wikimedia.org/show_bug.cgi?id=48894 [10:52:37] !ping [10:52:37] ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" [10:53:00] Is NFS having issues again...? [10:54:37] toollabs diead again [10:54:49] :( [10:55:37] works to me? [10:55:41] YuviPanda what doesn't work? [10:55:58] it's back [10:56:02] legoktm: geeek [10:56:02] :\ [10:56:20] petan: so i have to re-run my jobs? [10:57:30] petan: .. [10:57:33] !reboot [10:57:34] https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots/Documentation/wm-bot [10:57:41] ^^^ that key... [10:57:42] Amir1: depends [10:57:52] Amir1: check if they run or not [11:10:17] petan: I checked, they're working [11:10:33] thank you for guidance anyway [12:12:42] [bz] (8ASSIGNED - created by: 2Niklas Laxström, priority: 4High - 6major) [Bug 48203] Purging does not work on deployment-prep / beta labs - https://bugzilla.wikimedia.org/show_bug.cgi?id=48203 [12:16:00] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 51874] vhtcpd needs to support purge request send over unicast - https://bugzilla.wikimedia.org/show_bug.cgi?id=51874 [13:16:12] * YuviPanda waves [13:16:12] * YuviPanda just crossed 26 hours awake, not sure how useful he'll be today [13:16:12] wow! [13:16:51] 26 hours awake is far too many [13:17:04] doing a sleep cycle reset :) 5 more hours! [13:17:20] kma500: my sleep cycle had 'drifted' to a point where I was slepeing at 7am everyday. drastic action was necessary :) [13:18:24] YuviPanda: I hope the drastic action works :) [13:18:33] kma500: :) [13:19:03] it's been productive, however. 27 commits in that period of time [13:20:30] I don't know how people manage to be productive on no sleep. Congrats! It's a talent. [13:21:15] :) let's hope I can pop it to 30 before I hit the bed. [13:21:23] sleep also feels very much sweeter after these thigns [13:22:21] I took a shower! That is because I had hot water. Today is a good day. [13:22:29] I imagine. [13:22:47] You have power, coren? [13:22:53] Coren: I had really hot shower for one full hour! [13:22:59] keeps the sleep away :) [13:23:02] kma500: Yes, as of last evening. [13:23:07] excellent! [13:23:16] Coren: also welcome back to civilization! Happy to have you back whole and unharmed :) [13:23:28] I also need to buy myself a bigger generator. [13:24:21] funny how important this power thing is :) [13:24:58] It's actually a little scary when we notice how horribly dependent we are on it. [13:26:01] Coren where you live? [13:26:07] Coren US? Canada? [13:26:19] I am wondering which country has problems with power in 2013 :D [13:26:30] Canada [13:26:42] heh [13:26:50] but I guess it's not a problem in big cities only [13:27:06] Canada, in a suburb near Montreal; it was a freak storm. I get reliable power most of the time, but uprooted trees falling on power lines all over the place does not care. :-) [13:27:18] Storms know no boundaries [13:28:01] http://www.upi.com/Top_News/World-News/2013/07/20/At-least-one-killed-as-thunderstorms-sock-Ontario-and-Quebec/UPI-47391374352019/?spt=hs&or=tn [13:28:41] We gots tornadoes. I think the last time we had tornadoes was in the 70s. :-) [13:29:11] * T13|stillSleepy needs a shower too... [13:29:33] * T13|stillSleepy 's cats are avoiding him... :0 [13:29:40] Fun fact: a heat wave being broken by the Jet Stream shifting south all at once and bringing in a huge cold air mass slamming in the local air = bad news. [13:31:10] heh [13:31:20] Coren: has the freak weather stuff passed? [13:31:53] Freak weather indeed! [13:32:02] YuviPanda: Yeah, we've been back to normal weather for two days; the weather was crazy for just the one day but it took several more to fix the damage. There are still people without power. [13:32:38] ow. [13:34:08] Izawayz: ping [13:34:14] rschen7754|away: ping [13:34:40] Coren: how long do we need to wait for last 2 users who were informed 2 months ago that bots project will be closing before we finally kill their bots and shut it down? XD [13:35:17] petan: Have you announced a date? You want to do that at least two weeks in advance, on labs-l. [13:35:19] like 1 instance - 4cpu / 8gb of ram 80gb of storage is being blocked from deletion because of 1 tiny bot using like 20m of ram and no cpu [13:36:01] The other question is whether you know they've actually received the notification. :-) [13:36:12] Coren: I don't even remember but I know there were lot of announcements in lab-l and huge banner "DO NOT USE THIS INSTANCE IT WILL BE DELETED IN FEW DAYS" in motd [13:36:38] petan: That works. If you want to be generous, give them at least until the end of the week. [13:36:57] I will just wait then but it was Ryan who was like "omg we are out of storage, plz delete some instances" [13:37:00] BRB. Breakfast. [13:39:57] Hey, someone said a while ago that they were looking into getting me a shiny labs-mysql that won't hurt everyone. Did that ever happen? [13:40:22] I just realized I've kind of forgotten about that investigation I was doing a couple weeks ago that made nfs so sad [13:45:38] hi sumanah [13:45:45] hi kma500! how's it going? [13:45:58] good! how are you? [13:46:11] I'm all right :) [13:47:24] so http://etherpad.wmflabs.org/pad/p/Tool_Labs_Sprint_July_23 [13:47:29] I'm going to be in the office btw 10-10:30, but offline til then. Anything doc-related we need to figure out early? [13:47:35] yes. [13:48:26] kma500: I figure anyone who wants to can just dive in and start typing [13:48:32] e.g. what I just did on line 56 [13:48:52] kma500: you saw what Coren just said about his power problem :( [13:48:57] Just saw that. [13:49:08] Silke_WMDE_: hi, do you want to concentrate on any part of the docs in particular? [13:49:09] Coren has power again! [13:49:49] As long as you're fine with a very free-flowing process where people can just type on the Etherpad to fill in individual parts of the outline, I think we're set :) [13:51:22] Okay. Sounds good! [13:51:29] great! [13:51:37] it's not even 7am where you are - amazing that you are up :) [13:52:14] sumanah: It's not my fault! My husband set the alarm for five... [13:52:30] But I did want to check in since I'll be getting in on the later side. [13:52:36] Didn't want to hold anything up. [13:53:20] sumanah: I'm resetting my sleep cycle today (uptime: 27h right now!) so won't be there today. Will look at it and fill in things tomorrow when I wake uP :) [13:54:40] !ping [13:54:40] ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" ¤*POOF*¤ "Wadda need?" [13:54:43] am connected. nice [13:54:47] wm-bot: ping is pong [13:54:47] Hi YuviPanda, there is some error, I am a stupid bot and I am not intelligent enough to hold a conversation with you :-) [13:54:55] !ping del [13:54:55] !ping del [13:54:55] Successfully removed ping [13:54:57] Unable to find the specified key in db [13:55:06] !ping is pong [13:55:06] :) [13:55:06] ty petan [13:55:08] Key was added [13:55:50] see you later! [13:57:31] YuviPanda: :) [13:57:37] kma500: makes sense, thanks [14:00:07] sumanah: give me a few minutes, meeting [14:08:36] sure, Silke :) [14:10:38] manybubbles_: It's pending on some love from one of our DB people. Since it's becoming pressing for you, do you want me to poke Ken to see if he can spare one of them for a half-hour? [14:11:34] Coren: that't be wonderful! Ken has asked me to poke him if I need something so I can do it too. My problem with this is that I just keep forgetting about it because I had to put it on hold. [14:12:45] manybubbles_: Part of the issue is that I think Asher is still OoO. I'll poke Ken, but it won't hurt if you also tell him that's a blocker for you. [14:17:14] Coren: emailed. it isn't a blocker - just kind of annoying not to have [14:17:38] Well, you can't really do what it was you needed to do until that's available. In my book, that's a blocker. :-) [14:18:20] (Also sent email) [14:21:10] Coren: how it looks with teh dedicated sql server you worked on week ago [14:21:39] petan: Needs love from one of our DBAs. See above exchange with manybubbles [14:21:40] :-) [14:21:51] aha [14:22:01] oooh - if other people want it then more power! [14:25:50] Coren: did you notice mails are broken on tools? [14:25:57] Coren: I am not receiving any e-mail :( [14:26:09] petan: Have you checked the logs? [14:26:09] last mail is from 20+ days ago [14:26:13] logs where [14:26:22] mail system is very undocumented [14:26:26] !toolsadmin [14:26:26] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Documentation/Admin [14:31:29] hi Coren [14:31:55] a new column has been added to wikidata's wb_terms table [14:31:56] aude: Hello. [14:32:07] is there anything specific that needs to happen to have it appear in toollabs? [14:32:12] like rebuild a view? [14:32:13] petan: It's a normal exim4 on, predictably enough, tools-mail. :-) [14:32:31] * aude would be reassured to see it there :) [14:33:17] Hi! [14:33:31] Coren: it doesn't let me ssh :( [14:33:37] aude: What's the new column? [14:33:44] Coren: term_weight [14:33:56] it's there since thursday [14:34:07] aude: I see it replicated so yeah, only the vew being rebuilt is needed. I can do that if you give me a minute. [14:34:13] ok, thanks [14:34:13] petan: Hang on, will see why. [14:34:27] we'll be enabling wikidata to use the column in a few hours [14:36:12] aude: Should be there now. [14:36:25] awesome, thanks Coren ! [14:37:04] btw, my bot is running on toollabs now and it's doing well :) [14:37:14] great :) [14:37:27] petan: Ah, found why. When I rebooted the cluster after the NFS explosion, tools-mail was forgotten so it doesn't seem home. [14:37:28] * aude making regular backups in case stuff gets corrupt again [14:37:36] k [14:37:42] petan: Prepare to be spammed once the mail queues flush. :-) [14:37:49] cool [14:37:58] Coren: i am making use of postgres and postgis, though which i am doing via external (personally) hosted api [14:38:03] Coren: I presume there's a cluster reboot checklist to add that to now? :) [14:38:11] what are the plans to support postgres and postgis in toollabs? (any plans) [14:38:17] aude: cool [14:38:30] anything we can help with to make it happen? [14:39:00] sumanah: Not as such, beyond "Don't reboot the cluster, but if you have to reboot every host" which isn't proof against "Oops, forgot that one." :-) [14:39:07] aude: you helping document stuff? http://etherpad.wmflabs.org/pad/p/Tool_Labs_Sprint_July_23 [14:39:14] sumanah: not today [14:39:24] Coren: that seems inelegant to me .... but I defer to your judgment [14:39:57] sumanah: I don't think there's a whole section I can work on, it's rather single questions I can answer with the help of docs and (many) others I can't. [14:39:58] * aude thinks the documentation is decent or at least i was able to figure most out [14:40:01] sumanah: They are, thankfully, not order dependent. :-) [14:40:03] Silke_WMDE_: go ahead then :) [14:40:32] (I.e.: There is no black magic in having the cluster come up "right") [14:40:34] aude: yeah, fortunately this is an improvement sprint and not a write-from-scratch thing! [14:40:40] sumanah: yeah! [14:40:50] aude: you could write a case study - those are helpful [14:40:54] certainly the toolserver wiki has a lot of good info for newbies [14:40:58] @seen orsa [14:40:58] Technical_13: Last time I saw orsa they were quitting the network with reason: Remote host closed the connection N/A at 7/22/2013 8:44:59 PM (17h55m59s ago) [14:40:59] like how to do a query, and that's important [14:41:07] * aude needed that at one point [14:41:17] @notify orsa [14:41:17] I'll let you know when I see orsa around here [14:41:35] petan: tools-mail should let you in now, btw. [14:41:59] petan: Incidentally, since you're in the right group, no host should ever not let you in. If that happens, it's unfailingly a bug. [14:42:11] aude: what are you using postgist/postgres for? [14:42:15] * YuviPanda is interested too! [14:42:18] sumanah: something that i would like is https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Rules be filled in [14:42:20] aude: then that would be a great thing to link to in the FAQ [14:42:21] PostGIS ftw! :) [14:42:28] aude: have you put that in the Etherpad? [14:42:38] kma500: ^ aude is a Tool Labs user and has some requests :) [14:42:44] aude: I'm waiting on Luis / legal on this; we have a draft ready but it needs final tweaks and their blessing. [14:42:55] aude: It may well be stalled on the privacy policy stuff though. [14:42:57] YuviPanda: i am adding geocoordinates to wikidata and among other checks, i check it is geocoded in the correct province or state (that i am working on) [14:43:11] aude: where is your postgis instance hosted? your own vps? [14:43:16] and make a bounding box of the state, so i can select a batch of geocoordinates [14:43:19] YuviPanda: my own [14:43:21] right. [14:43:22] * Coren eeps as the mail backlog suddenly gets delivered. [14:43:26] and i have a secret api for it :) [14:43:33] Coren: ok [14:43:42] even if we have postgres / postgis with out an osm db, that's useful [14:43:55] like i can have my own, maybe with all the geo_tags geocoded [14:44:01] from wikipedia and wikidata [14:44:03] sumanah: Actually, if you might be able to help there by poking Luis with a gentle "When are we getting Labs TOS?" :-) [14:44:54] sorry to spam, but i was also wondering if it would be okay, per rules to host a such an "enhanced" version or copy of the geocoordinates [14:44:59] as a tool [14:45:01] aude: I can help setup puppet, etc. I'd love to have it as part of toollabs, provided Coren agrees :) [14:45:11] YuviPanda: cool :) [14:45:12] aude: I don't see why not. Why do you think there might be an issue? [14:45:29] Coren: i don't see an issue but not 100% sure [14:45:38] Coren: would having tools-postgres be an issue? :) [14:45:41] the toolserver had issues with, at least hosting any text content [14:45:53] YuviPanda: That's at the edge of the scope; Tool Labs users (as opposed to general Labs users) normally should not need to fiddle with puppet -- that's part of the point of tools. :-) [14:46:13] Coren: no no, *I* was offerint to setup tools-postgres via puppet so people can u se postgres [14:46:13] postgres should be available to all, same way as mysql is [14:46:17] indeed. [14:46:51] it's icky (especially since geo_tags is not in the dumps yet), to try to make an external copy of stuff geocoded [14:46:58] can be done, just a lot harder [14:47:05] YuviPanda: It would; having a DB in an instance is teh evils. I do know we need postgres for a few things, though, and it would be nice to have one for testing mediawiki against as well, so I'd rather add a postgress setup to the new physical DBs I'm working on this week. [14:47:25] Coren: woo! Is that before or after the NFS? and the uwsgi? [14:47:32] aude: file a bug for postgres/postgis? [14:47:40] YuviPanda: do you want to? [14:47:42] and yes, having it on a physical thing would be nice [14:47:49] aude: No issue with serving text on Labs. The toolserver limitation was partly technical and partly legal, but neither apply in Labs. [14:47:53] aude: you've the more immediate use case, so you should :) [14:47:57] Coren: that's what i thought [14:48:16] YuviPanda: it depends... if the geo_tags dump comes first or i get impatient [14:48:35] i believe that geo tags was added to the dumps but has not run everywhere yet [14:48:46] well, it's a bug report :) [14:49:11] k [14:49:23] ty aude [14:50:53] [bz] (8NEW - created by: 2Aude, priority: 4Unprioritized - 6normal) [Bug 51885] provide postgresql and postgis on toollabs - https://bugzilla.wikimedia.org/show_bug.cgi?id=51885 [14:50:57] there ^ [14:51:09] aude: ty! [14:51:20] Coren: any update on talking to Dell? [14:53:50] hmmmm, looks like my bot needs more work to do :) [14:54:02] YuviPanda: Nope. I was in the dark, remember? It's on my TODO for the day. [14:54:13] or it died [14:54:22] Coren: :) [14:54:28] Coren: apologies if I'm naggy. [14:54:33] * Coren has had no progress whatsoever on anything since Friday afternoon. [14:55:32] no, my bot needs more work [15:08:58] sumanah: The pad is really well prepared with all those links! Thanks. Also to Kirsten. Btw is she here? What's the nick? [15:09:42] Silke_WMDE_: kma500. [15:09:55] ah thx scfc_de! [15:14:38] scfc_de: You answered about database connections here: https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools/Help#GUI_tool_for_databasework What would be an example for such a local GUI tool to connect to tools-login? I've never used such a thing. [15:15:55] Silke_WMDE_: Me neither, I only used mysql (CLI) for testing. Any Windows users around? [15:16:29] Silke_WMDE_: http://dev.mysql.com/downloads/tools/workbench/ [15:17:02] Coren, Ok, thanks, I'll add that as an example. [15:24:39] Damianz: is cluebot running from tools now? [15:24:50] petan: 3 is [15:24:58] 3? [15:25:04] 3 processes? [15:25:07] or what [15:25:14] ClueBot III [15:25:36] hmm [15:26:44] !damianz [15:26:44] some weirdo around here [15:27:08] Thanks for the reminder wm-bot [15:27:08] Hey Technical_13, you are welcome! [15:27:42] spambot cyberbot Task / Running 2013-07-20 21:17:59 CPU: 58h56m VMEM: 325M/4.8G [15:27:51] vmem 325m/4.8g o.o [15:28:05] sounds like python [15:28:36] if all these python bots were written in c whole tools project would fit into one instance with 1 cpu and 256mb of ram :P [15:28:51] *properly written [15:29:53] Coren: i feel I have to catch up a bit... Are the two projects still called "tools" and "bots"? [15:30:09] tools and toolsbeta [15:30:17] bots is closed kind of [15:30:19] Silke_WMDE_: "toolsbeta". Bots is obsolete and on its last breath. [15:30:27] \o/ [15:30:34] given the speed of migration the last breath is going to take few years [15:30:44] :) [15:31:06] welcome back Silke_WMDE_ [15:31:21] thx JohannesK_WMDE! [15:33:43] Yesterday, I noticed that my mediawiki install on labs changed ( http://ase.wikipedia.wmflabs.org ). The /srv directory can't be found and it breaks the wiki. So far, I have been unable to fix it. Can anyone help or offer a suggestion? [15:41:16] @seenrx zhuy* [15:41:16] petan: Last time I saw zhufengwill they were quitting the network with reason: no reason was given at 10/15/2012 7:55:16 AM (281d7h46m0s ago) (multiple results were found: rick_zhu, Yiranzhu, xiaozhu, zhufeng, johnzhuyiyi and 16 more results) [15:41:24] @seenrx zhuyi* [15:41:24] petan: Last time I saw johnzhuyiyi they were quitting the network with reason: no reason was given at 11/7/2012 7:14:35 AM (258d8h26m49s ago) (multiple results were found: zhuyifei1999, zhuyifei1999_, zhuyifei1999_zzz, zhuyifei1999__) [15:41:35] @seenrx zhuyife.* [15:41:35] petan: Last time I saw zhuyifei1999 they were quitting the network with reason: Ping timeout: 250 seconds at 7/20/2013 9:20:29 AM (3d6h21m6s ago) (multiple results were found: zhuyifei1999_, zhuyifei1999_zzz, zhuyifei1999__) [15:49:08] petan: Who are the two last users on Bots? [15:49:21] scfc_de Izawayz and rschen7754|away [15:49:27] not only them [15:49:34] but these 2 are holding some big resources [15:49:55] also Beetstra, Damian z and me (+ addshore) [15:50:04] not sure if addshore did move all stuff or not yet [15:50:28] * sumanah comes back from meeting, reads backscroll [15:50:52] Coren: if you fwd me whatever the last mail you sent Luis was, I can sort of piggyback on that [15:52:00] Silke_WMDE_: https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta and https://wikitech.wikimedia.org/wiki/Nova_Resource:Bots - I added some information to those pages to help people understand their roles :) [15:52:14] slevinski: did you get the help you needed? [15:52:30] sumanah: My last email on the subject was basically a one-liner saying "Hey, what's up with that." :-) [15:52:30] no, still having problems. [15:52:45] sumanah: thx [15:53:01] sumanah: Sent. [15:53:27] slevinski: What project is this instance part of? [15:53:28] YuviPanda: why there is no toolsbeta-redis [15:53:39] signwriting project [15:53:44] you people are skipping 1 important step [15:53:55] just deploying stuff on production with no testing is evil [15:54:21] petan: phrasing like "you people" can sometimes rub people the wrong way, just an FYI :) [15:54:34] petan: technically, that'd be testing *changes* in infrastructure. New stuff can't break production when it lives on a different instance. :-) [15:54:58] petan: But yeah, it could have been tested there first. [15:55:03] ok but now when we need to test a change to redis, we need to create whole new instance first instead of jus testing a change [15:55:22] slevinski: so does the /srv directory exist with wrong permissions or is it just gone? <----- uninformed question [15:55:25] also, you and your apache 2.4 rollback... :P that could be tested as well on toolsbeta [15:55:31] petan: Yes, but since we have reddis in puppet creating a toolsbeta reddis is as simple as just adding it. :-0 [15:55:41] petan: toolsbeta was not usable when it was deployed. [15:55:43] who knows... :P [15:55:47] why not? [15:55:53] it was usable for weeks [15:55:55] petan: Because it didn't *exist* [15:55:59] o.O [15:56:10] the /svr directory exists. Permissions look fine [15:56:13] toolsbeta is there for many weeks, that 2.4 rollback was few days ago or something [15:56:22] weeks [15:56:26] dunno [15:56:34] but I am pretty sure the time when it happened it did exist [15:56:38] petan: That's because it lingered for a long time after not having been worked on. It was done long before toolsbeta got there. [15:56:45] It looks like the /var/www directory is no longer the default web directory. [15:57:12] slevinski: Have you checked for recent puppet changes? [15:57:15] I don't understand how did you update it then, but toolsbeta is still 2.2 while production is 2.4 [15:57:17] slevinski: As the instance was created over a year ago, I'm not sure how it was set up and what changes in the mean time have occured. I would suggest asking Ryan_Lane or andrewbogott. [15:57:42] meh no, prod is 2.2. [15:57:43] hmmm [15:58:03] slevinski I would suggest asking bugzilla [15:58:30] !rb [15:58:30] broken? report a bug: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs [15:59:00] I assumed it was a puppet change. Not sure how to check for recent changes. [15:59:15] recent changes are in gerrit [15:59:16] slevinski: http://hexm.de/mw-search might help you [15:59:44] slevinski: you can also search in Gerrit, e.g., https://gerrit.wikimedia.org/r/#/q/status:open+project:%255Eoperations.*,n,z [15:59:54] on how to search: https://gerrit.wikimedia.org/r/Documentation/user-search.html [16:00:04] petan: sorry, I only heard of the need to move yesterday. I'm willing to make the move now, but I would appreciate a walkthough for getting on to tool labs, as i'm still not that savvy with all this [16:00:22] Izhidez no problem [16:00:33] Izhidez don't tell me you didn't notice any of these huge motd banners [16:00:43] like "DON'T USE THIS INSTANCE" [16:00:55] "THIS INSTANCE IS GOING TO BE DELETED, GET OUT OF HERE" [16:01:09] Izhidez: more seriously, how did you hear of the need to move? [16:01:27] Izhidez: and if you tell us where you get your Wikimedia tech-specific information, we can inform you more usefully in the future [16:01:48] slevinski: When did it work for you the last time? [16:01:50] I saw the "This project will be closing" a long time ago, but didn't think it had that much effect. [16:02:39] sumanah: it's probably partially my fault. I think the email I have in labs right now is dead, and I've been inactive a fair bit so not watching labs-l at all. and I found out through rschen7754|away [16:03:16] Izhidez: ah, got it :) please do update your email in https://wikitech.wikimedia.org preferences [16:05:37] The problem was first noticed yesterday. It was working last week. I'd assume something happened over the weekend. [16:06:47] petan: no time yet. will do [16:06:55] sumanah: new address updated [16:07:35] petan: another thing though, I only go on labs when I have to reboot stuff...which is not that often, so it might be the reason I didn't see those notices [16:11:10] I know I saw a list of all projects in Labs somewhere. But where? any hints? [16:11:44] Silke_WMDE_: Main page, and then there's a link ("number of projects" or something like that). [16:12:30] got it, thx scfc_de [16:12:42] https://wikitech.wikimedia.org/wiki/Special:Ask/-5B-5BResource-20Type::project-5D-5D/-3F/-3FMember/-3FDescription/mainlabel%3D-2D/searchlabel%3Dprojects/offset%3D0 [16:13:03] I believe kma500 filed a bug, or has a TODO, to make that link a lot more prominent [16:13:15] you could beat her to the punch by filing it :) [16:20:00] [bz] (8NEW - created by: 2silke.meyer, priority: 4Unprioritized - 6normal) [Bug 51889] Make the link to list of all Labs projects more visible - https://bugzilla.wikimedia.org/show_bug.cgi?id=51889 [16:21:49] Am I right that in bugzilla the "bots" component is obsolete? [16:22:13] If so, I'd file a bug for deleting it too. [16:29:16]