[00:48:05] New review: Jeremyb; "could leave [[Main Page]] out of the import I think" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61069 [01:00:32] New review: Tim Starling; "Looks good." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/61078 [01:02:09] Change merged: Tim Starling; [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/61033 [01:02:33] Change abandoned: Tim Starling; "Superseded" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/55841 [02:12:34] !log LocalisationUpdate completed (1.22wmf2) at Mon Apr 29 02:12:33 UTC 2013 [02:12:43] Logged the message, Master [03:25:36] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 29 03:25:35 UTC 2013 [03:25:43] Logged the message, Master [04:12:16] New patchset: Ori.livneh; "Add 'tcpircbot' Puppet class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61078 [04:14:01] New review: Ori.livneh; "PS3 allows filtering incoming connections by IPv6 CIDR range, specified in the configuration. If a C..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61078 [04:19:34] I said it would be around 5 lines, it ended up being 150. I was only off by a factor of 30, which is about the norm for software estimation. [04:20:00] * Aaron|home hands ori-l a bear to waltz with [04:20:21] heh [04:45:56] New review: Tim Starling; "netaddr.IPNetwork.__contains__ is apparently undocumented, but I confirmed that it does exist. At fi..." [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/61078 [04:45:58] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61078 [06:27:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:28:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [06:32:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [06:34:21] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 208 seconds [06:35:21] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [06:39:49] New patchset: Mattflaschen; "Add site icon config for Wikipedia." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61348 [09:35:48] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:35:48] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [09:35:48] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [09:35:48] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [10:14:06] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:23] PROBLEM - Puppet freshness on vanadium is CRITICAL: No successful Puppet run in the last 10 hours [10:53:33] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 184 seconds [10:54:33] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 6 seconds [11:14:00] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 229 seconds [11:16:00] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 6 seconds [11:19:00] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 186 seconds [11:21:00] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 21 seconds [11:43:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 229 seconds [11:45:56] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [11:49:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 210 seconds [11:55:56] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [12:14:04] New patchset: Matthias Mullie; "Enable Auto-archive for enwiki, dewiki, frwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60803 [12:19:21] mlitn: is it documented somewhere? what does it do? [12:19:39] I vaguely remember thar "archiving" here doesn't have our usual meaning but rather means "expiry" or "deletion" [12:22:51] Nemo_bis: http://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Auto-archive_comments [12:22:56] I'll add it to the commit msg too ;) [12:23:25] thanks [12:24:40] AFTv5 has these "filters" (like unreviewed, useful, …); from a UI pov, archiving will remove feedback in the "unreviewed" filte & display them in the "archived" filter (similar like how marking feedback "useful" moves feedback from "unreviewed" to "useful") [12:29:36] New patchset: Matthias Mullie; "Enable Auto-archive for enwiki, dewiki, frwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60803 [12:30:11] New review: Matthias Mullie; "Should not be merged before https://rt.wikimedia.org/SelfService/Display.html?id=5016 is completed" [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/60803 [12:33:06] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [12:59:32] mark: around? varnish purging broken again https://bugzilla.wikimedia.org/show_bug.cgi?id=47825 [13:16:45] drdee: finally have a contact with Cisco...just sent them everything on an1007 [13:16:49] fyi ^ [13:17:04] woot woot! thank you soo much!!! [13:17:40] had to get through support contract nonsense [13:31:02] * jeremyb_ spies east coasters [13:34:43] * jeremyb_ wonders whose week it is [13:44:46] <^demon> paravoid: Ping [13:44:54] ^demon: pong [13:45:16] <^demon> Hi! Got a second to look at a 3-line puppet change for me? [13:45:48] sure [13:45:52] <^demon> https://gerrit.wikimedia.org/r/#/c/61050/ [13:48:10] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61050 [13:50:17] <^demon> Thanks! [13:50:38] sure [14:17:14] mark, are you already looking at the problem with projectgid.rb? [14:17:52] andrewbogott: no, I don't know what I would do about that [14:18:10] i mean, clearly a fact generating something from a conf file like that is pretty broken :) [14:18:22] considering that the old ganglia manifests used that fact too, I wonder how it even worked in the first place... [14:18:31] yeah, me too. [14:18:32] is there a script that generates gmond.conf first in labs or something? [14:18:47] perhaps we should strip that down to write it out to /etc/projectgid [14:18:48] Although it looks like gmond.conf used to be produced by a python script? [14:18:51] and then write a puppet fact to read from there [14:18:57] I don't follow how these parts fit together. [14:19:08] there was certainly logic to write out gmond.conf for labs in a puppet template [14:19:51] Is generate-ganglia-conf.py unrelated? [14:19:52] and where is that script then? [14:21:30] Probably it did something dumb before like write 0 to the port and then come back on a second pass and fill it in. [14:21:35] hm [14:21:39] yuck [14:21:45] but yeah that does seem to be it [14:21:54] perhaps we can strip that down to just be a source for the projectgid fact [14:22:05] and write that to /etc/projectgid [14:22:13] then we don't need to muck around with ldap in ruby ;) [14:22:41] Or just embed ldap shell commands in the fact. [14:22:52] yeah [14:22:56] but presumably it doesn't change does it [14:23:03] nope [14:23:10] ew :) [14:23:17] (ldap in the fact) [14:23:26] so writing it out to the filesystem would perform better and be useful to other tools as well [14:23:37] It would. But, who would do the writing out? [14:23:49] i don't know how instances get created [14:24:00] just one shell command at instance creation time would already do the trick ;) [14:24:11] True. Hm... [14:25:45] * andrewbogott needs breakfast [14:40:00] New patchset: coren; "Preleminary toollabs module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59969 [14:43:15] you can use ldap as a hiera backend :) [14:43:59] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [14:43:59] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [14:45:46] New patchset: coren; "Fix gridengine class to be parametrized" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61377 [15:04:24] New patchset: Andrew Bogott; "Band-aid patch to get puppet running again on labs." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61381 [15:04:31] New patchset: Jforrester; "Deploy VisualEditor opt-in alpha to viwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61382 [15:05:41] New patchset: Jforrester; "Deploy VisualEditor opt-in alpha to viwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61382 [15:06:51] mark: https://gerrit.wikimedia.org/r/61381 [15:07:09] (Hopefully not the last word, but I want to get labs running again) [15:07:11] so what is going to fix up the projectgid? [15:07:22] i.e. what causes that script to run? [15:07:35] apergos: hey [15:07:41] yo [15:07:51] what's up with ms-be2? [15:08:17] I haven't looked at it today [15:08:30] let's see [15:09:10] mark, ganglia.pp sets up a cron that calls it. [15:09:18] weird [15:09:26] Oh, although, maybe not anymore… hm. [15:09:30] then it doesn't anymore indeed :) [15:09:38] why did you add ms-be9 without adding ms-be2 at the same time? [15:09:57] New patchset: Ottomata; "Adding Aaron Halfakar's new keys:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61383 [15:09:57] and possibly lowering the weight of ms-be1... [15:10:06] we're losing days if not weeks :/ [15:10:19] oh. ms-be2. :-D [15:10:31] necause it wasn't ready to go in [15:10:50] it was still being set up by steve (might still be tbh) [15:10:56] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61383 [15:11:19] and I couldn't wait any longer so I just put ms-be9 to 33, bs-be12 to 100 and pushed it out [15:11:20] still, after a week? [15:11:27] yes, still, after a week [15:11:32] so what is he doing instead? [15:11:50] for at least a couple days he wasn't on site [15:12:03] but chris will know the details better [15:12:29] i'm glad you're so on top of this [15:12:45] you're welcome :-P [15:14:07] mark: Did you disable that script because you determined that we don't need it, or was it an oversight? I'm just trying to get context before I dig into understanding how this used to work... [15:14:21] andrewbogott: must have been an oversight [15:14:25] ok [15:14:31] i didn't realize gmond.conf was used in a circular fassion for projectgid [15:14:36] (seriously wtf) [15:14:55] fashion [15:15:08] so feel free to put that back, or even restore ganglia.pp entirely for labs [15:15:28] as long as the new module remains invoked for esams and manutius [15:15:57] mark: The projectgid thing is obviously silly, I'm just trying to figure out if that python script does anything else we care about. [15:16:05] yeah [15:17:20] apergos: hasn't ms-be9 already reached 33%? [15:18:54] I guess it's about providing aggregate per-project ganglia graphs. That seems useful. [15:19:12] that would be the clusters and gmetad stuff [15:19:19] i'm just not sure where this is supposed to run [15:19:20] on every host? [15:19:43] can't it do that only on the ganglia gmetad host with ldap info or something? [15:19:51] apergos: looks like it to me [15:19:54] anyway [15:19:59] if needed we can leave that part separate [15:20:03] they're distinct functions [15:20:07] gmond collects the data on a host [15:20:24] mark: yeah, it doesn't make sense to run on every host. I'm still reading… (and have never written a ganglia conf so it's slow going) [15:20:24] a gmond aggregator collects it per cluster (centrally now, also in labs before) [15:20:24] apergos: I'd say let's up ms-be9, up ms-be12, add ms-be2, lower e.g. ms-be1 all in one go [15:20:36] and the gmetad takes the data from the aggregators and writes it to RRDs on disk [15:20:43] (and then there's a web frontend that reads that again) [15:21:41] I already pushed out rigns for ms-be9 and 12 [15:21:57] I did that last week middle of the week [15:22:13] yes, and these are done now [15:22:16] that's what I'm saying [15:22:23] you put ms-be9 to 33% [15:22:32] and that's already done as far as I can see [15:23:15] both of them are [15:23:23] let me check on that [15:23:42] it would be great if they are [15:28:27] New review: Jdlrobson; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60885 [15:30:33] mark, that cron is part of ganglia::collector, which it looks like you left intact. That wasn't part of your refactor was it? [15:30:37] So probably the cron is still getting installed. [15:35:25] no [15:36:02] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [15:36:30] i only changed the monitor part (for now) [15:36:45] ok. I'm going to merge my bandaid and see what happens :) [15:37:02] apergos: ? [15:37:09] worst case it will allow labs to work again minus ganglia [15:37:17] yes? [15:38:05] weren't you checking? [15:38:07] yes [15:38:13] New review: Andrew Bogott; "Dumb but should help in the short-term." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/61381 [15:38:14] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61381 [15:38:16] I'm now checking on the status of ms-be2 [15:49:34] New patchset: ArielGlenn; "ms-be2 new mac addr, new disk layout (720xd)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61388 [15:51:32] Andrew, plz to review https://gerrit.wikimedia.org/r/#/c/61377/ [15:51:33] ? [15:51:36] andrewbogott: ^ [15:52:38] New patchset: ArielGlenn; "ms-be2 new mac addr, new disk layout (720xd)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61388 [15:57:07] apergos: raid cfg is finished on ms-be2 so once all the puppet changes are done you are free to install [15:57:24] ok thanks [15:59:23] New review: Andrew Bogott; "(1 comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/61377 [16:00:37] New patchset: ArielGlenn; "ms-be2 new mac addr, new disk layout (720xd), remove dup stanza" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61388 [16:02:33] New patchset: coren; "Fix gridengine class to be parametrized" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61377 [16:02:49] andrewbogott: Ask, and thou shall recieve. [16:03:25] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61388 [16:03:46] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61377 [16:04:50] I'm about to push your change andrewbogott, Coren [16:04:56] thx [16:05:22] done [16:06:50] New patchset: Faidon; "Enable LVS check for ms-fe.eqiad.wmnet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61392 [16:08:03] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61392 [16:09:17] apergos: Danke [16:09:25] yw [16:16:45] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:06] New patchset: ArielGlenn; "ability to batch pages-logging dump (works aoround wikidata issue)" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/61394 [16:18:45] RECOVERY - Host ms-be2 is UP: PING OK - Packet loss = 0%, RTA = 26.57 ms [16:20:55] PROBLEM - swift-account-replicator on ms-be2 is CRITICAL: Connection refused by host [16:21:05] PROBLEM - swift-account-server on ms-be2 is CRITICAL: Connection refused by host [16:21:06] PROBLEM - swift-container-replicator on ms-be2 is CRITICAL: Connection refused by host [16:21:06] PROBLEM - SSH on ms-be2 is CRITICAL: Connection refused [16:21:06] PROBLEM - swift-object-updater on ms-be2 is CRITICAL: Connection refused by host [16:21:15] PROBLEM - swift-container-updater on ms-be2 is CRITICAL: Connection refused by host [16:21:15] PROBLEM - swift-account-reaper on ms-be2 is CRITICAL: Connection refused by host [16:21:15] PROBLEM - swift-container-auditor on ms-be2 is CRITICAL: Connection refused by host [16:21:15] PROBLEM - DPKG on ms-be2 is CRITICAL: Connection refused by host [16:21:25] PROBLEM - swift-container-server on ms-be2 is CRITICAL: Timeout while attempting connection [16:21:26] PROBLEM - swift-account-auditor on ms-be2 is CRITICAL: Timeout while attempting connection [16:21:26] PROBLEM - Disk space on ms-be2 is CRITICAL: Timeout while attempting connection [16:21:26] PROBLEM - RAID on ms-be2 is CRITICAL: Timeout while attempting connection [16:21:26] PROBLEM - swift-object-auditor on ms-be2 is CRITICAL: Timeout while attempting connection [16:21:35] PROBLEM - swift-object-server on ms-be2 is CRITICAL: Timeout while attempting connection [16:21:35] PROBLEM - swift-object-replicator on ms-be2 is CRITICAL: Timeout while attempting connection [16:22:23] apergos: have you seen ms-be11's broken disk? [16:22:31] yeah [16:22:34] ok [16:23:09] something's also wrong with ms-be9's nagios checks [16:23:17] see nagios [16:23:19] ok that's odd [16:23:23] nrpe config or something [16:23:51] * paravoid sighs on the mess that nagios is [16:24:04] (in general, not swift specifically) [16:25:26] PROBLEM - Host ms-be2 is DOWN: PING CRITICAL - Packet loss = 100% [16:25:35] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/61394 [16:25:45] okay, afk for now [16:26:47] later [16:30:35] RECOVERY - Host ms-be2 is UP: PING OK - Packet loss = 0 [16:44:04] !log reedy synchronized php-1.22wmf3/ 'initial sync of 1.22wmf3 files' [16:44:13] Logged the message, Master [16:49:01] !log reedy synchronized docroot [16:49:09] Logged the message, Master [17:01:37] !log attempt to fix NRPE checks on ms-be9, restart as correct user [17:01:45] Logged the message, Master [17:03:28] Uhhh [17:03:29] The authenticity of host 'mw1047 (10.64.0.77)' can't be established. [17:03:29] RSA key fingerprint is 10:af:44:e5:3a:4f:82:68:d6:9b:70:2b:2c:64:dd:4f. [17:03:29] Are you sure you want to continue connecting (yes/no)? The authenticity of host 'mw1048 (10.64.0.78)' can't be established. [17:03:29] RSA key fingerprint is 0f:10:1c:6a:be:6f:09:30:f6:f3:cf:51:51:69:a7:0f. [17:03:29] Are you sure you want to continue connecting (yes/no)? The authenticity of host 'mw1049 (10.64.0.79)' can't be established. [17:03:31] RSA key fingerprint is e4:1e:35:34:c2:a2:71:85:90:e6:bf:07:82:7e:54:da. [17:03:36] etc etc [17:03:44] Reedy: should be fixed already [17:03:50] i know why [17:03:52] heh [17:03:56] * Reedy runs it again [17:04:30] !log reedy synchronized live-1.5 [17:04:33] !log csteipp synchronized php-1.22wmf2/includes/ [17:04:33] !log removing reinstalled ms-be2 from ssh_known_hosts with ssh-keygen and then making it world-readable again manually so deployers don't get errors :p [17:04:45] ^ [17:04:45] Logged the message, Master [17:04:45] :D [17:04:46] Logged the message, Master [17:04:53] Logged the message, Master [17:05:16] ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R ms-be2 [17:05:24] this stuff makes it just readable for root [17:05:50] but if i dont remove it then could not connect to ms-be2 which has been reinstalled [17:06:11] same thing Tim fixed a couple days ago [17:06:33] yeah [17:07:36] apergos: success https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=ms-be9 [17:08:23] I looked at that file but it was dated apr 27, after ms-be9 reinstall [17:08:25] !log fix NRPE checks on ms-be2, running as "weird user id" bug [17:08:32] Logged the message, Master [17:08:33] why would it have been broken for that? [17:08:54] look at the user id it was running as before: [17:09:02] 4294967295 16383 0.0 0.0 25352 1172 ? Ss 16:40 0:00 /usr/sbin/nrpe -c /etc/icinga/nrpe.cfg -d [17:09:05] after: [17:09:08] icinga 23497 0.0 0.0 25476 1176 ? Ss 17:08 0:00 /usr/sbin/nrpe -c /etc/icinga/nrpe.cfg -d [17:09:25] killed the process, used init script to start it [17:09:28] what I mean is, the file had obviously been recreated after the reinstall [17:09:39] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=ms-be2 [17:09:41] like a few days after [17:09:53] for ms-be9; why was that not enough? [17:10:53] i just know right now this was a known bug we had with NRPE init script afair, and we fixed it and just had to kill and restart it everywhere at some point [17:11:09] both, be-2 and be-9 were in that same state though [17:11:13] ok and you run this on neon? [17:11:23] no, on the hosts themselves [17:11:26] oh [17:11:33] that's even weirder [17:11:51] the "server" is on the hosts, because it accepts the requests to execute local stuff [17:12:03] nagios remote plugin executor [17:12:21] root@ms-be2:~# /etc/init.d/nagios-nrpe-server start [17:12:39] really [17:13:02] I wonder if that's [17:13:07] well, it checks for running processes on the boxes [17:13:35] yes, we don't pass arguments to commandlines over the net [17:13:40] they are hardcoded [17:13:57] the option is called "dont_blame_nrpe" :p [17:13:57] no I was going to ask something else [17:14:01] ah,heh [17:16:15] I wonder if I just fixed db55 then [17:16:54] looks like it :) [17:16:54] maybe so [17:17:21] good. hopefully I will now remember this little issue, ahving done it for one of the hosts [17:17:58] I suppose after the first reboot it would be ok [17:18:10] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=db55 yep, done [17:19:24] as long as it's running as "icinga", and it does [17:25:13] New patchset: Andrew Bogott; "Add robots.txt and a privacy policy to mediawiki_singlenode." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61069 [17:25:13] New patchset: Andrew Bogott; "Add unicorn logo to labs mediawiki installs." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61401 [17:25:56] unicorn! <3 [17:29:45] !log reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache [17:29:52] Logged the message, Master [17:34:42] assume it's just a hiccup.... [17:34:43] PHP fatal error in /usr/local/apache/common-local/wmf-config/CommonSettings.php line 2779: [17:34:46] require() [function.require]: Failed opening required '/usr/local/apache/common-local/php-1.22wmf3/../wmf-config/ExtensionMessages-1.22wmf3.php [17:34:53] (include_path='/usr/local/apache/common-local/php-1.22wmf3/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/usr/local/apache/common-local/php-1.22wmf3:/usr/local/lib/php:/usr/share/php') [17:34:59] on test2 [17:35:04] Or scap being fail [17:35:19] As it's only test2wiki I'm not going to fix it to confirm whether scap does [17:35:24] !log removing asw-c1 for replacement [17:35:25] k [17:35:31] Logged the message, Master [17:35:34] thanks for the heads up though [17:40:01] New patchset: BBlack; "Work-In-Progress vhtcpd code." [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [17:43:45] Can someone fix the permissions of /home/wikipedia/common/php-1.22wmf2/.git/modules/extensions/WikiLove/index.lock ? Aaron is being greedy [17:50:11] ok, who killed ms-fe [17:50:17] also, where is the bot [17:50:26] eh? [17:50:28] !log restarting icinga-wm [17:50:36] Logged the message, Mistress of the network gear. [17:50:39] we just got paged for ms-fe lvs death [17:50:40] eqiad [17:50:46] but no icinga bot echo [17:51:15] oh eqiad [17:51:17] *whew* [17:51:26] I was only messin with tampa I swear [17:51:29] oh, lookit, pmtpa [17:51:31] as well [17:51:40] appservers.svc.pmtpa.wmnet is error [17:51:46] heh [17:51:50] Error: invalid magic word 'rootpagename' [17:52:05] sounds like waiting for localisation cache [17:52:13] pybal has depooled 2 (the other two aren't depooled because of the limits) mostly fails on health checks [17:52:17] i didn't get a page [17:52:19] what's up? [17:52:30] leslie and i got pages, our phones went off at same time [17:52:38] I got no pages either [17:52:38] for ms-fe.eqiad? [17:52:40] ms-fe.eqiad.wmnet [17:52:42] yep [17:52:50] that's not yet in use until later today [17:52:57] so you can ignore that for another few hours [17:53:15] cool [17:53:26] ok [17:53:35] aude: Or we get to beat someone up [17:53:43] heh [18:03:03] !log deploying new frontend squid conf [18:03:10] Logged the message, notpeter [18:05:57] !log reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache [18:06:05] Logged the message, Master [18:07:26] Actually. Can a root just delete /home/wikipedia/common/php-1.22wmf2/.git/modules/extensions/WikiLove/index.lock please? [18:07:48] Reedy: sure [18:08:00] Thanks [18:08:22] aude: Looks like we get to beat someone up [18:08:37] Reedy: done [18:08:48] thanks [18:09:54] Reedy: did you take down test2wiki just now? [18:10:00] Not just now [18:10:07] Last 40 minutes or so [18:10:40] https://gerrit.wikimedia.org/r/#/c/60198/ [18:10:40] new magic word [18:11:09] thanks aude [18:12:04] usually is localisation cache rebuild [18:12:16] w/o it, things fail hard [18:12:25] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61069 [18:12:37] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61401 [18:13:59] Oh wth [18:14:58] New review: preilly; "This is so full of win! :-)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61401 [18:18:52] paravoid: any chance you could review the ipython stuff? [18:21:15] !log reedy synchronized php-1.22wmf3/extensions/ [18:21:24] Logged the message, Master [18:27:54] Running scap number 2 [18:37:02] !log reedy synchronized php-1.22wmf2/.git/modules/extensions/WikiLove [18:37:10] Logged the message, Master [18:46:14] !log reedy Started syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it [18:46:22] Logged the message, Master [19:06:08] !log reedy Finished syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it [19:06:15] Logged the message, Master [19:06:52] Reedy: it no longer MWExceptions … but it calls itself 1.22alpha [19:09:43] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf3 [19:09:51] Logged the message, Master [19:11:04] New patchset: coren; "Preleminary toollabs module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59969 [19:11:28] New patchset: Reedy; "testwiki, test2wiki and mediawikiwiki to 1.22wmf3" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61417 [19:12:11] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61417 [19:12:45] Change abandoned: Catrope; "No longer needed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55137 [19:12:53] Change abandoned: Catrope; "No longer needed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55135 [19:15:20] I am pushing out new Parsoid code now, so please ignore any Parsoid-related alerts in the next minutes [19:19:12] !log kaldari synchronized php-1.22wmf2/extensions/Echo/maintenance/setEmailOptionTemp.php [19:19:20] Logged the message, Master [19:21:49] New patchset: coren; "Fix gridengine puppet manifest syntax" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61419 [19:22:02] the Parsoid update is done now [19:22:21] Ryan_Lane: Can you check out and +2 ^^ so that I don't look so stupid anymore? [19:23:01] :D [19:23:20] oh. it needs a jenkins ru [19:23:21] *run [19:23:36] Jenkin lags by ~1 min usually [19:25:00] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61419 [19:25:03] done [19:25:19] for obvious things like that you can self-merge, too :) [19:25:35] Yes, so, actually using puppet syntax in puppet manifests = a good thing. :-) [19:26:06] * Coren isn't entirely certain where his brain was residing during that particular one. [19:27:10] heh [19:32:04] New patchset: coren; "Preleminary toollabs module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59969 [19:37:27] New patchset: Faidon; "Ceph: be more resilient to malfunctioning OSDs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61423 [19:37:40]