[00:00:06] Krinkle, Ryan_Lane: i think that's a reasonable possible explanation for the slowness you saw (if it happens again, you can probably confirm with the 'show-queue' commands and other diagnostics) [00:00:08] yeah, would be nice to have ^demon here. he's the one mostly managing this [00:01:24] Krinkle: cool, thanks, i see the archives:) [00:02:57] Krinkle, Ryan_Lane: https://github.com/openstack-infra/config/blob/master/modules/openstack_project/manifests/review.pp [00:03:02] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 28 seconds [00:03:12] Krinkle, Ryan_Lane: (maybe pass that on to ^demon, or have him ping me next time he's around) [00:03:18] * Ryan_Lane nods [00:03:34] I'm reading up. I'll likely increase it now [00:03:34] Krinkle, Ryan_Lane: (maybe pass that on to ^demon, or have him ping me next time he's around) [00:03:37] Krinkle, Ryan_Lane: we had some growing pains with our gerrit, and eventually found those google groups links in the comments [00:03:43] I doubt anyone has touched that since I initial set it up [00:03:52] *initially [00:03:57] Krinkle, Ryan_Lane: which i think are the secret tuning documentation. :/ [00:04:03] heh [00:04:16] fwiw, zuul already has enabled debug.log since the lats (fixed) issue [00:04:53] Krinkle, Ryan_Lane: so anyway, the URLs in that file should help explain some of the params, and then there's our values, plus some (pretty poor) notes [00:05:11] thanks [00:07:48] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [00:07:59] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 00:07:55 UTC 2013 [00:08:48] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:08] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 00:09:02 UTC 2013 [00:09:48] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:09] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 00:10:04 UTC 2013 [00:10:48] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:08] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 00:10:59 UTC 2013 [00:11:48] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:49] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 00:11:47 UTC 2013 [00:12:48] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:08] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 00:13:06 UTC 2013 [00:13:49] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 217 seconds [00:18:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [00:22:24] Krinkle: Do you know why PhantomJS is timing out on REL1_20? [00:22:41] csteipp: It isn't timing out, Apache is serving HTTP 500 [00:22:55] csteipp: https://bugzilla.wikimedia.org/show_bug.cgi?id=47639 [00:22:58] Ah, cool. [00:22:59] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [00:23:24] as a result, there is no QUnit on the page (or anything on the page for that matter) so the grunt-phantomjs is timing out [00:23:56] csteipp: looks like there is a bug in < 1.21 that causes sqlite permissions errors [00:23:59] 1.21 and master are fine [00:24:23] It is only happening to qunit and not phpunit because phpunit is testing from command line, bypassing Apache. [00:24:29] it isn't a bug with qunit though [00:25:16] That makes sense, I hadn't seen that bug. I'll leave you and hasher to it. Thanks! [00:33:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 232 seconds [00:34:59] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [00:45:58] !log tstarling synchronized wmf-config/PrivateSettings.php [00:46:06] Logged the message, Master [00:48:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 217 seconds [00:49:20] The Nagios page said http://nagios.wikimedia.org "is currently an alias to [[spence]]". Is that still true for its replacement https://icinga.wikimedia.org/ ? [00:53:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [00:58:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [00:59:23] spagewmf: I think icinga runs on neon [00:59:36] I noticed that the Parsoid cluster is being hit by both [00:59:56] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 21 seconds [01:06:41] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [01:16:28] Published patchset: Ryan Lane; "Firewall redis from the outside world on vumi" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61732 [01:16:30] Published patchset: Ryan Lane; "Firewall off redis on stat1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61735 [01:16:44] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61732 [01:16:59] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61735 [01:20:02] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 188 seconds [01:24:01] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [01:25:01] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 13 seconds [01:26:20] Published patchset: Tim Starling; "Use a password for session redis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61734 [01:29:01] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [01:29:44] New review: Aaron Schulz; "Don't forget $wgJobQueueAggregator (which could also be switched to use the actual queue server inst..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61734 [01:30:02] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 11 seconds [01:37:55] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:37:55] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [01:37:55] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [01:37:55] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:38:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 232 seconds [01:43:56] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 16 seconds [01:48:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [01:50:55] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 28 seconds [01:53:33] New patchset: Tim Starling; "Use a password for job queue and session redis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61734 [01:54:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 214 seconds [01:58:26] New patchset: Tim Starling; "Require a password for Redis" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61740 [01:58:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [02:02:43] New patchset: Aaron Schulz; "Turned autoResync back on again for multiwrite backend." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61741 [02:02:56] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 28 seconds [02:05:10] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61741 [02:07:53] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [02:09:08] !log aaron synchronized wmf-config/filebackend.php 'Turned autoResync back on again for multiwrite backend' [02:09:16] Logged the message, Master [02:14:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [02:16:40] !log LocalisationUpdate completed (1.22wmf2) at Wed May 1 02:16:40 UTC 2013 [02:16:49] Logged the message, Master [02:17:23] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [02:18:03] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [02:25:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 197 seconds [02:27:30] !log LocalisationUpdate completed (1.22wmf3) at Wed May 1 02:27:30 UTC 2013 [02:27:38] Logged the message, Master [02:29:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [02:31:04] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [02:43:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 185 seconds [02:49:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [02:56:27] New patchset: Tim Landscheidt; "Preliminary toollabs module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59969 [02:58:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [03:02:16] PROBLEM - RAID on analytics1017 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [03:02:55] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 28 seconds [03:05:48] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [03:13:58] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [03:18:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [03:23:58] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 232 seconds [03:29:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 220 seconds [03:35:59] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [03:49:56] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed May 1 03:49:55 UTC 2013 [03:50:00] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 182 seconds [03:50:04] Logged the message, Master [03:53:00] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 22 seconds [03:59:00] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [04:00:00] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 13 seconds [04:06:18] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:07:59] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 04:07:53 UTC 2013 [04:08:18] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:08] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 04:08:59 UTC 2013 [04:09:18] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:08] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 04:10:02 UTC 2013 [04:10:18] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:59] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 04:10:57 UTC 2013 [04:11:18] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:48] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 04:11:47 UTC 2013 [04:12:18] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:38] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 04:12:29 UTC 2013 [04:13:18] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:39] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 04:13:35 UTC 2013 [04:14:18] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:18:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [04:21:59] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [04:27:10] New patchset: Faidon; "Add an initial ferm module & base::firewall class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61744 [04:28:59] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 218 seconds [04:34:28] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [04:38:03] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [04:44:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [04:47:03] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 23 seconds [04:57:06] New patchset: Tim Starling; "Move logmsgbot from fenari to neon" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61746 [04:59:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [05:02:02] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 11 seconds [05:02:29] TimStarling: logmsgbot is not registered. [05:04:09] yeah, I guess we should register it [05:04:21] New review: Tim Starling; "I confirmed with strace that nc -q0 does an orderly shutdown:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61746 [05:05:03] is it possible to convince gerrit that not every long number is a partial hash? [05:05:07] https://bugzilla.wikimedia.org/show_bug.cgi?id=22313 [05:05:12] "Register nickserv account for logmsgbot" [05:05:31] funny [05:05:45] https://bugzilla.wikimedia.org/show_bug.cgi?id=45780 "Gerrit mangles Wikipedia permalinks with anchor" [05:05:50] 6 comments and nobody actually does it [05:06:15] At least Gerrit didn't mangle your comment like it does with URLs. [05:06:26] You can add preceding 0s as a workaround. [05:06:31] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [05:06:33] Though you usually need like 30 of them. [05:08:15] gerrit-wm is also unregistered. [05:08:27] I think registering an in-use nick is difficult/impossible. [05:11:30] Heh, I thought of netcat the other day when we were discussing UDP, XMPP, etc. [05:14:02] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [05:14:04] Ryan_Lane, re stat1 lockdown & firewal, will your change affect UDP events that vumi sends to analytics? [05:14:19] Ryan_Lane, sorry, not stat1, zhen & silver [05:18:04] New patchset: Tim Starling; "Move logmsgbot from fenari to neon" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61746 [05:20:02] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 13 seconds [05:24:00] TimStarling: https://gerrit.wikimedia.org/r/#/c/61627/ adds a README to the tcpircbot module (..making good on my threat to extend it) that explains more clearly how it is parametrized [05:38:15] logmsgbot: /msg NickServ SET PRIVATE ON [05:43:58] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 232 seconds [05:44:58] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 24 seconds [05:59:19] yurik: did you look at the change? [05:59:41] yurik: because all it does it block the redis port outside of our network [06:00:25] Ryan_Lane, there was no link in the email, wasn't sure of the best place to check. Cool, sounds like no issues there. [06:01:15] "I've firewalled these to 208.80.152.0/22" ;) [06:01:31] Ryan_Lane, my point exactly! [06:02:05] too cryptic for a casual observer that notices that it has to do with vumi that i will have to support (at least in part) [06:02:16] that's cryptic? [06:02:42] sigh - i wasn't sure what exactly you blocked. UDP traffic does not go through the firewalls too well. [06:03:11] hence it was easier to ask then to assume everything is in order [06:03:15] heh [06:03:28] well, I thought the thread would provide enough context as to what it meant [06:04:01] * Ryan_Lane shrugs [06:04:05] no issues asking, though [06:04:23] Ryan_Lane, some day it will be enough for me. I hope. [06:04:42] :) [06:06:04] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [06:12:14] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [06:23:44] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 186 seconds [06:24:44] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 5 seconds [06:26:44] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 06:26:37 UTC 2013 [06:27:04] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [06:27:24] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 06:27:21 UTC 2013 [06:28:04] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [06:28:05] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 06:27:57 UTC 2013 [06:29:04] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [06:45:27] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [06:45:27] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [07:08:15] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [07:38:24] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [08:06:54] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [08:07:45] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 08:07:43 UTC 2013 [08:07:54] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:25] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 08:08:22 UTC 2013 [08:08:54] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:04] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 08:08:57 UTC 2013 [08:09:54] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:04] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 08:12:54 UTC 2013 [08:13:54] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [08:18:04] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:45] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 214 seconds [08:24:44] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 22 seconds [08:59:43] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 203 seconds [09:01:43] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 18 seconds [09:05:51] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [09:13:41] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 198 seconds [09:14:41] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 11 seconds [09:37:30] g/win 5 [10:05:46] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [10:08:21] !log Removed old Toolserver IP range 91.198.174.192/27 on primary and backup router interfaces [10:08:29] Logged the message, Master [10:30:05] !log reedy synchronized php-1.22wmf2/extensions/ParserFunctions/ 'Update to master to fix common fatals' [10:30:13] Logged the message, Master [10:31:19] professor.pmtpa.wmnet [10.0.6.30] 2003 (cfinger) : Connection timed out [10:32:01] Reedy: that's not how you do it. You have to do it like this: [10:32:24] PROBLEM - Connectivity on professor is CRITICAL: CRIT Connection timed out [10:32:42] the repetition of CRITICAL / CRIT is essential [10:33:01] but then as the tension is almost unbearable you have to transition rapidly to: [10:33:40] RECOVERY - Connectivity on professor is OK: OK Ping reply 23 ms [10:34:04] now practice [10:36:34] New patchset: Reedy; "Update php symlink to 1.22wmf2" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61762 [10:36:43] ori-l: I think it's bedtime [10:36:51] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61762 [10:37:15] thereabouts, yeah [11:06:01] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [11:30:52] New patchset: ArielGlenn; "script to shovel in pages-logging xml file into logging table" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/61765 [11:31:17] Change merged: ArielGlenn; [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/61765 [11:32:41] aww [11:38:01] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [11:38:01] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [11:38:02] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:38:02] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [12:06:26] New patchset: Physikerwelt; "Intial version of puppet script for LaTeXML" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61767 [12:07:51] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:01] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 12:08:55 UTC 2013 [12:09:51] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:01] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 12:09:58 UTC 2013 [12:10:51] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:01] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 12:10:57 UTC 2013 [12:11:51] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:01] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 12:11:51 UTC 2013 [12:12:50] New review: Physikerwelt; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61767 [12:12:51] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:21] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 12:13:13 UTC 2013 [12:13:51] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:51] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 12:13:47 UTC 2013 [12:14:51] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:51] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [12:26:32] RECOVERY - search indices - check lucene status page on search1017 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.024 second response time [12:31:25] New patchset: ArielGlenn; "debian packaging for php utfnormal extension" [operations/debs/utfnormal] (master) - https://gerrit.wikimedia.org/r/61768 [12:47:16] PROBLEM - SSH on gadolinium is CRITICAL: Server answer: [12:48:16] RECOVERY - SSH on gadolinium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:48:37] PROBLEM - SSH on caesium is CRITICAL: Server answer: [12:49:37] RECOVERY - SSH on caesium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:04:26] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [13:32:30] Change merged: ArielGlenn; [operations/debs/utfnormal] (master) - https://gerrit.wikimedia.org/r/61768 [13:39:01] New patchset: ArielGlenn; "add .gitignore, .gitreview files" [operations/debs/utfnormal] (master) - https://gerrit.wikimedia.org/r/61772 [13:40:41] Change merged: ArielGlenn; [operations/debs/utfnormal] (master) - https://gerrit.wikimedia.org/r/61772 [14:02:57] New patchset: Reedy; "(bug 47418) Babel configuration for th.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60409 [14:03:05] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60409 [14:03:13] New patchset: Reedy; "$wgNamespaceRobotPolicies for dewiki: Add 101 and 829" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60445 [14:03:32] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60445 [14:05:43] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [14:07:26] !log reedy synchronized wmf-config/InitialiseSettings.php [14:07:35] Logged the message, Master [14:07:45] mw1182: ssh: Could not resolve hostname mw1182: Name or service not known [14:11:12] Reedy: i will take a look at mw1182 [14:12:20] New patchset: Reedy; "(bug 45979) Set $wgCategoryCollation to 'uca-vi' on all Vietnamese-language wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60705 [14:12:31] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60705 [14:13:14] !log reedy synchronized wmf-config/InitialiseSettings.php [14:13:22] Logged the message, Master [14:34:41] PROBLEM - Puppet freshness on cp1031 is CRITICAL: No successful Puppet run in the last 10 hours [14:39:24] New patchset: ArielGlenn; "have rest of eqiad snapshot hosts use standard snap partition recipe" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61778 [14:41:52] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61778 [14:54:47] New review: Hydriz; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/61428 [15:05:02] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [15:13:18] New patchset: Demon; "General code cleanup" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/60444 [15:18:03] Ryan_Lane: when you have a few minutes, puppet-merge is claiming it can't ff and it's showing a different commit id for https://gerrit.wikimedia.org/r/#/c/61735/ than gerrit knows.. any ideas? [15:19:33] Change merged: jenkins-bot; [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/60444 [15:22:03] New patchset: Burthsceh; "(bug 47933) Rename Module talk namespace for Japanese Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61782 [15:53:42] !log Converted link between cr2-knams:xe-1/0/0 and csw2-esams:xe-0/1/1 to LACP aggregated link cr2-knams:ae1 / csw2-esams:ae2 [15:53:50] Logged the message, Master [15:57:06] apergos: I cherry-picked some commits in yesterday [15:57:18] eurgh [15:57:19] git reset --hard origin/master [15:57:24] err [15:57:28] origin/production [15:57:39] let me see how that does [15:57:50] (yeah, too bad our branch isn't master, eh? ) [15:58:56] did you already do this? I see my change showing up in git log [15:59:13] no, I didn't [15:59:53] hrm [16:00:21] well puppet-merge tells me there's nothing to merge now so... [16:00:55] thanks? :-D [16:04:26] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:05:50] midas touch [16:08:06] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 16:07:56 UTC 2013 [16:08:26] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:06] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 16:09:02 UTC 2013 [16:09:26] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:06] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 16:10:05 UTC 2013 [16:10:26] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:06] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 16:11:01 UTC 2013 [16:11:27] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:56] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 16:11:49 UTC 2013 [16:12:26] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:36] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 16:12:31 UTC 2013 [16:13:06] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:26] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:47] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Wed May 1 16:13:40 UTC 2013 [16:14:26] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:26] New patchset: Ori.livneh; "Lint mediawiki_singlenode module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61787 [16:46:26] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [16:46:27] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [16:48:11] New patchset: Danny B.; "cswiktionary: Set display title restriction" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61789 [16:53:10] !log torrus deadlocked, fixin [16:53:18] Logged the message, RobH [16:56:57] ottomata, I'm a bit uneasy about the inbetween state that puppetmaster::self is in right now. Are you mostly done with your dev work there and it's just a question of updating the docs? [17:00:56] heya LeslieCarr, you there? [17:01:03] yes [17:01:08] !log torrus compiling, manutilus apache offline [17:01:15] Logged the message, RobH [17:01:34] we want to move the udp2log mluticast relay that is currently running on oxygen over to gadolinium so we can upgrade oxygen without losing logs [17:01:57] there shouldn't be a problem with running 2 multicast relays on the same mulitcast group at once, should there? [17:02:25] i guess the question is: do you know if there are any problems if there are multiple senders of multicast traffic? [17:06:20] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:04] LeslieCarr: ^^ :) [17:11:11] should not be [17:11:11] nope [17:11:11] well data could be double received by the receivers [17:13:11] New patchset: Lcarr; "latest release is wm10 not wm8 in our apt repo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61793 [17:13:15] New review: Reedy; "Why are you changing quotes at the same time and moving stuff around?" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/61782 [17:14:49] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61793 [17:15:28] hmm, nooo, i think it wouldn't [17:15:31] LeslieCarr: [17:15:36] because each log will only be sent ot a single relay [17:15:39] oh okay [17:15:40] plan: [17:15:40] then yeah [17:15:48] i was thinking plan was could be sent out by boh at once [17:15:49] set up multicast relay on gadolinum [17:16:02] then deploy config changes to frontends to use gadoilnium IP instead of oxygen IP [17:16:08] once we are sure that has all rolled out [17:16:12] then we can stop oxygen relay [17:19:48] !log puppet working on bits caches again after gerrit change 61793 [17:19:54] cool [17:19:57] Logged the message, Mistress of the network gear. [17:20:07] New review: Andrew Bogott; "I tested this and it works fine. I have a couple of other minor patches to this module pending; I'l..." [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/61787 [17:20:08] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61787 [17:24:34] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61627 [17:33:02] !log kaldari synchronized php-1.22wmf2/extensions/Echo/Hooks.php 'syncing Hooks in Echo' [17:33:09] Logged the message, Master [17:35:03] Reedy, re https://bugzilla.wikimedia.org/show_bug.cgi?id=47945 I created the RT ticket for the DNS entry, for the apache redirect one (that you created last year: https://bugzilla.wikimedia.org/show_bug.cgi?id=36477#c3 ) - is that just a date change? [17:38:00] New patchset: BBlack; "Work-In-Progress vhtcpd code." [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [17:38:53] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [17:38:57] * Reedy looks blankly [17:39:00] Hey Krinkle, it looks like PhantomJS is still broken for REL1_20.. is there a way to disable that? Or make it non voting so we can get the security release merged today? [17:39:26] ohai Reedy :) [17:39:37] csteipp: open Special:JavaScriptTest/qunit in your browser (having checked out the gerrit change) and verify it runs [17:39:39] (I forgot hashar was out today on vacation, otherwise I'd be bugging him) [17:39:41] then just override the score and merge it [17:45:43] New review: Faidon; "Yes, it'd be great to reuse existing packages instead of embedding dozens of jars in the same packag..." [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/53170 [17:59:26] Probably needs to be announced first [17:59:43] And there's 2 to create currently [17:59:53] !log kaldari synchronized php-1.22wmf3/extensions/Echo/Hooks.php 'syncing Hooks in Echo' [17:59:55] Logged the message, Master [18:00:06] Reedy, yeah it was announced :) [18:00:09] re https://gerrit.wikimedia.org/r/61805 # Uploads are offsite on the previous one RewriteCond %{SERVER_ADDR} !^211\.115\.107 [18:00:09] was added? [18:00:24] What? [18:00:33] I assume Thehelpfulone means this: http://lists.wikimedia.org/pipermail/wikimedia-l/2013-May/125659.html [18:01:08] I don't [18:01:16] oh never mind Reedy, in the RT ticket you created for wikimania2013wiki, https://rt.wikimedia.org/Ticket/Display.html?id=2904 that pastebin you linked to had the line " RewriteCond %{SERVER_ADDR} !^211\.115\.107" but I see it's not there on the live version [18:01:18] New patchset: Reedy; "Add virtualhost for wikimania2014wiki" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/61804 [18:01:18] * greg-g steps out of it [18:03:51] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl to 1.22wmf3 [18:03:59] Logged the message, Master [18:07:33] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [18:08:00] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia to 1.22wmf3 [18:08:07] Logged the message, Master [18:09:25] * Reedy kicks professor [18:11:18] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf3 [18:11:25] Logged the message, Master [18:13:29] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikivoyage and wikiversity to 1.22wmf3 [18:13:38] Logged the message, Master [18:15:33] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wiktionary to 1.22wmf3 [18:15:42] Logged the message, Master [18:16:03] https://gerrit.wikimedia.org/r/#/c/61807/ [18:16:12] Reedy: whenever ready for wikidata ^ [18:16:14] New patchset: Reedy; "Everything bar open 'pedia projects over to 1.22wmf3" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61808 [18:16:35] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61808 [18:18:43] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [18:19:47] !log reedy synchronized php-1.22wmf3/extensions/ 'Update extensions for wikibase deploy' [18:19:54] Logged the message, Master [18:20:18] Lots of class not found fatals [18:20:23] I wonder if they are just transitionary [18:20:26] !log owa1/2/3 offlinging and wiping [18:20:35] Logged the message, RobH [18:20:57] I'm going with yes [18:23:04] Reedy: we need localisation cache updated :/ [18:23:06] not urgent [18:23:16]