[00:01:25] PROBLEM - Host arsenic is DOWN: PING CRITICAL - Packet loss = 100% [00:04:47] !log reedy synchronized wmf-config/ [00:04:52] Logged the message, Master [00:07:07] RECOVERY - Host arsenic is UP: PING OK - Packet loss = 0%, RTA = 26.51 ms [00:18:28] !log on gallium: moving data directories to a tmpfs [00:18:28] Logged the message, Master [00:19:27] New patchset: Reedy; "Move everything in the multiversion repo up a level..." [operations/mediawiki-multiversion] (master) - https://gerrit.wikimedia.org/r/31985 [00:19:42] Change merged: Reedy; [operations/mediawiki-multiversion] (master) - https://gerrit.wikimedia.org/r/31978 [00:20:43] Change abandoned: Reedy; "(no reason)" [operations/mediawiki-multiversion] (master) - https://gerrit.wikimedia.org/r/31985 [00:21:52] New patchset: Reedy; "Move everything in the multiversion repo up a level..." [operations/mediawiki-multiversion] (master) - https://gerrit.wikimedia.org/r/31987 [00:22:38] New review: Reedy; "We can now git submodule this into mediawiki-config now..." [operations/mediawiki-multiversion] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/31987 [00:23:15] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:48] Reedy: https://gerrit.wikimedia.org/r/#/c/31987/ abandoned or not? [00:29:25] yeah, I made a replacement [00:29:34] I nuked the useful README accidentally [00:29:37] so I just abandoned and redid it [00:29:49] it's not actually abandoned though [00:29:51] uhh, that is the replacement, even [00:30:07] 31985 is abandoned [00:30:48] yeah I see [00:30:57] I was getting confused the way the emails were grouped [00:32:08] heh [00:37:52] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.030 seconds [00:40:49] !log reedy synchronized php-1.21wmf3/cache/l10n/ 'Update l10n cache' [00:40:59] Logged the message, Master [00:43:15] -rw-r--r-- 1 udp2log udp2log 311 Nov 5 15:19 faapi.log [00:43:15] -rw-r--r-- 1 udp2log udp2log 221 Nov 5 15:19 fataapi.log [00:43:15] -rw-r--r-- 1 udp2log udp2log 117 Nov 5 15:19 fatalapi.log [00:43:15] -rw-r--r-- 1 udp2log udp2log 141 Nov 5 15:19 fatapi.log [00:43:16] wth? [00:43:43] I keep poking mutante to delete those :) [00:44:01] heh [00:44:12] I didn't realise the api was fat [00:44:44] !log awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend [00:44:49] Logged the message, Master [00:45:11] i wonder if faapi servers pr0n [00:47:32] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [00:49:55] !log awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/modules/mf-references.js 'touch file' [00:50:01] Logged the message, Master [00:59:33] !log awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/ 'touch files' [00:59:39] Logged the message, Master [00:59:42] !log awjrichards synchronized php-1.21wmf2/extensions/MobileFrontend/ [00:59:48] Logged the message, Master [01:04:11] binasher, LeslieCarr, mutante: would one of you mind flushing the mobile varnish cache? [01:07:39] New patchset: Reedy; "Add UserMerge to extension-list" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31996 [01:07:51] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31996 [01:10:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:34] New patchset: Reedy; "Add SUL login configuration for Wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31998 [01:12:25] New review: Reedy; "Marking as do not submit. Image missing" [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/31998 [01:13:15] or any other opsen? anyone available to flush mobile varnish cache? http://wikitech.wikimedia.org/view/MobileFrontend#Flushing_the_cache [01:14:37] !log slamming the apache cluster by wiping out all mobile caching yet again [01:14:45] Logged the message, Master [01:14:47] hah [01:14:54] hehehe [01:15:00] thanks, binasher [01:15:07] np [01:23:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.812 seconds [01:30:09] New patchset: Asher; "* replace our memcached ganglia plugin with one that works * fix pyconf install directory original source: git://github.com/ganglia/gmond_python_modules.git" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31999 [01:30:31] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31999 [01:38:01] New patchset: Reedy; "Add SUL login configuration for Wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31998 [01:38:26] New patchset: Reedy; "Add SUL login configuration for Wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31998 [01:39:10] New patchset: Asher; "path fix" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32001 [01:39:19] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32001 [01:41:41] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 244 seconds [01:41:53] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 253 seconds [01:43:29] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [01:46:38] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [01:48:17] New patchset: Reedy; "Include multiversion as a git submodule" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32004 [01:48:35] New patchset: CSteipp; "Disable editing on Wikivoyage by unmerged accts" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32005 [01:49:02] New review: Reedy; "This will need slightly careful handling on fenari - need to remove the existing directory first!" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/32004 [01:53:08] New patchset: Reedy; "Disable editing on Wikivoyage by unmerged accts" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32005 [01:53:51] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32005 [01:56:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:14] New patchset: Reedy; "Tidyup/simplify wikivoyage related config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32008 [01:57:47] !log reedy synchronized wmf-config/ [01:57:53] Logged the message, Master [01:58:13] New patchset: Reedy; "Tidyup/simplify wikivoyage related config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32008 [01:59:51] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 238 seconds [01:59:51] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 239 seconds [02:11:41] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.134 seconds [02:14:37] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [02:14:37] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [02:14:37] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [02:28:55] !log LocalisationUpdate completed (1.21wmf3) at Tue Nov 6 02:28:55 UTC 2012 [02:29:04] Logged the message, Master [02:55:06] !log LocalisationUpdate completed (1.21wmf2) at Tue Nov 6 02:55:06 UTC 2012 [02:55:13] Logged the message, Master [03:50:04] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [03:58:51] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 2 seconds [04:25:37] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [05:26:46] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: Puppet has not run in the last 10 hours [05:56:44] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [05:56:44] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [05:56:44] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [06:39:54] PROBLEM - Puppet freshness on arsenic is CRITICAL: Puppet has not run in the last 10 hours [07:08:49] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [08:37:22] New review: Hashar; "Approved. Will let you deploy it ;-) poke me online this afternoon." [operations/mediawiki-config] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/32004 [08:38:02] PROBLEM - Puppet freshness on erzurumi is CRITICAL: Puppet has not run in the last 10 hours [09:04:30] New patchset: Hashar; "import updated export-0.8.xsd" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32016 [09:06:29] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32016 [09:12:10] !log hashar synchronized docroot/mediawiki/xml 'import updated export-0.8.xsd' [09:12:19] Logged the message, Master [09:54:14] New patchset: Eloquence; "Add namespace aliases, namespace settings + extra namespaces for Wikivoyage." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32018 [10:09:06] New review: Hashar; "You have added the dewikivoyage namespaces under wgNamespaceAlias. I think they should be defined un..." [operations/mediawiki-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/32018 [10:14:43] New patchset: Eloquence; "Add namespace aliases, namespace settings + extra namespaces for Wikivoyage." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32018 [10:18:37] hmm [10:18:41] any known caching issues atm? [10:18:51] seeing some wide spread ones... [10:19:21] Vandalism that's staying visible for everyone after reversion until I action=purge it [10:22:20] !log nikerabbit synchronized php-1.21wmf3/extensions/Translate/utils/MessageGroupStats.php [10:22:24] Logged the message, Master [10:33:37] !log nikerabbit synchronized php-1.21wmf3/extensions/WebFonts/resources/ext.webfonts.fontlist.js [10:33:43] Logged the message, Master [10:48:14] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [10:54:10] New patchset: J; "increase memorylimit for videoscalers" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32026 [11:32:45] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32026 [11:33:52] !log reedy synchronized wmf-config/CommonSettings.php [11:34:02] Logged the message, Master [11:37:44] !log reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/ [11:37:50] Logged the message, Master [12:15:14] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [12:15:19] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [12:15:19] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [12:36:22] j^: ping? [13:47:56] New patchset: Hashar; "sync Zuul module with upstream and update our conf" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31009 [13:48:41] New review: Hashar; "PS4 made the URL, used in Gerrit comments, to points to the Jenkins console." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/31009 [14:22:00] paravoid: pong [14:26:40] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [14:33:17] New review: Silke Meyer; "Does not work - please see comment. Please improve - thx!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31252 [15:19:23] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [15:27:33] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: Puppet has not run in the last 10 hours [15:30:09] New patchset: Hashar; "beta: extensions now updated with git submodule update" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32047 [15:32:33] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.003 second response time on port 11000 [15:37:24] New review: Andrew Bogott; "This looks fine. But, I strongly encourage you to replace your 'git pull' bits with either 'git fet..." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/32047 [15:38:12] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32047 [15:58:08] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [15:58:08] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [15:58:08] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [16:05:13] <^demon> !log cleaned up fenari:/h/w/c/php-1.21wmf3 history so it can properly ff on pull [16:05:19] Logged the message, Master [16:06:48] !log demon synchronized php-1.21wmf3/extensions/Wikibase/repo/Wikibase.php 'Syncing out 856bad9d' [16:06:54] Logged the message, Master [16:29:05] New patchset: Tpt; "Fix a confusion in namespaces of br Wikisource: Author namespace where confused with index namespace." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32052 [16:41:11] PROBLEM - Puppet freshness on arsenic is CRITICAL: Puppet has not run in the last 10 hours [16:47:11] New patchset: Mark Bergsma; "Set bond-master on LACP slave interfaces, as required by Precise" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32054 [16:50:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32054 [16:52:15] New patchset: Mark Bergsma; "Revert "Set bond-master on LACP slave interfaces, as required by Precise"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32055 [16:52:31] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32055 [16:59:48] !log reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/ [16:59:56] Logged the message, Master [17:00:17] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32004 [17:00:55] New review: Anomie; "Approved but not merged yet?" [operations/mediawiki-multiversion] (master) C: 1; - https://gerrit.wikimedia.org/r/31987 [17:01:40] !log reedy synchronized multiversion/ 'Resync multiversion after changing to a submodule' [17:01:40] Logged the message, Master [17:01:56] PHP fatal error in /usr/local/apache/common-local/live-1.5/MWVersion.php line 17: [17:01:56] require_once() [function.require]: Failed opening required '/usr/local/apache/common-local/multiversion/MWVersion.php' (include_path='.:/usr/share/php:/usr/local/apache/common/php') [17:01:57] Reedy: [17:02:12] ffs [17:02:28] Feclk [17:02:42] Change merged: Reedy; [operations/mediawiki-multiversion] (master) - https://gerrit.wikimedia.org/r/31987 [17:02:58] Fixing [17:03:02] Reedy: Thanks. :-) [17:03:12] !log reedy synchronized multiversion/ 'Resync multiversion after changing to a submodule' [17:03:18] Logged the message, Master [17:03:28] GRRR [17:03:28] Fixed [17:03:43] win 58 [17:03:45] just got a page [17:03:58] Reedy: that you? [17:03:58] me too [17:03:58] Yup [17:04:02] okay [17:04:03] Sorry about that [17:04:04] is it fixed? [17:04:07] yes [17:04:52] oh, good [17:05:12] good :) [17:05:13] I thought a change had been merged, but apparently it hadn't been :( [17:05:27] shit happens [17:05:27] lol @ asher [17:05:32] New patchset: Reedy; "Update multiversion to master" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32059 [17:05:44] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32059 [17:07:40] phew [17:10:25] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [17:13:22] New patchset: Mark Bergsma; "Make sure to set the slave interfaces as up" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32060 [17:16:27] New review: Mark Bergsma; "This currently breaks Lucid hosts" [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/32060 [17:21:35] New patchset: Matthias Mullie; "Enable AFTv5 on beta" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32061 [17:46:36] New patchset: CSteipp; "Give 'usermerge' right to bureaucrats" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32067 [17:54:47] New review: Cmcmahon; "we'd like AFTv5 working on beta labs" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/32061 [17:57:40] RECOVERY - Host sq69 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [17:58:07] PROBLEM - Host sq70 is DOWN: PING CRITICAL - Packet loss = 100% [18:01:25] PROBLEM - Host sq69 is DOWN: PING CRITICAL - Packet loss = 100% [18:02:37] RECOVERY - Host sq69 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [18:06:22] PROBLEM - Host sq69 is DOWN: PING CRITICAL - Packet loss = 100% [18:18:15] !log reedy synchronized php-1.21wmf3/extensions/WikimediaMaintenance/ [18:18:22] Logged the message, Master [18:22:03] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: [18:22:08] Logged the message, Master [18:25:11] New patchset: Reedy; "Update dblists and wikiversions for wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32073 [18:25:24] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32073 [18:26:59] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: [18:27:05] Logged the message, Master [18:30:28] !log reedy synchronized wmf-config/InitialiseSettings.php [18:30:34] Logged the message, Master [18:33:29] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32008 [18:35:24] !log reedy synchronized wmf-config/ [18:35:32] Logged the message, Master [18:38:09] New patchset: Reedy; "Add frwikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32075 [18:38:36] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32075 [18:39:25] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: [18:39:35] Logged the message, Master [18:39:35] PROBLEM - Puppet freshness on erzurumi is CRITICAL: Puppet has not run in the last 10 hours [18:39:50] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: [18:39:51] Logged the message, Master [18:43:24] New patchset: Reedy; "Set licensing for wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32078 [18:44:13] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32078 [18:44:53] !log reedy synchronized wmf-config/InitialiseSettings.php [18:44:58] Logged the message, Master [18:45:16] New patchset: Reedy; "Add namespace aliases, namespace settings + extra namespaces for Wikivoyage." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32018 [18:45:40] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32018 [18:46:16] !log reedy synchronized wmf-config/InitialiseSettings.php [18:46:26] Logged the message, Master [18:47:46] RECOVERY - Host sq69 is UP: PING OK - Packet loss = 0%, RTA = 0.92 ms [18:50:14] RECOVERY - Host sq70 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [18:57:20] !log reedy synchronized php-1.21wmf3/extensions/WikimediaMaintenance/ [18:57:23] Logged the message, Master [18:57:44] RECOVERY - Varnish HTTP bits on sq69 is OK: HTTP OK HTTP/1.1 200 OK - 632 bytes in 0.004 seconds [19:03:17] PROBLEM - Host sq70 is DOWN: PING CRITICAL - Packet loss = 100% [19:18:26] drdee / ottomata around? [19:18:32] aight [19:18:41] yup [19:18:47] at your service [19:18:59] there's a bit of a jump in locke packet loss starting roughly last week [19:19:21] from negative percentages (wtf?) to ~3% [19:19:30] ideas? [19:19:43] negative is just a bug in how we calculate packet loss [19:20:02] how much packet loss do you observe? [19:20:02] should I think in absolute values? [19:20:30] just looking at ganglia, it seems to have jumped from ~1% for the past year-ish, to 3% [19:21:48] I added a couple of strings to the bannerImpression 1:100 filter, could that be it? [19:22:20] that filter doesn't seem that busy in the process list [19:32:50] does the timing correspond to when you added the filters? [19:34:08] ~.~. [19:47:59] ottomata: it's a little hard to tell from ganglia's crappy graphs, but I think no. it looks like the packet loss started a couple days before I tweaked the filter. [19:48:49] hm [19:48:58] sigh [19:49:00] I made changes on 10/4, 10/18, 10/29, 10/30, and 11/2 [19:49:14] fundraiser are fickle :-P [19:49:31] http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Miscellaneous+pmtpa&h=locke.wikimedia.org&v=0.0496745833333&m=packet_loss_average&jr=&js=&vl=%25 [19:50:00] that graph makes it look as though the loss started 12 days ago [19:50:30] which would be 10/26-ish [19:51:00] fwiw my changes are the only ones for that whole time range [19:51:46] yeah interseting [19:52:01] New patchset: Dzahn; "add redirect rules for education / LiAnna (RT-3843)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32092 [19:52:16] i suppose it could be changes in the log data [19:52:32] New patchset: Dzahn; "add redirect rules for education / LiAnna (RT-3843)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32092 [19:52:40] i guess so, but i dunno [19:52:52] i would think it would be more gradual than that if it were just growth [19:53:03] that is pretty dramatic [19:53:03] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32092 [19:53:35] can we look in the server logs to see what happened on the 26th? [19:54:04] maybe it's also a good habit to start logging our actions regarding udp2log [19:56:12] yeah [19:56:29] !log restarted udp2log on locke [19:56:37] Logged the message, Master [19:57:17] dzahn is doing a graceful restart of all apaches [19:57:41] !log dzahn gracefulled all apaches [19:57:43] Logged the message, Master [19:58:12] hey lookit that: http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=xlarge&c=Miscellaneous+pmtpa&h=locke.wikimedia.org&v=-0.607140252101&m=packet_loss_average&jr=&js=&vl=%25 [19:59:34] iiiinteresting [19:59:43] PROBLEM - Varnish traffic logger on cp1024 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:59:43] PROBLEM - Varnish traffic logger on cp1022 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:59:43] PROBLEM - Varnish traffic logger on cp1044 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:59:43] PROBLEM - Varnish traffic logger on cp1026 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:59:43] PROBLEM - Varnish traffic logger on cp1034 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:59:43] PROBLEM - Varnish traffic logger on cp1030 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:59:43] PROBLEM - Varnish traffic logger on cp1036 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:59:48] so our lesson is [19:59:51] PROBLEM - Varnish traffic logger on cp1042 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:59:51] PROBLEM - Varnish traffic logger on cp1032 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [19:59:58] always restart udp2log ourselves? [19:59:58] New patchset: Dzahn; "fix missing slashes in redirect rule" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32095 [19:59:59] after making a change? [20:00:15] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32095 [20:00:18] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:00:23] well there wasn't a change near the window where the pain started [20:00:28] drdee, there were defunct udp2log child procs on locke, which happens sometimes after puppet restarts udp2log [20:00:29] hm [20:00:36] PROBLEM - Varnish traffic logger on cp1021 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:00:36] PROBLEM - Varnish traffic logger on cp1043 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:00:36] PROBLEM - Varnish traffic logger on cp1027 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:00:36] PROBLEM - Varnish traffic logger on cp1035 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:00:44] so I'd say the lesson is that udp2log is untrustworthy [20:00:44] haha [20:00:45] uh huh [20:00:59] PROBLEM - Varnish traffic logger on cp1023 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:01:14] PROBLEM - Varnish traffic logger on cp1025 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:01:30] RECOVERY - Varnish traffic logger on cp1042 is OK: PROCS OK: 3 processes with command name varnishncsa [20:02:06] puppet should be monitoring defunct procs though... hmmm [20:02:15] err not puppet, nagios [20:03:01] RECOVERY - Varnish traffic logger on cp1044 is OK: PROCS OK: 3 processes with command name varnishncsa [20:03:05] oxygen is full of defunct processes too [20:03:31] dzahn is doing a graceful restart of all apaches [20:03:50] !log dzahn gracefulled all apaches [20:03:54] 13 of 'em [20:03:54] RECOVERY - Varnish traffic logger on cp1027 is OK: PROCS OK: 3 processes with command name varnishncsa [20:03:56] Logged the message, Master [20:04:48] RECOVERY - Varnish traffic logger on cp1032 is OK: PROCS OK: 3 processes with command name varnishncsa [20:06:18] RECOVERY - Varnish traffic logger on cp1036 is OK: PROCS OK: 3 processes with command name varnishncsa [20:08:04] RECOVERY - Varnish traffic logger on cp1024 is OK: PROCS OK: 3 processes with command name varnishncsa [20:08:05] ottomata: so the analytics 720s should be arriving today [20:08:12] I will be on site tomorrow to rack them for you [20:09:21] cooooool! [20:09:28] drdee, dschoon ^ [20:09:39] yaus [20:09:40] hey whaddya know. from locke: udp2log 19914 2.6 0.0 0 0 ? Z 19:56 0:20 [packet-loss] [20:10:03] * Jeff_Green declares packet-loss Le Broken [20:10:03] RobH to get all the scotch. [20:10:13] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [20:10:34] RECOVERY - Varnish traffic logger on cp1021 is OK: PROCS OK: 3 processes with command name varnishncsa [20:11:20] RECOVERY - Varnish traffic logger on cp1034 is OK: PROCS OK: 3 processes with command name varnishncsa [20:12:09] RECOVERY - Varnish traffic logger on cp1035 is OK: PROCS OK: 3 processes with command name varnishncsa [20:14:28] New patchset: Matthias Mullie; "Update WV config based on their current LocalSettings config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32096 [20:14:33] RECOVERY - Varnish traffic logger on cp1022 is OK: PROCS OK: 3 processes with command name varnishncsa [20:14:42] RECOVERY - Varnish traffic logger on cp1030 is OK: PROCS OK: 3 processes with command name varnishncsa [20:15:09] PROBLEM - Varnish traffic logger on cp1041 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:15:37] and now oxygen is not logging any packet loss at all [20:16:16] PROBLEM - Varnish traffic logger on cp1044 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:16:21] PROBLEM - Varnish traffic logger on cp1042 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [20:19:30] RECOVERY - Varnish traffic logger on cp1026 is OK: PROCS OK: 3 processes with command name varnishncsa [20:20:42] RECOVERY - Varnish traffic logger on cp1023 is OK: PROCS OK: 3 processes with command name varnishncsa [20:22:41] RECOVERY - Varnish traffic logger on cp1025 is OK: PROCS OK: 3 processes with command name varnishncsa [20:27:09] RECOVERY - Varnish traffic logger on cp1043 is OK: PROCS OK: 3 processes with command name varnishncsa [20:33:09] RECOVERY - Varnish traffic logger on cp1042 is OK: PROCS OK: 3 processes with command name varnishncsa [20:34:39] RECOVERY - Varnish traffic logger on cp1044 is OK: PROCS OK: 3 processes with command name varnishncsa [20:41:43] RECOVERY - Varnish traffic logger on cp1041 is OK: PROCS OK: 3 processes with command name varnishncsa [20:48:54] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [20:52:43] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32052 [20:53:37] !log reedy synchronized wmf-config/InitialiseSettings.php [20:53:39] Logged the message, Master [20:59:09] apergos: around? [21:00:04] !log imported frwikivoyage db dumps [21:00:15] Logged the message, Master [21:00:15] hexmode: left a while ago, has a day off [21:00:44] mutante: ah... So, can you help me get a tarball in place? [21:01:13] (not now, just want to check who I need to deal with and I knew apergos could do it) [21:01:28] hexmode: put a file on download.wm? yeah [21:01:37] just busy right now importing wikivoyage db and stuff [21:01:44] np [21:01:45] where is the file now? [21:02:44] mutante: it isn't ... my day has been crazy, so I'm just getting to it :( [21:03:14] hexmode: alright [21:03:22] mutante: but I don't want to keep you up too late, so let me know if you have to call it a night [21:03:28] binasher: just finished the first language import [21:03:43] mutante: great! no issues? [21:03:45] like "fr" is on all servers, no errors, looks ok so far [21:04:13] i am trying to NOT bunzip2 files on the db servers though and use 100% CPU :p [21:05:05] need to unpack, insert the one bin log line and repack, then transfer, unpack again [21:05:29] 100% cpu of one cpu core? ;) [21:06:05] oh yeah, true, that was just top [21:06:21] yeah, that shouldn't be a problem [21:06:46] otherwise there's always "nice" [21:06:57] "frwikivoyage" [21:07:02] ok, yep [21:11:37] frwikivoyage looks good [21:19:54] mark,we discussed setting up the email forwarders for @wikivoyage.org on mchenry - any issues with that? there's only 3-5 forwarders and we don't anticipate creating new ones [21:20:13] yes thats fine [21:20:27] if they have an alternative domain we can forward to that [21:20:38] or alternatively we can forward it to their mailserver unaltered, so on wikivoyage.org [21:20:46] it's just forwarders from wikivoyage.org to personal email addresses, I think [21:20:53] ok that's easiest then [21:21:27] just creating the alias file on mchenry, and having the correct MX records in DNS should make that work [21:22:00] cool [21:27:30] New patchset: Dereckson; "(bug 41834) Namespaces configuration for Wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32138 [21:31:05] New patchset: Matthias Mullie; "Update WV config based on their current LocalSettings config" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32096 [21:31:18] Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32138 [21:39:54] New patchset: Dereckson; "(bug 41834) Namespaces configuration for Wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32139 [21:40:30] New review: Dereckson; "Followup: Change If518dcc0" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32138 [21:42:22] New review: Dzahn; "fix array name per IRC talk. yep" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/32139 [21:42:22] Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32139 [22:01:39] !log adding wikivoyage.org/.de alias file on mchenry [22:01:44] Logged the message, Master [22:14:48] !log reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/ [22:14:49] Logged the message, Master [22:16:07] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [22:16:07] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [22:16:07] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [22:21:44] mutante: can you help me out now? [22:22:34] PROBLEM - MySQL Slave Running on db1019 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Duplicate entry 3 for key PRIMARY on query. Default datab [22:22:52] PROBLEM - MySQL Replication Heartbeat on db39 is CRITICAL: CRIT replication delay 239 seconds [22:23:01] PROBLEM - MySQL Replication Heartbeat on db1019 is CRITICAL: CRIT replication delay 247 seconds [22:23:14] PROBLEM - MySQL Replication Heartbeat on db64 is CRITICAL: CRIT replication delay 257 seconds [22:23:29] PROBLEM - MySQL Slave Running on db39 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Duplicate entry 3 for key PRIMARY on query. Default datab [22:23:34] hexmode: yep [22:23:37] PROBLEM - MySQL Slave Running on db64 is CRITICAL: CRIT replication Slave_IO_Running: Yes Slave_SQL_Running: No Last_Error: Error Duplicate entry 3 for key PRIMARY on query. Default datab [22:23:46] PROBLEM - MySQL Replication Heartbeat on db1003 is CRITICAL: CRIT replication delay 294 seconds [22:23:59] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 301 seconds [22:24:09] mutante: files are here: http://mah.everybody.org/mediawiki/ [22:24:14] PROBLEM - MySQL Replication Heartbeat on db1010 is CRITICAL: CRIT replication delay 319 seconds [22:25:19] hexmode: specific wishes where they should be ? URL-wise? [22:26:59] mutante: http://download.wikimedia.org/mediawiki/1.20 [22:27:25]