[00:03:07] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Returned itwikisource to 1.21wmf5 [00:03:15] Logged the message, Master [00:10:50] !log olivneh synchronized php-1.21wmf5/extensions/EventLogging/content/JsonSchemaHooks.php 'Update SCHEMA_NS perms on wmf5' [00:10:57] Logged the message, Master [00:12:59] LeslieCarr: Did my fix actually fix it? [00:15:00] RoanKattouw: nope, err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template pybal/pybal.conf.erb: undefined method `sort' for :undef:Symbol at /var/lib/git/operations/puppet/manifests/lvs.pp:754 on node lvs4.wikimedia.org [00:15:34] OK so same error :9 [00:15:36] * :( [00:15:43] I wonder if it's possible to get a more precise line number there [00:20:44] !log catrope synchronized php-1.21wmf4/extensions/VisualEditor 'Updating VisualEditor to master' [00:20:50] Logged the message, Master [00:21:05] !log catrope synchronized php-1.21wmf5/extensions/VisualEditor 'Updating VisualEditor to master' [00:21:11] Logged the message, Master [00:23:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:39:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.022 seconds [00:41:42] !log olivneh synchronized php-1.21wmf5/extensions/EventLogging/content/JsonSchemaHooks.php 'Assign NS_SCHEMA rights to editinterface' [00:41:49] Logged the message, Master [00:52:10] !log aaron synchronized php-1.21wmf5/includes/filebackend/SwiftFileBackend.php 'deployed db4574540c993aeab8f0840de916e47fdac6199f' [00:52:17] Logged the message, Master [01:00:03] PROBLEM - Host parsoid.svc.eqiad.wmnet is DOWN: CRITICAL - Network Unreachable (10.2.2.28) [01:00:30] PROBLEM - Host parsoid.svc.pmpta.wmnet is DOWN: CRITICAL - Network Unreachable (10.2.1.28) [01:00:52] RoanKattouw: i'm going to disable those nagios checks as they paged us [01:02:36] LeslieCarr: :O I guess I should have put in critical=>false or something? [01:02:41] Sorry about that :( [01:02:45] yeah :) [01:02:47] is ok [01:03:00] Well, we learned something about LVS I guess :) [01:03:11] (Also, puppet on lvs4 is broken) [01:04:42] !log catrope synchronized php-1.21wmf4/cache/l10n/ 'Rebuild l10n cache' [01:04:49] Logged the message, Master [01:12:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:16:07] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [01:17:36] !log catrope synchronized php-1.21wmf5/cache/l10n/ 'Rebuild l10n cache' [01:17:44] Logged the message, Master [01:26:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.006 seconds [01:31:06] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [01:38:00] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [01:38:22] RoanKattouw_away: afaik, puppet broken on lvs4 is known issue and not new.. have a nice weekend [01:58:54] New review: Eloquence; "To clarify, this is just for a small subset of articles right now, correct? Can the #/% of articles ..." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/34964 [01:59:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.027 seconds [02:24:14] !log LocalisationUpdate completed (1.21wmf5) at Sat Dec 1 02:24:14 UTC 2012 [02:24:22] Logged the message, Master [02:45:47] !log LocalisationUpdate completed (1.21wmf4) at Sat Dec 1 02:45:47 UTC 2012 [02:45:55] Logged the message, Master [03:17:10] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [03:19:07] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [03:22:43] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [03:23:19] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [03:53:11] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.051 second response time [05:33:43] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [07:01:08] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [07:01:08] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [07:17:11] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [07:17:11] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [08:07:00] PROBLEM - Puppet freshness on lvs3 is CRITICAL: Puppet has not run in the last 10 hours [08:07:00] PROBLEM - Puppet freshness on lvs4 is CRITICAL: Puppet has not run in the last 10 hours [08:20:03] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [08:42:57] ehm, is memcached on labsconsole down? it is extremely slow and tells me I don't have cookies enabled [09:27:59] Ryan_Lane, is memcached on labsconsole down? it is extremely slow and tells me that I don't have cookies enabled [09:29:15] looking [09:31:02] it was having issues [09:31:12] I restarted it [09:31:28] ugh. seems I have a bug with sessions still [09:32:09] with project keystone tokens [09:32:13] * Ryan_Lane groans [09:48:19] New patchset: MaxSem; "WIP: OSM module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36222 [11:17:05] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [11:32:05] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [11:39:08] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [13:21:43] New patchset: Raimond Spekking; "Enable AFTv5 for dewiki (WiP - DO NOT MERGE)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34964 [13:23:26] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [15:34:17] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [16:52:53] New patchset: Silke Meyer; "Added install_path param for wikidata.pp, files are affected by (https://gerrit.wikimedia.org/r/#/c/35313/2)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36353 [16:56:20] New review: Silke Meyer; "I think, we overlooked this when testing the new $install_path parameter (https://gerrit.wikimedia.o..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/36353 [17:02:35] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [17:02:35] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [17:04:05] New patchset: Isarra; "(Bug 39361) Clean up Wikimedia site favicon" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/36105 [17:12:03] New review: Isarra; "Krinkle: This one is bug 39361 attachment 11430." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/36105 [17:18:38] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [17:18:38] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [17:26:17] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [17:27:47] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.002 second response time on port 11000 [18:07:41] PROBLEM - Puppet freshness on lvs3 is CRITICAL: Puppet has not run in the last 10 hours [18:07:41] PROBLEM - Puppet freshness on lvs4 is CRITICAL: Puppet has not run in the last 10 hours [18:20:44] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [21:17:58] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [21:32:58] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [21:40:01] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [23:24:04] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours