[00:00:04] New patchset: Bhartshorne; "first draft of the swift cleaner stuff. I know this doesn't work but I want to check it in for reviews." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3134 [00:00:14] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.404 seconds [00:00:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3134 [00:02:21] where could i find statistics of browsers visiting either given wiki or all our servers? [00:03:23] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 6.462 seconds [00:03:34] Danny_B|backup: http://stats.wikimedia.org/wikimedia/squids/SquidReportClients.htm [00:03:47] thank you [00:06:05] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:27] can't believe ie7 is still that high :-/ [00:08:32] * saper still has ie7 on vista [00:12:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 7.067 seconds [00:23:42] gn8 folks [00:27:30] New patchset: Lcarr; "Splitting off the icinga packages and specific config files into their own manifest file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3135 [00:27:40] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/3135 [00:28:51] New patchset: Lcarr; "Splitting off the icinga packages and specific config files into their own manifest file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3135 [00:29:03] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3135 [00:29:20] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3135 [00:31:48] Danny_B|backup: and IE6 is hanging on [00:32:40] well, i wanted to get rid of cellspacing and i can't [00:33:26] i don't care about ie6, but ie7 still occupies to much to get rid of cellspacing :-/ [00:36:20] New patchset: Lcarr; "Moving icinga specific files to own class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3136 [00:36:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3136 [00:37:58] Change abandoned: Lcarr; "i hate you gerrit" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3135 [00:38:58] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3136 [00:39:00] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3136 [00:46:37] Can I speak to a sysadmin? [00:46:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:19] New patchset: Lcarr; "explicitly calling out nagios_mysql_check_pass" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3137 [00:47:31] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3137 [00:48:23] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3137 [00:48:26] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3137 [00:52:52] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.545 seconds [00:54:25] New patchset: Lcarr; "fixing out of scope variable" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3138 [00:54:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3138 [00:55:21] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3138 [00:55:23] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3138 [00:58:43] New patchset: Lcarr; "fixing other out of scope variables" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3139 [00:58:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3139 [00:59:44] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3139 [00:59:47] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3139 [01:06:13] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:55] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:13:07] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.859 seconds [01:15:31] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [01:15:31] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [01:15:31] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [01:15:31] PROBLEM - Puppet freshness on amssq32 is CRITICAL: Puppet has not run in the last 10 hours [01:15:31] PROBLEM - Puppet freshness on amssq31 is CRITICAL: Puppet has not run in the last 10 hours [01:18:29] New patchset: Lcarr; "removing unneeded /etc/icinga check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3140 [01:18:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3140 [01:19:05] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3140 [01:19:08] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3140 [01:19:25] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:17] New patchset: Ryan Lane; "Upping svn rev for ldap tools to update manage-volumes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3141 [01:24:29] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3141 [01:24:33] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3141 [01:24:35] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3141 [01:28:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:52] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 7.194 seconds [01:34:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 5.667 seconds [01:43:05] PROBLEM - Misc_Db_Lag on db10 is CRITICAL: (Return code of 255 is out of bounds) [01:43:05] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:43:14] PROBLEM - MySQL slave status on es2 is CRITICAL: CRITICAL: Access denied for user nagios@208.80.152.161 (using password: YES) [01:43:32] PROBLEM - Misc_Db_Slave on db10 is CRITICAL: CRITICAL: Access denied for user nagios@spence.wikimedia.org (using password: YES) [01:43:32] PROBLEM - MySQL replication status on es1003 is CRITICAL: (Return code of 255 is out of bounds) [01:43:32] PROBLEM - MySQL master status on es3 is CRITICAL: CRITICAL: Access denied for user nagios@208.80.152.161 (using password: YES) [01:43:32] PROBLEM - Misc_Db_Master on db9 is CRITICAL: CRITICAL: Access denied for user nagios@spence.wikimedia.org (using password: YES) [01:43:50] PROBLEM - MySQL replication status on es1 is CRITICAL: (Return code of 255 is out of bounds) [01:43:50] PROBLEM - MySQL replication status on es4 is CRITICAL: (Return code of 255 is out of bounds) [01:44:08] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: (Return code of 255 is out of bounds) [01:44:08] PROBLEM - MySQL slave status on es1003 is CRITICAL: CRITICAL: Access denied for user nagios@208.80.152.161 (using password: YES) [01:44:08] PROBLEM - MySQL slave status on es1 is CRITICAL: CRITICAL: Access denied for user nagios@208.80.152.161 (using password: YES) [01:44:08] PROBLEM - MySQL replication status on es1004 is CRITICAL: (Return code of 255 is out of bounds) [01:44:26] PROBLEM - MySQL slave status on es4 is CRITICAL: CRITICAL: Access denied for user nagios@208.80.152.161 (using password: YES) [01:44:26] PROBLEM - MySQL replication status on db1025 is CRITICAL: (Return code of 255 is out of bounds) [01:44:35] PROBLEM - MySQL master status on db1008 is CRITICAL: CRITICAL: Access denied for user nagios@208.80.152.161 (using password: YES) [01:44:44] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.300 seconds [01:44:44] PROBLEM - MySQL slave status on db1025 is CRITICAL: CRITICAL: Access denied for user nagios@208.80.152.161 (using password: YES) [01:44:44] PROBLEM - MySQL replication status on storage3 is CRITICAL: (Return code of 255 is out of bounds) [01:44:53] PROBLEM - MySQL replication status on es2 is CRITICAL: (Return code of 255 is out of bounds) [01:45:02] PROBLEM - MySQL slave status on storage3 is CRITICAL: CRITICAL: Access denied for user nagios@208.80.152.161 (using password: YES) [01:51:02] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:14] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.601 seconds [01:59:53] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.266 seconds [02:01:32] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:11] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:10:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:15:12] New patchset: Lcarr; "Revert "fixing out of scope variable"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3142 [02:15:24] New patchset: Lcarr; "Revert "fixing other out of scope variables"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3143 [02:15:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3142 [02:15:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3143 [02:16:06] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3142 [02:16:09] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3142 [02:16:13] New patchset: Dzahn; "swift process monitoring (RT-2593)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3144 [02:16:25] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3143 [02:16:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3144 [02:16:25] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3143 [02:16:41] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.038 seconds [02:16:41] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.313 seconds [02:17:13] !log LocalisationUpdate completed (1.19) at Wed Mar 14 02:17:13 UTC 2012 [02:17:16] Logged the message, Master [02:17:35] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [02:18:20] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.581 seconds [02:18:47] New review: Dzahn; "expect NRPE to break when merging any change to nrpe_local.cfg - be prepared to restart nagios-nrpe-..." [operations/puppet] (production); V: 1 C: 1; - https://gerrit.wikimedia.org/r/3144 [02:22:59] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:26] New patchset: Lcarr; "fixing the nagios checkcommands template again" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3145 [02:24:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3145 [02:24:38] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:25:06] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3145 [02:25:08] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3145 [02:40:59] PROBLEM - Puppet freshness on virt4 is CRITICAL: Puppet has not run in the last 10 hours [02:44:53] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [02:47:42] RECOVERY - Misc_Db_Lag on db10 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:47:42] RECOVERY - MySQL slave status on es2 is OK: OK: [02:48:00] RECOVERY - Misc_Db_Slave on db10 is OK: OK: [02:48:00] RECOVERY - MySQL master status on es3 is OK: OK: [02:48:00] RECOVERY - Misc_Db_Master on db9 is OK: OK: [02:48:00] RECOVERY - MySQL replication status on es1003 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:48:27] RECOVERY - MySQL replication status on es1 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:48:27] RECOVERY - MySQL replication status on es4 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:48:27] RECOVERY - MySQL slave status on es1003 is OK: OK: [02:48:45] RECOVERY - MySQL slave status on es4 is OK: OK: [02:48:45] RECOVERY - MySQL replication status on es1004 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : s [02:48:45] RECOVERY - MySQL slave status on es1 is OK: OK: [02:48:54] RECOVERY - MySQL replication status on db1025 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:49:03] RECOVERY - MySQL master status on db1008 is OK: OK: [02:49:03] RECOVERY - MySQL slave status on db1025 is OK: OK: [02:49:21] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 11s [02:49:39] RECOVERY - MySQL slave status on storage3 is OK: OK: [02:49:48] RECOVERY - MySQL replication status on es2 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:50:42] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 50s [02:53:51] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.047 second response time [03:01:39] PROBLEM - Puppet freshness on virt2 is CRITICAL: Puppet has not run in the last 10 hours [03:04:59] good night :) [03:10:12] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.862 seconds [03:16:39] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.242 seconds [03:31:12] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:31] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.136 seconds [03:34:43] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Wed Mar 14 03:34:21 UTC 2012 [03:40:07] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:42:04] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:10] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.363 seconds [03:46:34] New patchset: Lcarr; "fixing paths" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3146 [03:46:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3146 [03:47:24] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3146 [03:47:27] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3146 [03:50:19] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.630 seconds [03:56:46] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:43] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 5.447 seconds [04:11:28] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:13:34] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 8.855 seconds [04:45:30] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:49:33] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [05:41:39] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [06:26:57] New patchset: Dzahn; "also allow public esams net (91.198.174.0./25), not just private, snmp access" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3147 [06:27:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3147 [06:28:57] New patchset: Dzahn; "also allow public esams net (91.198.174.0./25), not just private, snmp access" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3147 [06:29:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3147 [06:30:29] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3147 [06:30:32] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3147 [07:17:17] RECOVERY - Puppet freshness on amslvs1 is OK: puppet ran at Wed Mar 14 07:16:44 UTC 2012 [07:17:44] RECOVERY - Puppet freshness on ssl3001 is OK: puppet ran at Wed Mar 14 07:17:28 UTC 2012 [07:17:44] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Wed Mar 14 07:17:33 UTC 2012 [07:18:11] RECOVERY - Puppet freshness on ssl3002 is OK: puppet ran at Wed Mar 14 07:17:54 UTC 2012 [07:18:47] RECOVERY - Puppet freshness on amssq45 is OK: puppet ran at Wed Mar 14 07:18:13 UTC 2012 [07:19:41] RECOVERY - Puppet freshness on amssq35 is OK: puppet ran at Wed Mar 14 07:19:27 UTC 2012 [07:22:50] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [07:24:56] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [07:25:14] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Wed Mar 14 07:24:59 UTC 2012 [07:25:14] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Wed Mar 14 07:25:08 UTC 2012 [07:25:14] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Wed Mar 14 07:25:09 UTC 2012 [07:25:41] RECOVERY - Puppet freshness on amssq34 is OK: puppet ran at Wed Mar 14 07:25:16 UTC 2012 [07:25:41] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Wed Mar 14 07:25:22 UTC 2012 [07:27:47] RECOVERY - Puppet freshness on ssl3003 is OK: puppet ran at Wed Mar 14 07:27:27 UTC 2012 [07:27:47] RECOVERY - Puppet freshness on ssl3004 is OK: puppet ran at Wed Mar 14 07:27:32 UTC 2012 [07:27:56] PROBLEM - Puppet freshness on knsq24 is CRITICAL: Puppet has not run in the last 10 hours [07:27:56] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [07:28:05] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Wed Mar 14 07:27:55 UTC 2012 [07:28:41] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Wed Mar 14 07:28:15 UTC 2012 [07:28:41] RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Wed Mar 14 07:28:19 UTC 2012 [07:29:17] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Wed Mar 14 07:28:46 UTC 2012 [07:29:17] RECOVERY - Puppet freshness on cp3002 is OK: puppet ran at Wed Mar 14 07:28:57 UTC 2012 [07:29:17] RECOVERY - Puppet freshness on amssq33 is OK: puppet ran at Wed Mar 14 07:28:59 UTC 2012 [07:30:11] RECOVERY - Puppet freshness on maerlant is OK: puppet ran at Wed Mar 14 07:29:41 UTC 2012 [07:30:47] RECOVERY - Puppet freshness on amssq37 is OK: puppet ran at Wed Mar 14 07:30:14 UTC 2012 [07:30:47] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Wed Mar 14 07:30:32 UTC 2012 [07:31:41] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Wed Mar 14 07:31:31 UTC 2012 [07:31:50] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [07:31:50] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [07:32:17] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Wed Mar 14 07:32:07 UTC 2012 [07:34:41] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Wed Mar 14 07:34:14 UTC 2012 [07:34:41] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Wed Mar 14 07:34:27 UTC 2012 [07:35:44] RECOVERY - Puppet freshness on knsq25 is OK: puppet ran at Wed Mar 14 07:35:25 UTC 2012 [07:35:44] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Wed Mar 14 07:35:40 UTC 2012 [07:37:14] RECOVERY - Puppet freshness on nescio is OK: puppet ran at Wed Mar 14 07:36:45 UTC 2012 [07:40:14] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Wed Mar 14 07:40:02 UTC 2012 [07:40:41] RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Wed Mar 14 07:40:18 UTC 2012 [07:40:41] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Wed Mar 14 07:40:40 UTC 2012 [07:40:50] PROBLEM - Puppet freshness on ms-be5 is CRITICAL: Puppet has not run in the last 10 hours [07:41:44] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Wed Mar 14 07:41:30 UTC 2012 [07:41:44] RECOVERY - Puppet freshness on amssq36 is OK: puppet ran at Wed Mar 14 07:41:31 UTC 2012 [07:42:47] RECOVERY - Puppet freshness on amssq39 is OK: puppet ran at Wed Mar 14 07:42:26 UTC 2012 [07:43:14] RECOVERY - Puppet freshness on amssq32 is OK: puppet ran at Wed Mar 14 07:43:04 UTC 2012 [07:43:41] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Wed Mar 14 07:43:16 UTC 2012 [07:43:41] RECOVERY - Puppet freshness on hooft is OK: puppet ran at Wed Mar 14 07:43:26 UTC 2012 [07:45:11] RECOVERY - Puppet freshness on amssq43 is OK: puppet ran at Wed Mar 14 07:44:43 UTC 2012 [07:46:41] RECOVERY - Puppet freshness on amssq31 is OK: puppet ran at Wed Mar 14 07:46:16 UTC 2012 [07:49:50] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [07:58:50] RECOVERY - Memcached on srv254 is OK: TCP OK - 0.002 second response time on port 11000 [07:59:08] RECOVERY - Memcached on srv255 is OK: TCP OK - 0.001 second response time on port 11000 [08:02:26] RECOVERY - Memcached on srv257 is OK: TCP OK - 0.003 second response time on port 11000 [08:24:41] RECOVERY - DPKG on snapshot3 is OK: All packages OK [11:17:28] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [11:41:40] Hello everybody! We are doing R&D for our startup; we are developing a product which will display information retrieved from wikipedia servers through API. [11:42:28] I'd like to get in contact with you to know everything which is needed to know to design the product correctly against wikipedia policy + to know tech limits (e.g. query limits? ) [11:43:00] We also would like to know how it would be possible to set a "partenrship", in which we donate part of the profit to wikipedia foundation [11:43:19] Any help on this? [11:43:43] gg4u: you can use (1) API - registered bots have higher limits (2) download.wikimedia.org (3) toolserver.org has some data but no text of revisions [11:44:04] gg4u: what are you trying to do? [11:44:27] for watching recent changes there are also possibilities of using IRC or RSS [11:47:02] @saper: we are dev a client which "aggregate" information from different sources [11:47:59] saper: we have a sample prototype for movies. Suppose I wanna see the content of blade runner; I'd like to retrieve content form CC licences ad wikipedia [11:48:17] The product is a client for mobile devices. [11:49:21] The same concept could be applied to ANY article in wikipedia [11:50:18] My concern is: is there a limit of query? I want to avoid to duplicate the wikipedia dump on other servers. I just want to read it. [11:55:53] New patchset: Hashar; "gerrit played ping pong between http / https URL" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3148 [11:56:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3148 [11:56:08] ^^ that one is an easy change :-) [12:04:06] PROBLEM - Disk space on search1017 is CRITICAL: DISK CRITICAL - free space: /a 5002 MB (3% inode=99%): [12:04:17] @saper: I don't think it is a matter of bot. Suppose there are 10.000 users with their mobile phones browsing the wiki in the same time, via API through the client we are developing (a mobile app). I want to know how to design the app in order to guarantee people can browse it without delays or "dead queries". [12:08:27] PROBLEM - Disk space on search1017 is CRITICAL: DISK CRITICAL - free space: /a 5000 MB (3% inode=99%): [12:09:23] gg4u: afair https://www.mediawiki.org/wiki/API:FAQ#get_the_content_of_a_page_.28wikitext.29.3F is not ratelimited, not sure if https://www.mediawiki.org/wiki/API:Parsing_wikitext#parse is [12:15:50] gg4u: if you feel you need some form of business relationship with the Wikimedia Foundation (I'm not sure), http://wikimediafoundation.org/wiki/User:Kul is the guy to talk to [12:23:15] saper, thank you for both the info [12:23:26] saper, on the first link I read [12:23:28] You can retrieve 50 pages per API request: http://en.wikipedia.org/w/api.php?action=query&prop=revisions&rvprop=content&titles=Main_Page%7CArticles This also works with generators. [12:24:50] does this "50 pages per API" refer to the mobile client or our server which fires the pre-built queries for each of the users' clients? [12:25:27] Also, if api are limited, how do mobile app for browsing wikipiedia works ? [12:33:21] gg4u: can you check this? http://news.gmane.org/gmane.science.linguistics.wikipedia.technical there is a lot of discussion on this recently, including giving mobile interface to all mediawiki installations, not only those on the WMF cluster. [12:43:14] PROBLEM - Puppet freshness on virt4 is CRITICAL: Puppet has not run in the last 10 hours [12:46:32] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [12:54:22] @saper: tks. plz help me understand correctly: I browsed and found this [12:54:31] http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/18313/match=limit+query [12:55:07] What does "Just implement user key webservices API with query limits, and you're done." mean exactly? [12:56:17] From my understanding, I can skip an intermediary host server, but should I obtain a specific user key for any user downloading a mobile client-side app ? [12:58:13] The api should call for a page to each "click" of the user in mobile client. Is there maybe any material/documentation t see what other app developers did for browsing wikipedia from mobiles? (I don't really think they set up a host, I am pretty sure they just rely on wikipedia servers) [13:03:29] PROBLEM - Puppet freshness on virt2 is CRITICAL: Puppet has not run in the last 10 hours [13:06:11] New patchset: Hashar; "publicly list mediawiki extensions git repo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3149 [13:06:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3149 [13:10:32] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:59] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:42] gg4u: sorry, I don't know how mobile is done currently. There are no API "key", you may have a bot account for particular project that bumps up the limits. [13:12:50] gg4u: if you plan to be push edits as well it will be interesting to know the real IP address of the editor [13:28:59] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:38:58] New patchset: Hashar; "publicly list mediawiki extensions git repo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3149 [13:39:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3149 [13:39:20] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [13:39:37] New review: Hashar; "That second patch set makes the change no more dependent on another pending one." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/3149 [13:39:56] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 4.174 seconds [13:40:26] @saper: ok saper, I'll try to learn more on bots. we don't plan to include edit. [13:40:32] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.182 seconds [13:40:33] is there a channel where to learn more on mobile? [13:41:19] gg4u: I guess with pure browsing you could be fine with action=render or something [13:42:03] http://thread.gmane.org/gmane.science.linguistics.wikipedia.technical probably (even last post has most interesting links) [13:44:10] gg4u: you might also go browser through Foundaton's powerpointware which you migh find on meta.wikimedia.org (or maybe older stuff on strategy.wikimedia.org) and find pointers there, https://meta.wikimedia.org/wiki/Mobile_Projects https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2012/February etc. [13:46:33] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:50] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:19] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.178 seconds [13:59:37] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:46] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.504 seconds [14:06:13] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:21] 9~wj [14:07:34] New review: RobH; "Someone fixing apache host files, huzzah!" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3148 [14:07:36] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3148 [14:15:18] New review: Hashar; "Yeah one less redirect!!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3148 [14:33:31] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.087 seconds [14:46:07] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:13] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 7.621 seconds [14:54:22] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.960 seconds [15:00:58] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:07] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:34] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 9.45025310924 (gt 8.0) [15:04:46] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.883 seconds [15:11:04] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:28:19] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.683 seconds [15:34:10] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.766 seconds [15:34:37] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:38:04] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 14.3817131667 (gt 8.0) [15:43:28] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [15:46:46] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:11] !log reedy synchronized wmf-config/CommonSettings.php 'add vcs for extdist updates' [16:04:15] Logged the message, Master [16:17:55] etherpad has problems when you're connected via https [16:18:02] it keeps disconnecting [16:21:23] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.181 seconds [16:27:59] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:23] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.722 seconds [16:40:44] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:29] PROBLEM - Host ms-be5 is DOWN: PING CRITICAL - Packet loss = 100% [16:47:38] RECOVERY - Host ms-be5 is UP: PING OK - Packet loss = 0%, RTA = 1.28 ms [16:47:56] RECOVERY - DPKG on ms-be5 is OK: All packages OK [16:48:14] RECOVERY - Disk space on ms-be5 is OK: DISK OK [16:48:50] RECOVERY - RAID on ms-be5 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [16:50:56] RECOVERY - Puppet freshness on ms-be5 is OK: puppet ran at Wed Mar 14 16:50:46 UTC 2012 [17:11:29] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.3287510084 [17:17:45] @saper: thank you very much for your help! [17:18:41] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.751 seconds [17:24:05] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [17:25:08] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:44] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 9.79999766667 (gt 8.0) [17:26:02] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [17:31:53] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.57810478992 [17:33:05] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [17:33:05] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [17:34:01] PROBLEM - Host ms-be5 is DOWN: PING CRITICAL - Packet loss = 100% [17:34:46] New patchset: Lcarr; "Allowing icinga in sudoers as with nagios + group gammu" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3153 [17:34:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3153 [17:35:40] RECOVERY - Host ms-be5 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [17:35:46] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3153 [17:35:48] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3153 [17:40:46] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 10.6636874167 (gt 8.0) [17:42:52] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.85144512605 [17:49:55] New patchset: Bhartshorne; "bumping up the number of replicator processes running on swift storage bricks to improve time to balance" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3154 [17:50:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3154 [17:50:55] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3154 [17:50:58] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3154 [17:51:43] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [18:16:26] New patchset: Lcarr; "pushing http to http /icinga and https to https /icinga" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3155 [18:16:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3155 [18:17:11] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3155 [18:17:14] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3155 [18:18:07] PROBLEM - Swift HTTP on zinc is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:12] New patchset: Bhartshorne; "dropping down to 2 from 4. put latency increased too much." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3156 [18:21:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3156 [18:21:33] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3156 [18:21:36] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3156 [18:23:13] PROBLEM - Swift HTTP on magnesium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:49] PROBLEM - Swift HTTP on copper is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:16] PROBLEM - Swift HTTP on zinc is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:43] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.623 seconds [18:26:13] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 9.165386 (gt 8.0) [18:31:01] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:14] !log preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing' [18:31:17] Logged the message, Master [18:31:26] !log push zero change for carrier testing [18:31:29] Logged the message, Master [18:32:31] PROBLEM - Swift HTTP on zinc is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:49] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 6.601 seconds [18:36:34] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 20.0373309167 (gt 8.0) [18:39:56] Is there any operative edit counter? [18:41:24] New patchset: Asher; "db18,19 decom" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3157 [18:41:36] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3157 [18:42:16] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:52] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.51411680672 [18:49:37] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.532 seconds [18:55:15] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:54] New patchset: Lcarr; "making sure conf.d exists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3158 [18:57:07] New patchset: Lcarr; "more icinga tweaks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3159 [18:57:12] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 8.112 seconds [18:57:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3158 [18:57:19] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3158 [18:57:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3159 [18:57:20] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3158 [18:57:35] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3159 [18:57:37] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3159 [19:01:42] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.146 seconds [19:03:57] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:15] !log preilly synchronized php-1.19/extensions/MobileFrontend/MobileFrontend.body.php 'zero needs to add x-images to vary header' [19:05:18] Logged the message, Master [19:06:03] !log preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging' [19:06:07] Logged the message, Master [19:06:14] !log pushing x-images header for vary support [19:06:17] Logged the message, Master [19:06:30] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CRIT replication delay 335 seconds [19:06:39] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 344 seconds [19:15:48] PROBLEM - DPKG on db58 is CRITICAL: Connection refused by host [19:16:15] PROBLEM - Disk space on db58 is CRITICAL: Connection refused by host [19:16:33] PROBLEM - MySQL disk space on db58 is CRITICAL: Connection refused by host [19:17:27] PROBLEM - RAID on db58 is CRITICAL: Connection refused by host [19:17:36] PROBLEM - SSH on db58 is CRITICAL: Connection refused [19:18:21] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:39] RECOVERY - RAID on db58 is OK: OK: State is Optimal, checked 12 logical device(s) [19:21:48] RECOVERY - SSH on db58 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [19:22:15] RECOVERY - DPKG on db58 is OK: All packages OK [19:22:33] RECOVERY - Disk space on db58 is OK: DISK OK [19:22:51] RECOVERY - MySQL disk space on db58 is OK: DISK OK [19:28:15] PROBLEM - Swift HTTP on magnesium is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:57] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.885 seconds [19:30:57] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 6.729 seconds [19:32:27] PROBLEM - Host ms-be1 is DOWN: PING CRITICAL - Packet loss = 100% [19:37:15] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:37:15] RECOVERY - Host ms-be1 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [19:37:24] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:20] New patchset: Lcarr; "more icinga fixes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3160 [19:45:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3160 [19:46:31] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3160 [19:46:34] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3160 [19:47:01] PROBLEM - Swift HTTP on copper is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:27] PROBLEM - Swift HTTP on zinc is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:47:36] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 8.531 seconds [19:47:36] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.137 seconds [19:53:54] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:55:46] New patchset: Lcarr; "another apache updated" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3161 [19:55:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3161 [19:56:00] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3161 [19:56:02] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3161 [20:00:03] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:10] New patchset: Lcarr; "making check all spelled out for purging nagios resources" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3162 [20:06:18] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/3162 [20:07:06] New patchset: Lcarr; "making check all spelled out for purging nagios resources" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3162 [20:07:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3162 [20:07:26] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3162 [20:07:29] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3162 [20:08:09] PROBLEM - Swift HTTP on zinc is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:39] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.953 seconds [20:17:38] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:32] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.447 seconds [20:21:50] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.018 seconds [20:24:42] New patchset: Lcarr; "Remove default icinga conf file from apache" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3163 [20:24:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3163 [20:25:20] New patchset: Lcarr; "Remove default icinga conf file from apache" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3163 [20:25:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3163 [20:25:42] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3163 [20:25:45] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3163 [20:27:01] (Cannot contact the database server: Unknown error (10.0.6.48)) [20:27:10] while submitting an edit (not even a Wikimedia error) [20:29:51] !r 112300 [20:29:51] https://www.mediawiki.org/wiki/Special:Code/MediaWiki/112300 [20:30:00] !log preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging' [20:30:04] Logged the message, Master [20:30:32] PROBLEM - MySQL Replication Heartbeat on db12 is CRITICAL: CRIT replication delay 186 seconds [20:31:08] PROBLEM - MySQL Replication Heartbeat on db1017 is CRITICAL: CRIT replication delay 221 seconds [20:31:08] PROBLEM - MySQL Replication Heartbeat on db36 is CRITICAL: CRIT replication delay 221 seconds [20:31:08] PROBLEM - MySQL Replication Heartbeat on db1043 is CRITICAL: CRIT replication delay 221 seconds [20:31:08] PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 221 seconds [20:31:26] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 240 seconds [20:31:53] PROBLEM - MySQL Replication Heartbeat on db32 is CRITICAL: CRIT replication delay 265 seconds [20:31:53] PROBLEM - MySQL Replication Heartbeat on db52 is CRITICAL: CRIT replication delay 265 seconds [20:32:11] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 285 seconds [20:33:00] !log preilly synchronized php-1.19/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'changes for zero needed for carrier testing header for image support debugging' [20:33:03] Logged the message, Master [20:33:59] PROBLEM - MySQL Replication Heartbeat on db38 is CRITICAL: CRIT replication delay 391 seconds [20:34:17] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:34:44] RECOVERY - MySQL Replication Heartbeat on db12 is OK: OK replication delay 0 seconds [20:35:11] RECOVERY - MySQL Replication Heartbeat on db1017 is OK: OK replication delay 0 seconds [20:35:11] RECOVERY - MySQL Replication Heartbeat on db36 is OK: OK replication delay 0 seconds [20:35:11] RECOVERY - MySQL Replication Heartbeat on db1043 is OK: OK replication delay 0 seconds [20:35:11] RECOVERY - MySQL Replication Heartbeat on db1033 is OK: OK replication delay 0 seconds [20:35:38] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 0 seconds [20:35:56] RECOVERY - MySQL Replication Heartbeat on db52 is OK: OK replication delay 0 seconds [20:35:56] RECOVERY - MySQL Replication Heartbeat on db32 is OK: OK replication delay 0 seconds [20:35:56] RECOVERY - MySQL Replication Heartbeat on db38 is OK: OK replication delay 0 seconds [20:36:14] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.614 seconds [20:36:23] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 0 seconds [20:36:23] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:39:17] New patchset: Asher; "disabling log_queries_not_using_indexes for now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3164 [20:39:29] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3164 [20:41:59] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3130 [20:42:01] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3130 [20:42:18] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3157 [20:42:21] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3157 [20:42:32] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:29] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 6.117 seconds [20:45:29] New patchset: Asher; "disabling log_queries_not_using_indexes for now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3164 [20:45:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3164 [20:46:01] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3164 [20:46:04] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3164 [20:50:56] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:42] New review: Ryan Lane; "Let's make this a cron that generates a static file, so that we don't need to include php on the ger..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/3149 [20:57:14] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.924 seconds [21:03:23] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 2.372 seconds [21:09:32] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:09:41] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:13:06] New patchset: Hashar; "publicly list mediawiki extensions git repo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3149 [21:13:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3149 [21:18:50] New patchset: Hashar; "publicly list mediawiki extensions git repo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3149 [21:18:59] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [21:19:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3149 [21:20:51] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3149 [21:20:54] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3149 [21:24:05] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.406 seconds [21:24:05] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.122 seconds [21:36:32] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:36:32] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:38:46] New patchset: Ryan Lane; "Upping tools again" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3165 [21:38:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3165 [21:40:06] New patchset: Lcarr; "fixing ordering for icinga" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3166 [21:40:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3166 [21:40:49] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3166 [21:40:52] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3166 [21:40:53] New patchset: Ryan Lane; "Adding docroot for 443 on gerrit" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3167 [21:41:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3167 [21:41:10] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3167 [21:41:25] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3165 [21:41:27] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3167 [21:41:29] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3165 [21:46:22] New patchset: Ryan Lane; "Changing ls one-liner to only output extension names, rather than directory contents" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3168 [21:46:34] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3168 [21:47:27] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3168 [21:47:29] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3168 [21:49:27] New patchset: Bhartshorne; "trying to puppetize a cronjob to run the swift cleaner changed swiftcleanermanager to only allow one instance at a time (using a pidfile)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3134 [21:49:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3134 [21:52:10] New patchset: Lcarr; "removing icinga default conf" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3169 [21:52:22] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3169 [21:52:27] who can delete pages with more than 5000 revisions? even bureaucrats can't now... [21:52:40] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/3169 [21:52:43] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3169 [21:53:37] New patchset: Bhartshorne; "trying to puppetize a cronjob to run the swift cleaner changed swiftcleanermanager to only allow one instance at a time (using a pidfile)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3134 [21:53:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3134 [21:54:28] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.226 seconds [21:57:18] The mobile site is indexed by Google? [21:58:47] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3134 [21:58:49] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3134 [22:00:55] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:33] Joan: don't believe so - did you see it indexed ? [22:02:24] New patchset: Bhartshorne; "installing the swift cleaner on iron" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3170 [22:02:36] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3170 [22:02:40] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3170 [22:02:43] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3170 [22:09:10] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 6.817 seconds [22:09:19] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 7.094 seconds [22:10:37] New patchset: Bhartshorne; "correcting variable scope in template file, correcting path to conf file in cron invocation." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3171 [22:10:43] LeslieCarr: It came up in a Google Alert. And there's no code preventing it from being indexed. [22:10:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3171 [22:10:54] LeslieCarr: I guess I'll file a bug. [22:11:08] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3171 [22:11:10] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3171 [22:14:14] LeslieCarr: https://bugzilla.wikimedia.org/show_bug.cgi?id=35233 [22:21:29] New patchset: Bhartshorne; "trying with local scope" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3172 [22:21:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3172 [22:22:20] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3172 [22:22:22] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3172 [22:38:25] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:38:34] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:37] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.622 seconds [22:44:25] PROBLEM - Puppet freshness on virt4 is CRITICAL: Puppet has not run in the last 10 hours [22:48:28] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [22:48:46] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:43] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 5.565 seconds [22:51:01] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 7.418 seconds [22:55:40] New patchset: Bhartshorne; "added option to ignore previous state when running swiftcleaner, fixed bug in pidfile detection" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3173 [22:55:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3173 [22:56:19] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3173 [22:56:22] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3173 [22:57:10] PROBLEM - Swift HTTP on zinc is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:57:10] PROBLEM - Swift HTTP on copper is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:25] PROBLEM - Puppet freshness on virt2 is CRITICAL: Puppet has not run in the last 10 hours [23:11:17] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:11:35] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:23] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.334 seconds [23:13:32] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.164 seconds [23:14:17] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [23:14:26] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 0 seconds [23:22:14] PROBLEM - Swift HTTP on copper is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:14] PROBLEM - Swift HTTP on zinc is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:50] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:08] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:34:05] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.726 seconds [23:37:50] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 0.003 seconds [23:42:38] PROBLEM - Swift HTTP on copper is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:20] New patchset: Bhartshorne; "correcting scrubstate logic" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3177 [23:45:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/3177 [23:46:31] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/3177 [23:46:33] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/3177 [23:48:29] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:48:29] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:50:26] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 6.631 seconds [23:50:26] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 6.636 seconds