[00:10:13] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:38] :q [00:25:58] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.019 seconds [00:40:58] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [00:57:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.025 seconds [01:20:34] New patchset: Jdlrobson; "provide e-mail context for other forms of feedback other than technical" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/26920 [01:41:25] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 249 seconds [01:41:34] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 202 seconds [01:47:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:22] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 12 seconds [01:53:43] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 2 seconds [02:01:49] PROBLEM - BGP status on cr2-eqiad is CRITICAL: CRITICAL: No response from remote host 208.80.154.197, [02:02:47] !log LocalisationUpdate failed: git pull of extensions failed [02:02:59] Logged the message, Master [02:03:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.032 seconds [02:03:55] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [02:05:19] gah [02:12:07] looks like localisation update died during updating the extensions.. [02:17:24] !log Manually re-running localisation update [02:17:36] Logged the message, Master [02:36:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.041 seconds [02:39:31] !log LocalisationUpdate completed (1.21wmf1) at Sat Oct 6 02:39:31 UTC 2012 [02:39:42] Logged the message, Master [02:40:19] good [03:11:10] !log LocalisationUpdate completed (1.20wmf12) at Sat Oct 6 03:11:10 UTC 2012 [03:11:20] Logged the message, Master [03:25:58] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [03:25:58] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [03:28:58] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [05:18:10] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sat Oct 6 05:18:01 UTC 2012 [07:52:07] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [07:52:07] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [07:52:07] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [07:52:07] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [07:52:07] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [08:27:04] PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours [08:42:13] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [09:52:59] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [10:41:53] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [10:57:38] RECOVERY - check_job_queue on spence is OK: JOBQUEUE OK - all job queues below 10,000 [10:58:05] RECOVERY - check_job_queue on neon is OK: JOBQUEUE OK - all job queues below 10,000 [12:05:19] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [12:40:57] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.426 seconds [13:18:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.209 seconds [13:27:18] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [13:27:18] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [14:00:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.287 seconds [14:14:35] !log lcarr synchronized wmf-config/throttle.php 'I updated a throttle' [14:14:47] Logged the message, Master [14:15:24] Reedy: do you happen to be around ? :) [14:20:42] LeslieCarr: you want to clear the memcache key? ;) [14:20:47] jeremyb: psychic [14:20:52] yay! [14:22:19] lol [14:22:38] 04 21:40:16 < Reedy> enwiki:acctcreate:ip:12.183.19.7 [14:22:39] 04 21:40:20 < Reedy> commonswiki:acctcreate:ip:12.183.19.7 [14:22:40] commit 9ccdd75cef19b0fa205ab5c1e8685c1c24505119 [14:22:42] Author: Leslie Carr [14:22:43] that's the format [14:22:50] hahaha [14:22:54] That doesn't look right, isn't ttijhof Krinkle? [14:23:20] well LeslieCarr spends a lot of time in Krinkle's native land... [14:23:27] which land is that ? [14:23:31] nl [14:24:15] oh, then yes i do [14:24:49] speaking of which: met some people from Groningen last night and they were quite surprised when I asked if that was near friesland. ;-P [14:25:22] LeslieCarr: anyway, do you know how to clear keys? [14:25:29] chad and i figured it out [14:25:30] :) [14:25:41] cool ;) [14:25:55] * jeremyb wonders where these events are [14:26:04] grace hopper in baltimore! [14:26:08] you missed me talking :) [14:26:13] ahh! [14:26:24] video'd? [14:26:30] i don't think so [14:26:31] :( [14:26:40] ;( [14:28:00] LeslieCarr: did you do the whole 4 days? did we get a booth? [14:28:10] we were part of the free and open source booth [14:28:14] * jeremyb wonders about the India grace hopper too [14:28:48] * Damianz wonders if LeslieCarr is free and opensource and if so where he can download her source code [14:29:03] and i was going to be doing the 4 days , long long story short, plane diverted to pittsburgh, very few hotels and no taxis, got 3 hours of sleep at a shitty motel with bedbugs while sharing the room, slept wednesday instead of doing conference stuff [14:29:08] * LeslieCarr rolls her eyes at Damianz [14:29:09] * jeremyb refers Damianz to cory doctorow's latest book [14:29:32] LeslieCarr: ewww [14:29:49] yeah, the wonders of travel [14:29:56] lol [14:48:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:31] hello ehashman [14:51:40] hello LeslieCarr [15:04:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.042 seconds [15:13:30] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 190 seconds [15:14:06] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 216 seconds [15:15:45] PROBLEM - MySQL Slave Delay on db1020 is CRITICAL: CRIT replication delay 220 seconds [15:16:03] PROBLEM - MySQL Replication Heartbeat on db1020 is CRITICAL: CRIT replication delay 231 seconds [15:16:30] PROBLEM - MySQL Replication Heartbeat on db33 is CRITICAL: CRIT replication delay 198 seconds [15:17:24] PROBLEM - MySQL Slave Delay on db33 is CRITICAL: CRIT replication delay 227 seconds [15:17:42] RECOVERY - Puppet freshness on virt1001 is OK: puppet ran at Sat Oct 6 15:17:26 UTC 2012 [15:19:39] RECOVERY - Puppet freshness on virt1002 is OK: puppet ran at Sat Oct 6 15:19:25 UTC 2012 [15:21:09] RECOVERY - MySQL Replication Heartbeat on db33 is OK: OK replication delay 1 seconds [15:22:03] RECOVERY - MySQL Slave Delay on db33 is OK: OK replication delay 0 seconds [15:23:51] RECOVERY - MySQL Replication Heartbeat on db1020 is OK: OK replication delay 0 seconds [15:24:54] RECOVERY - MySQL Slave Delay on db1020 is OK: OK replication delay 0 seconds [15:30:27] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [15:31:12] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [15:36:04] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:06] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.035 seconds [15:56:23] !log lcarr synchronized wmf-config/throttle.php 'I updated a throttle' [15:56:34] Logged the message, Master [16:24:57] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:03] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [16:32:27] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 1.57 ms [16:33:21] LeslieCarr: how goes the open sourcing? progress?! [16:37:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.536 seconds [17:11:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:10] ehashman: yes, we have opened all the source [17:27:08] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.504 seconds [17:29:28] schweet [17:52:47] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [17:52:47] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [17:52:47] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [18:00:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:29] PROBLEM - Host search32 is DOWN: PING CRITICAL - Packet loss = 100% [18:04:56] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [18:16:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.029 seconds [18:27:44] PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours [18:44:39] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [18:49:00] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.888 seconds [19:15:46] New patchset: DamianZaremba; "Updated pdns related manifests/templates to support an array of ldap hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/26956 [19:16:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/26956 [19:28:03] New patchset: Ryan Lane; "Updated pdns related manifests/templates to support an array of ldap hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/26956 [19:29:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/26956 [19:32:07] New patchset: Demon; "Use literal booleans rather than quoted booleans" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/26957 [19:33:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/26957 [19:35:20] New patchset: DamianZaremba; "Updated pdns related manifests/templates to support an array of ldap hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/26956 [19:36:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/26956 [19:36:22] New review: DamianZaremba; ":)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/26956 [19:37:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:37:46] New review: Demon; "Works on labs ;-)" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/26957 [20:26:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.029 seconds [20:43:05] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [21:15:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:38] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 219 seconds [21:27:47] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 226 seconds [21:30:56] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.036 seconds [21:33:47] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [21:33:56] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [22:03:56] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:11] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [22:19:41] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.056 seconds [22:52:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.058 seconds [23:28:05] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [23:28:05] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [23:41:08] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.029 seconds