[00:35:22] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [01:41:58] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 285 seconds [01:42:16] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 253 seconds [01:46:19] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 0 seconds [01:49:28] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 16 seconds [02:04:19] New patchset: SPQRobin; "(bug 37413) Closure of vo.wikibooks.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/10755 [02:04:26] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/10755 [02:10:19] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [02:11:01] New review: Peachey88; "Also needs to edit other files (Removing user rights etc) as well I believe for closure to work prop..." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/10755 [02:18:32] New review: SPQRobin; "Yeah, I think that's usually needed, but vowikibooks doesn't have any configuration." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/10755 [02:26:55] New patchset: SPQRobin; "Bring r114112 and r115471 from svn to git (changes to missing.php)." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/10756 [02:27:02] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/10756 [03:09:25] RECOVERY - Puppet freshness on spence is OK: puppet ran at Sun Jun 10 03:08:59 UTC 2012 [05:43:50] PROBLEM - Puppet freshness on es1003 is CRITICAL: Puppet has not run in the last 10 hours [05:43:50] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [05:43:50] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [06:47:01] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [08:52:44] PROBLEM - Puppet freshness on es1004 is CRITICAL: Puppet has not run in the last 10 hours [09:44:15] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [10:36:41] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [12:11:46] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [13:59:12] Ryan_Lane: did we enable IPv6 just for our wikis or other stuff too? [13:59:21] https://bugzilla.wikimedia.org/show_bug.cgi?id=22028 - https://lists.wikimedia.org not reachable over IPv6 [15:44:52] PROBLEM - Puppet freshness on es1003 is CRITICAL: Puppet has not run in the last 10 hours [15:44:52] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [15:44:52] PROBLEM - Puppet freshness on professor is CRITICAL: Puppet has not run in the last 10 hours [16:47:52] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [18:53:28] PROBLEM - Puppet freshness on es1004 is CRITICAL: Puppet has not run in the last 10 hours [19:45:07] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [20:37:56] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [22:12:31] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [23:17:58] PROBLEM - swift-container-auditor on ms-be3 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [23:26:49] RECOVERY - swift-container-auditor on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor