[00:00:14] RECOVERY - MySQL Replication Heartbeat on db1046 is OK: OK replication delay 0 seconds [00:10:44] PROBLEM - Puppet freshness on hydrogen is CRITICAL: Puppet has not run in the last 10 hours [01:41:51] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 243 seconds [01:42:36] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 251 seconds [01:49:21] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 657s [01:51:18] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [01:54:45] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 25 seconds [01:54:45] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 26s [02:41:43] RECOVERY - Puppet freshness on hydrogen is OK: puppet ran at Sat Jul 21 02:41:36 UTC 2012 [02:43:40] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Sat Jul 21 02:43:08 UTC 2012 [03:27:10] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [06:58:12] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [07:22:12] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [08:57:57] New patchset: Hashar; "job runner now supports being run on a specific job type" [operations/debs/wikimedia-job-runner] (master) - https://gerrit.wikimedia.org/r/11610 [08:58:35] New review: Hashar; "Patchset3:" [operations/debs/wikimedia-job-runner] (master) C: 0; - https://gerrit.wikimedia.org/r/11610 [09:11:54] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [09:12:30] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [09:15:21] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [09:38:45] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.032 second response time [09:56:45] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [09:59:23] New patchset: Aaron Schulz; "Update revdelete rights per core change I6b69919a1bdc502becc1ae4ac9169b8b0e85bfdc." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16217 [09:59:45] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [13:28:10] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [16:59:13] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [17:23:21] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [17:41:30] Change merged: Aaron Schulz; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16217 [17:56:54] working copy: 36M, repository 106M [17:57:43] hmm... down to 81M [18:44:02] Platonides: do you have core +2? I can't remember. [18:50:17] Aaron|home, I have [19:33:45] were the guidelines to (de)assign it defined in the end? [19:57:24] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [20:00:24] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [20:36:42] New patchset: MaxSem; "FeaturedFeeds-related tweaks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16234 [22:57:35] New patchset: Alex Monk; "(bug 38565) Let hewiki admins grant/remove accountcreator" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/16237 [23:28:58] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours