[00:10:22] PROBLEM - Puppet freshness on mw1077 is CRITICAL: Puppet has not run in the last 10 hours [00:12:23] PROBLEM - Puppet freshness on mw1104 is CRITICAL: Puppet has not run in the last 10 hours [00:15:22] PROBLEM - Puppet freshness on mw1043 is CRITICAL: Puppet has not run in the last 10 hours [00:16:22] PROBLEM - Puppet freshness on mw1080 is CRITICAL: Puppet has not run in the last 10 hours [00:18:23] PROBLEM - Puppet freshness on mw1003 is CRITICAL: Puppet has not run in the last 10 hours [00:21:22] PROBLEM - Puppet freshness on virt5 is CRITICAL: Puppet has not run in the last 10 hours [00:22:22] PROBLEM - Puppet freshness on mw1089 is CRITICAL: Puppet has not run in the last 10 hours [00:23:22] PROBLEM - Puppet freshness on mw1129 is CRITICAL: Puppet has not run in the last 10 hours [00:26:24] PROBLEM - Puppet freshness on mw1141 is CRITICAL: Puppet has not run in the last 10 hours [00:30:22] PROBLEM - Puppet freshness on gadolinium is CRITICAL: Puppet has not run in the last 10 hours [00:34:23] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [00:34:23] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [00:34:23] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [00:34:23] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [00:34:23] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [00:38:31] is gerrit allive? [00:38:44] git-review -d 54654 is hanging on core\ [00:39:03] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [00:41:40] git pull is hanging too... he's dead, Jim! [00:42:59] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -2.7027 [00:50:26] hello? ops? [01:04:50] New patchset: Rangilo Gujarati; "Added Edward Baker to English Planet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55500 [01:32:16] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [01:32:16] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [01:32:17] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [01:32:17] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [01:35:06] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [01:39:06] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.44828 [01:53:06] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [01:59:04] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.44828 [02:08:01] !log LocalisationUpdate completed (1.21wmf12) at Sun Mar 24 02:08:00 UTC 2013 [02:08:08] Logged the message, Master [02:11:24] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: Puppet has not run in the last 10 hours [02:11:24] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [02:11:24] PROBLEM - Puppet freshness on cerium is CRITICAL: Puppet has not run in the last 10 hours [02:11:24] PROBLEM - Puppet freshness on cp1028 is CRITICAL: Puppet has not run in the last 10 hours [02:11:24] PROBLEM - Puppet freshness on cp3003 is CRITICAL: Puppet has not run in the last 10 hours [02:11:24] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [02:11:24] PROBLEM - Puppet freshness on cp3010 is CRITICAL: Puppet has not run in the last 10 hours [02:14:22] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [02:16:42] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [02:17:42] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [02:19:23] PROBLEM - Puppet freshness on mw1093 is CRITICAL: Puppet has not run in the last 10 hours [02:29:22] PROBLEM - Puppet freshness on db1049 is CRITICAL: Puppet has not run in the last 10 hours [02:31:22] PROBLEM - Puppet freshness on mw1065 is CRITICAL: Puppet has not run in the last 10 hours [02:31:22] !log LocalisationUpdate completed (1.21wmf11) at Sun Mar 24 02:31:22 UTC 2013 [02:31:36] Logged the message, Master [02:33:03] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [02:37:05] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.33333 [02:55:02] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [03:07:05] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -2.0 [03:34:55] good afternoon (Sunday AEST) [03:39:21] PROBLEM - Puppet freshness on constable is CRITICAL: Puppet has not run in the last 10 hours [03:43:21] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [04:17:21] PROBLEM - Puppet freshness on mw1158 is CRITICAL: Puppet has not run in the last 10 hours [04:17:21] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [04:21:21] PROBLEM - Puppet freshness on mw1118 is CRITICAL: Puppet has not run in the last 10 hours [04:27:21] PROBLEM - Puppet freshness on mw1085 is CRITICAL: Puppet has not run in the last 10 hours [04:29:23] PROBLEM - Puppet freshness on cp1034 is CRITICAL: Puppet has not run in the last 10 hours [04:44:23] PROBLEM - Puppet freshness on mw35 is CRITICAL: Puppet has not run in the last 10 hours [04:55:03] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [05:23:10] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.7037 [05:31:58] PROBLEM - SSH on lvs5 is CRITICAL: Server answer: [05:32:58] RECOVERY - SSH on lvs5 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [05:38:58] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [05:43:03] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.7037 [06:07:01] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [06:12:59] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.84615 [06:13:26] PROBLEM - Puppet freshness on mw44 is CRITICAL: Puppet has not run in the last 10 hours [06:13:54] New review: Ori.livneh; "This change is continuing to see reviewer activity, so I'm leaving it open. If it's not viable, aban..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50306 [06:15:15] PROBLEM - Puppet freshness on mw1092 is CRITICAL: Puppet has not run in the last 10 hours [06:16:15] PROBLEM - Puppet freshness on colby is CRITICAL: Puppet has not run in the last 10 hours [06:21:18] PROBLEM - Puppet freshness on mw62 is CRITICAL: Puppet has not run in the last 10 hours [06:29:05] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [06:33:01] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is 0.0 [07:09:06] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [07:13:05] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.57143 [07:44:47] PROBLEM - SSH on lvs6 is CRITICAL: Server answer: [07:45:48] RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [07:49:16] PROBLEM - Puppet freshness on db67 is CRITICAL: Puppet has not run in the last 10 hours [07:50:16] PROBLEM - Puppet freshness on ms-fe2 is CRITICAL: Puppet has not run in the last 10 hours [07:52:48] PROBLEM - SSH on lvs6 is CRITICAL: Server answer: [07:53:46] RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [08:11:18] PROBLEM - Puppet freshness on mw1126 is CRITICAL: Puppet has not run in the last 10 hours [08:12:18] PROBLEM - Puppet freshness on mw1099 is CRITICAL: Puppet has not run in the last 10 hours [08:14:16] PROBLEM - Puppet freshness on mw1152 is CRITICAL: Puppet has not run in the last 10 hours [08:14:16] PROBLEM - Puppet freshness on mw69 is CRITICAL: Puppet has not run in the last 10 hours [08:15:16] PROBLEM - Puppet freshness on mw1001 is CRITICAL: Puppet has not run in the last 10 hours [08:15:16] PROBLEM - Puppet freshness on mw1103 is CRITICAL: Puppet has not run in the last 10 hours [08:16:16] PROBLEM - Puppet freshness on mw1025 is CRITICAL: Puppet has not run in the last 10 hours [08:16:16] PROBLEM - Puppet freshness on mw1109 is CRITICAL: Puppet has not run in the last 10 hours [08:17:16] PROBLEM - Puppet freshness on grosley is CRITICAL: Puppet has not run in the last 10 hours [08:17:16] PROBLEM - Puppet freshness on locke is CRITICAL: Puppet has not run in the last 10 hours [08:17:16] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [08:17:16] PROBLEM - Puppet freshness on mw1116 is CRITICAL: Puppet has not run in the last 10 hours [08:17:16] PROBLEM - Puppet freshness on mw1155 is CRITICAL: Puppet has not run in the last 10 hours [08:17:16] PROBLEM - Puppet freshness on tola is CRITICAL: Puppet has not run in the last 10 hours [08:19:16] PROBLEM - Puppet freshness on hooft is CRITICAL: Puppet has not run in the last 10 hours [08:19:16] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [08:19:16] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [08:20:16] PROBLEM - Puppet freshness on mw1012 is CRITICAL: Puppet has not run in the last 10 hours [08:20:16] PROBLEM - Puppet freshness on mw48 is CRITICAL: Puppet has not run in the last 10 hours [08:21:17] PROBLEM - Puppet freshness on mw1063 is CRITICAL: Puppet has not run in the last 10 hours [08:33:16] PROBLEM - Puppet freshness on cp3022 is CRITICAL: Puppet has not run in the last 10 hours [09:06:16] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [10:11:16] PROBLEM - Puppet freshness on mw1077 is CRITICAL: Puppet has not run in the last 10 hours [10:13:16] PROBLEM - Puppet freshness on mw1104 is CRITICAL: Puppet has not run in the last 10 hours [10:16:16] PROBLEM - Puppet freshness on mw1043 is CRITICAL: Puppet has not run in the last 10 hours [10:17:06] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [10:17:18] PROBLEM - Puppet freshness on mw1080 is CRITICAL: Puppet has not run in the last 10 hours [10:19:16] PROBLEM - Puppet freshness on mw1003 is CRITICAL: Puppet has not run in the last 10 hours [10:22:17] PROBLEM - Puppet freshness on virt5 is CRITICAL: Puppet has not run in the last 10 hours [10:23:16] PROBLEM - Puppet freshness on mw1089 is CRITICAL: Puppet has not run in the last 10 hours [10:24:17] PROBLEM - Puppet freshness on mw1129 is CRITICAL: Puppet has not run in the last 10 hours [10:27:07] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -2.27273 [10:27:35] PROBLEM - Puppet freshness on mw1141 is CRITICAL: Puppet has not run in the last 10 hours [10:31:15] PROBLEM - Puppet freshness on gadolinium is CRITICAL: Puppet has not run in the last 10 hours [10:34:16] PROBLEM - SSH on searchidx1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:35:15] RECOVERY - SSH on searchidx1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [10:35:16] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [10:35:16] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [10:35:16] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [10:35:16] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [10:35:16] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: Puppet has not run in the last 10 hours [10:35:16] PROBLEM - Puppet freshness on analytics1003 is CRITICAL: Puppet has not run in the last 10 hours [11:23:05] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [11:27:05] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.44828 [11:33:15] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [11:33:15] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [11:33:15] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [11:33:15] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [12:12:18] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: Puppet has not run in the last 10 hours [12:12:18] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [12:12:18] PROBLEM - Puppet freshness on cp1028 is CRITICAL: Puppet has not run in the last 10 hours [12:12:18] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [12:12:18] PROBLEM - Puppet freshness on cerium is CRITICAL: Puppet has not run in the last 10 hours [12:15:35] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [12:20:14] PROBLEM - Puppet freshness on mw1093 is CRITICAL: Puppet has not run in the last 10 hours [12:30:15] PROBLEM - Puppet freshness on db1049 is CRITICAL: Puppet has not run in the last 10 hours [12:32:24] PROBLEM - Puppet freshness on mw1065 is CRITICAL: Puppet has not run in the last 10 hours [13:40:14] PROBLEM - Puppet freshness on constable is CRITICAL: Puppet has not run in the last 10 hours [13:44:14] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [14:18:14] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [14:18:14] PROBLEM - Puppet freshness on mw1158 is CRITICAL: Puppet has not run in the last 10 hours [14:22:14] PROBLEM - Puppet freshness on mw1118 is CRITICAL: Puppet has not run in the last 10 hours [14:28:14] PROBLEM - Puppet freshness on mw1085 is CRITICAL: Puppet has not run in the last 10 hours [14:30:15] PROBLEM - Puppet freshness on cp1034 is CRITICAL: Puppet has not run in the last 10 hours [14:45:17] PROBLEM - Puppet freshness on mw35 is CRITICAL: Puppet has not run in the last 10 hours [16:14:15] PROBLEM - Puppet freshness on mw44 is CRITICAL: Puppet has not run in the last 10 hours [16:16:15] PROBLEM - Puppet freshness on mw1092 is CRITICAL: Puppet has not run in the last 10 hours [16:17:14] PROBLEM - Puppet freshness on colby is CRITICAL: Puppet has not run in the last 10 hours [16:22:15] PROBLEM - Puppet freshness on mw62 is CRITICAL: Puppet has not run in the last 10 hours [16:27:04] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [16:31:04] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.0303 [17:50:20] PROBLEM - Puppet freshness on db67 is CRITICAL: Puppet has not run in the last 10 hours [17:51:20] PROBLEM - Puppet freshness on ms-fe2 is CRITICAL: Puppet has not run in the last 10 hours [18:12:20] PROBLEM - Puppet freshness on mw1126 is CRITICAL: Puppet has not run in the last 10 hours [18:13:06] !log restarting glusterd processes on labstore* [18:13:13] Logged the message, Master [18:13:20] PROBLEM - Puppet freshness on mw1099 is CRITICAL: Puppet has not run in the last 10 hours [18:15:20] PROBLEM - Puppet freshness on mw1152 is CRITICAL: Puppet has not run in the last 10 hours [18:15:20] PROBLEM - Puppet freshness on mw69 is CRITICAL: Puppet has not run in the last 10 hours [18:16:20] PROBLEM - Puppet freshness on mw1001 is CRITICAL: Puppet has not run in the last 10 hours [18:16:20] PROBLEM - Puppet freshness on mw1103 is CRITICAL: Puppet has not run in the last 10 hours [18:17:20] PROBLEM - Puppet freshness on mw1025 is CRITICAL: Puppet has not run in the last 10 hours [18:17:20] PROBLEM - Puppet freshness on mw1109 is CRITICAL: Puppet has not run in the last 10 hours [18:18:20] PROBLEM - Puppet freshness on grosley is CRITICAL: Puppet has not run in the last 10 hours [18:18:20] PROBLEM - Puppet freshness on locke is CRITICAL: Puppet has not run in the last 10 hours [18:18:20] PROBLEM - Puppet freshness on mw1116 is CRITICAL: Puppet has not run in the last 10 hours [18:18:20] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [18:18:20] PROBLEM - Puppet freshness on mw1155 is CRITICAL: Puppet has not run in the last 10 hours [18:18:21] PROBLEM - Puppet freshness on tola is CRITICAL: Puppet has not run in the last 10 hours [18:20:21] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [18:20:22] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [18:20:22] PROBLEM - Puppet freshness on hooft is CRITICAL: Puppet has not run in the last 10 hours [18:21:20] PROBLEM - Puppet freshness on mw1012 is CRITICAL: Puppet has not run in the last 10 hours [18:21:20] PROBLEM - Puppet freshness on mw48 is CRITICAL: Puppet has not run in the last 10 hours [18:22:20] PROBLEM - Puppet freshness on mw1063 is CRITICAL: Puppet has not run in the last 10 hours [18:34:20] PROBLEM - Puppet freshness on cp3022 is CRITICAL: Puppet has not run in the last 10 hours [19:07:21] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [20:12:20] PROBLEM - Puppet freshness on mw1077 is CRITICAL: Puppet has not run in the last 10 hours [20:12:59] Ryan_Lane: The OpenID identity URL... must it be in the form of /wiki/User:Foo ? [20:13:04] (Hi.) [20:14:21] PROBLEM - Puppet freshness on mw1104 is CRITICAL: Puppet has not run in the last 10 hours [20:17:20] PROBLEM - Puppet freshness on mw1043 is CRITICAL: Puppet has not run in the last 10 hours [20:18:20] PROBLEM - Puppet freshness on mw1080 is CRITICAL: Puppet has not run in the last 10 hours [20:20:20] PROBLEM - Puppet freshness on mw1003 is CRITICAL: Puppet has not run in the last 10 hours [20:23:20] PROBLEM - Puppet freshness on virt5 is CRITICAL: Puppet has not run in the last 10 hours [20:24:22] PROBLEM - Puppet freshness on mw1089 is CRITICAL: Puppet has not run in the last 10 hours [20:25:23] PROBLEM - Puppet freshness on mw1129 is CRITICAL: Puppet has not run in the last 10 hours [20:28:20] PROBLEM - Puppet freshness on mw1141 is CRITICAL: Puppet has not run in the last 10 hours [20:32:24] PROBLEM - Puppet freshness on gadolinium is CRITICAL: Puppet has not run in the last 10 hours [20:35:00] PROBLEM - SSH on lvs6 is CRITICAL: Server answer: [20:36:04] RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [20:36:20] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [20:36:21] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [20:36:21] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [20:36:21] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [20:36:21] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [20:43:18] * Damianz puts icinga-wm in his cage [21:28:28] PROBLEM - LVS HTTP IPv4 on m.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 403 Forbidden - pattern not found - 679 bytes in 0.001 second response time [21:28:28] RECOVERY - LVS HTTP IPv4 on m.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 14214 bytes in 0.002 second response time [21:33:24] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [21:33:25] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [21:33:25] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [21:33:25] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [22:13:20] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: Puppet has not run in the last 10 hours [22:13:20] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [22:13:20] PROBLEM - Puppet freshness on cp1028 is CRITICAL: Puppet has not run in the last 10 hours [22:13:20] PROBLEM - Puppet freshness on cerium is CRITICAL: Puppet has not run in the last 10 hours [22:13:20] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [22:13:20] PROBLEM - Puppet freshness on cp3003 is CRITICAL: Puppet has not run in the last 10 hours [22:13:20] PROBLEM - Puppet freshness on cp3010 is CRITICAL: Puppet has not run in the last 10 hours [22:13:50] New patchset: Diederik; "Added domain referer info to blog query, and ignore search and preview urls." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55390 [22:16:20] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [22:21:19] PROBLEM - Puppet freshness on mw1093 is CRITICAL: Puppet has not run in the last 10 hours [22:31:19] PROBLEM - Puppet freshness on db1049 is CRITICAL: Puppet has not run in the last 10 hours [22:33:20] PROBLEM - Puppet freshness on mw1065 is CRITICAL: Puppet has not run in the last 10 hours [23:36:30] New patchset: Tim Starling; "Remove checkers.php and various obsolete spam regexes" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55525 [23:37:40] New patchset: Faidon; "Revert "Cache 4xx responses on the mobile backend servers for 5m"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55526 [23:37:49] TimStarling: ^^^ [23:39:36] New patchset: Faidon; "Revert "Cache 4xx responses on the mobile backend servers for 5m"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55526 [23:40:36] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55526 [23:40:38] paravoid: you think the error messages you were seeing were 404s? [23:41:11] oh, 4xx [23:41:18] PROBLEM - Puppet freshness on constable is CRITICAL: Puppet has not run in the last 10 hours [23:41:20] they were 403s [23:41:20] so 403 would do the trick also [23:41:47] if (beresp.status >= 400) { [23:41:47] set beresp.ttl = 5m; [23:41:47] } [23:41:52] that's what that setting expands to [23:42:08] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Sun Mar 24 23:41:59 UTC 2013 [23:42:58] right [23:45:18] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [23:47:06] grumble [23:47:08] not enough [23:50:06] New patchset: Faidon; "Revert "Remove hit_for_pass code in generic Wikimedia VCL"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55527 [23:51:21] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55527 [23:52:39] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Sun Mar 24 23:52:32 UTC 2013 [23:55:36] New review: Ori.livneh; "I've thought about it a bit more, and I don't think it's a good idea to advertise drafts on IRC. The..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50044 [23:55:51] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Sun Mar 24 23:55:40 UTC 2013 [23:56:01] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Sun Mar 24 23:55:55 UTC 2013