[00:10:14] PROBLEM - Puppet freshness on mw44 is CRITICAL: Puppet has not run in the last 10 hours [00:12:14] PROBLEM - Puppet freshness on mw1092 is CRITICAL: Puppet has not run in the last 10 hours [00:13:14] PROBLEM - Puppet freshness on colby is CRITICAL: Puppet has not run in the last 10 hours [00:15:04] PROBLEM - Parsoid on wtp1004 is CRITICAL: Connection refused [00:15:41] !log That's me messing with wtp1004 [00:15:48] Logged the message, Mr. Obvious [00:17:39] !log Deployed master of Parsoid onto wtp1004; the code is broken but the box is depooled for Gabriel's benchmarking work [00:17:46] Logged the message, Mr. Obvious [00:18:15] PROBLEM - Puppet freshness on mw62 is CRITICAL: Puppet has not run in the last 10 hours [00:28:00] will it review? [00:28:16] New patchset: Dzahn; "Rework the RT manifests so it can be installed in Labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47026 [00:29:49] New review: Dzahn; "recheck" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47026 [00:31:08] will it blend ? [00:31:18] New review: Dzahn; "now just not merging this on Friday because the Exec doing the db init might trigger, mysql version ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47026 [00:32:39] LeslieCarr: yes, but Friday after 5:) if that would run on the live instance.. it might be not so good: rt-setup-database --action init --dba root , heh [00:32:48] nice weekend.. byw [00:54:45] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [01:08:48] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.0303 [01:46:38] RECOVERY - Puppet freshness on db67 is OK: puppet ran at Sat Mar 23 01:46:32 UTC 2013 [01:47:38] RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Sat Mar 23 01:47:28 UTC 2013 [02:08:19] PROBLEM - Puppet freshness on mw1126 is CRITICAL: Puppet has not run in the last 10 hours [02:09:18] PROBLEM - Puppet freshness on mw1099 is CRITICAL: Puppet has not run in the last 10 hours [02:11:18] PROBLEM - Puppet freshness on mw1152 is CRITICAL: Puppet has not run in the last 10 hours [02:11:19] PROBLEM - Puppet freshness on mw69 is CRITICAL: Puppet has not run in the last 10 hours [02:12:18] PROBLEM - Puppet freshness on mw1001 is CRITICAL: Puppet has not run in the last 10 hours [02:12:18] PROBLEM - Puppet freshness on mw1103 is CRITICAL: Puppet has not run in the last 10 hours [02:13:18] PROBLEM - Puppet freshness on mw1025 is CRITICAL: Puppet has not run in the last 10 hours [02:13:18] PROBLEM - Puppet freshness on mw1109 is CRITICAL: Puppet has not run in the last 10 hours [02:13:44] !log LocalisationUpdate completed (1.21wmf12) at Sat Mar 23 02:13:44 UTC 2013 [02:13:52] Logged the message, Master [02:15:18] PROBLEM - Puppet freshness on cp1001 is CRITICAL: Puppet has not run in the last 10 hours [02:15:18] PROBLEM - Puppet freshness on db1017 is CRITICAL: Puppet has not run in the last 10 hours [02:15:18] PROBLEM - Puppet freshness on db45 is CRITICAL: Puppet has not run in the last 10 hours [02:15:18] PROBLEM - Puppet freshness on db40 is CRITICAL: Puppet has not run in the last 10 hours [02:15:18] PROBLEM - Puppet freshness on grosley is CRITICAL: Puppet has not run in the last 10 hours [02:15:18] PROBLEM - Puppet freshness on locke is CRITICAL: Puppet has not run in the last 10 hours [02:15:19] PROBLEM - Puppet freshness on mw1155 is CRITICAL: Puppet has not run in the last 10 hours [02:15:19] PROBLEM - Puppet freshness on db1024 is CRITICAL: Puppet has not run in the last 10 hours [02:15:20] PROBLEM - Puppet freshness on mw1116 is CRITICAL: Puppet has not run in the last 10 hours [02:15:20] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [02:15:21] PROBLEM - Puppet freshness on sq83 is CRITICAL: Puppet has not run in the last 10 hours [02:15:21] PROBLEM - Puppet freshness on sq53 is CRITICAL: Puppet has not run in the last 10 hours [02:15:22] PROBLEM - Puppet freshness on tola is CRITICAL: Puppet has not run in the last 10 hours [02:16:18] PROBLEM - Puppet freshness on db1018 is CRITICAL: Puppet has not run in the last 10 hours [02:16:18] PROBLEM - Puppet freshness on db1021 is CRITICAL: Puppet has not run in the last 10 hours [02:16:18] PROBLEM - Puppet freshness on db1033 is CRITICAL: Puppet has not run in the last 10 hours [02:16:19] PROBLEM - Puppet freshness on db1039 is CRITICAL: Puppet has not run in the last 10 hours [02:16:19] PROBLEM - Puppet freshness on db53 is CRITICAL: Puppet has not run in the last 10 hours [02:16:19] PROBLEM - Puppet freshness on hooft is CRITICAL: Puppet has not run in the last 10 hours [02:16:19] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [02:16:19] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [02:16:20] PROBLEM - Puppet freshness on sq49 is CRITICAL: Puppet has not run in the last 10 hours [02:16:20] PROBLEM - Puppet freshness on sq80 is CRITICAL: Puppet has not run in the last 10 hours [02:16:21] PROBLEM - Puppet freshness on sq81 is CRITICAL: Puppet has not run in the last 10 hours [02:17:18] PROBLEM - Puppet freshness on mw1012 is CRITICAL: Puppet has not run in the last 10 hours [02:17:18] PROBLEM - Puppet freshness on mw48 is CRITICAL: Puppet has not run in the last 10 hours [02:18:18] PROBLEM - Puppet freshness on mw1063 is CRITICAL: Puppet has not run in the last 10 hours [02:28:36] New review: Peachey88; "Sieband, Why did you -1 this? you comment does not make sense with your review" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31823 [02:30:20] PROBLEM - Puppet freshness on cp3022 is CRITICAL: Puppet has not run in the last 10 hours [02:36:54] !log LocalisationUpdate completed (1.21wmf11) at Sat Mar 23 02:36:53 UTC 2013 [02:37:02] Logged the message, Master [03:03:19] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [04:09:18] PROBLEM - Puppet freshness on mw1077 is CRITICAL: Puppet has not run in the last 10 hours [04:10:18] PROBLEM - Puppet freshness on mw1104 is CRITICAL: Puppet has not run in the last 10 hours [04:13:19] PROBLEM - Puppet freshness on mw1043 is CRITICAL: Puppet has not run in the last 10 hours [04:14:18] PROBLEM - Puppet freshness on mw1080 is CRITICAL: Puppet has not run in the last 10 hours [04:16:18] PROBLEM - Puppet freshness on mw1003 is CRITICAL: Puppet has not run in the last 10 hours [04:19:18] PROBLEM - Puppet freshness on virt5 is CRITICAL: Puppet has not run in the last 10 hours [04:20:18] PROBLEM - Puppet freshness on mw1089 is CRITICAL: Puppet has not run in the last 10 hours [04:21:18] PROBLEM - Puppet freshness on mw1129 is CRITICAL: Puppet has not run in the last 10 hours [04:24:20] PROBLEM - Puppet freshness on mw1141 is CRITICAL: Puppet has not run in the last 10 hours [04:28:18] PROBLEM - Puppet freshness on gadolinium is CRITICAL: Puppet has not run in the last 10 hours [04:32:19] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [04:32:20] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [04:32:20] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [04:32:20] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [04:32:20] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [04:32:20] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: Puppet has not run in the last 10 hours [04:32:20] PROBLEM - Puppet freshness on analytics1003 is CRITICAL: Puppet has not run in the last 10 hours [05:20:39] PROBLEM - LVS HTTP IPv4 on m.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 403 Forbidden - pattern not found - 677 bytes in 0.001 second response time [05:21:38] RECOVERY - LVS HTTP IPv4 on m.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 15685 bytes in 0.002 second response time [05:28:49] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [05:30:16] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [05:30:17] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [05:30:17] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [05:30:17] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [05:32:56] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.7037 [06:08:58] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [06:09:26] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: Puppet has not run in the last 10 hours [06:09:26] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [06:09:26] PROBLEM - Puppet freshness on cerium is CRITICAL: Puppet has not run in the last 10 hours [06:09:26] PROBLEM - Puppet freshness on cp1028 is CRITICAL: Puppet has not run in the last 10 hours [06:09:26] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [06:12:13] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [06:12:53] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -2.5641 [06:17:13] PROBLEM - Puppet freshness on mw1093 is CRITICAL: Puppet has not run in the last 10 hours [06:27:13] PROBLEM - Puppet freshness on db1049 is CRITICAL: Puppet has not run in the last 10 hours [06:29:13] PROBLEM - Puppet freshness on mw1065 is CRITICAL: Puppet has not run in the last 10 hours [06:48:53] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [06:52:55] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.7037 [07:37:13] PROBLEM - Puppet freshness on constable is CRITICAL: Puppet has not run in the last 10 hours [07:41:14] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [07:47:35] New patchset: Yurik; "Unified default language redirect from m. & zero." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55302 [08:15:15] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [08:15:16] PROBLEM - Puppet freshness on mw1158 is CRITICAL: Puppet has not run in the last 10 hours [08:19:13] PROBLEM - Puppet freshness on mw1118 is CRITICAL: Puppet has not run in the last 10 hours [08:25:17] PROBLEM - Puppet freshness on mw1085 is CRITICAL: Puppet has not run in the last 10 hours [08:27:13] PROBLEM - Puppet freshness on cp1034 is CRITICAL: Puppet has not run in the last 10 hours [08:42:13] PROBLEM - Puppet freshness on mw35 is CRITICAL: Puppet has not run in the last 10 hours [10:11:13] PROBLEM - Puppet freshness on mw44 is CRITICAL: Puppet has not run in the last 10 hours [10:13:13] PROBLEM - Puppet freshness on mw1092 is CRITICAL: Puppet has not run in the last 10 hours [10:14:13] PROBLEM - Puppet freshness on colby is CRITICAL: Puppet has not run in the last 10 hours [10:19:14] PROBLEM - Puppet freshness on mw62 is CRITICAL: Puppet has not run in the last 10 hours [10:49:35] New review: Siebrand; "Hydriz made an inline comment that requirements follow-up" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31823 [10:50:24] New review: Siebrand; "Darn auto correct for Dutch :(" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31823 [11:34:56] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [11:38:57] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -2.85714 [11:47:15] PROBLEM - Puppet freshness on db67 is CRITICAL: Puppet has not run in the last 10 hours [11:48:15] PROBLEM - Puppet freshness on ms-fe2 is CRITICAL: Puppet has not run in the last 10 hours [12:09:15] PROBLEM - Puppet freshness on mw1126 is CRITICAL: Puppet has not run in the last 10 hours [12:10:15] PROBLEM - Puppet freshness on mw1099 is CRITICAL: Puppet has not run in the last 10 hours [12:12:15] PROBLEM - Puppet freshness on mw1152 is CRITICAL: Puppet has not run in the last 10 hours [12:12:15] PROBLEM - Puppet freshness on mw69 is CRITICAL: Puppet has not run in the last 10 hours [12:12:52] New patchset: Yuvipanda; "Added Yuvi Panda to English Planet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55462 [12:13:16] PROBLEM - Puppet freshness on mw1001 is CRITICAL: Puppet has not run in the last 10 hours [12:13:17] PROBLEM - Puppet freshness on mw1103 is CRITICAL: Puppet has not run in the last 10 hours [12:14:15] PROBLEM - Puppet freshness on mw1025 is CRITICAL: Puppet has not run in the last 10 hours [12:14:15] PROBLEM - Puppet freshness on mw1109 is CRITICAL: Puppet has not run in the last 10 hours [12:16:15] PROBLEM - Puppet freshness on cp1001 is CRITICAL: Puppet has not run in the last 10 hours [12:16:15] PROBLEM - Puppet freshness on db1017 is CRITICAL: Puppet has not run in the last 10 hours [12:16:15] PROBLEM - Puppet freshness on db1024 is CRITICAL: Puppet has not run in the last 10 hours [12:16:15] PROBLEM - Puppet freshness on db40 is CRITICAL: Puppet has not run in the last 10 hours [12:16:15] PROBLEM - Puppet freshness on db45 is CRITICAL: Puppet has not run in the last 10 hours [12:16:16] PROBLEM - Puppet freshness on grosley is CRITICAL: Puppet has not run in the last 10 hours [12:16:16] PROBLEM - Puppet freshness on locke is CRITICAL: Puppet has not run in the last 10 hours [12:16:17] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [12:16:17] PROBLEM - Puppet freshness on mw1116 is CRITICAL: Puppet has not run in the last 10 hours [12:16:18] PROBLEM - Puppet freshness on mw1155 is CRITICAL: Puppet has not run in the last 10 hours [12:16:18] PROBLEM - Puppet freshness on sq53 is CRITICAL: Puppet has not run in the last 10 hours [12:16:19] PROBLEM - Puppet freshness on sq83 is CRITICAL: Puppet has not run in the last 10 hours [12:16:19] PROBLEM - Puppet freshness on tola is CRITICAL: Puppet has not run in the last 10 hours [12:17:15] PROBLEM - Puppet freshness on db1018 is CRITICAL: Puppet has not run in the last 10 hours [12:17:15] PROBLEM - Puppet freshness on db1021 is CRITICAL: Puppet has not run in the last 10 hours [12:17:15] PROBLEM - Puppet freshness on db1033 is CRITICAL: Puppet has not run in the last 10 hours [12:17:15] PROBLEM - Puppet freshness on db1039 is CRITICAL: Puppet has not run in the last 10 hours [12:17:15] PROBLEM - Puppet freshness on db53 is CRITICAL: Puppet has not run in the last 10 hours [12:17:16] PROBLEM - Puppet freshness on hooft is CRITICAL: Puppet has not run in the last 10 hours [12:17:16] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [12:17:18] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [12:17:18] PROBLEM - Puppet freshness on sq49 is CRITICAL: Puppet has not run in the last 10 hours [12:17:18] PROBLEM - Puppet freshness on sq80 is CRITICAL: Puppet has not run in the last 10 hours [12:17:18] PROBLEM - Puppet freshness on sq81 is CRITICAL: Puppet has not run in the last 10 hours [12:18:15] PROBLEM - Puppet freshness on mw1012 is CRITICAL: Puppet has not run in the last 10 hours [12:18:15] PROBLEM - Puppet freshness on mw48 is CRITICAL: Puppet has not run in the last 10 hours [12:19:16] PROBLEM - Puppet freshness on mw1063 is CRITICAL: Puppet has not run in the last 10 hours [12:31:17] PROBLEM - Puppet freshness on cp3022 is CRITICAL: Puppet has not run in the last 10 hours [13:04:16] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [14:08:56] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [14:09:36] PROBLEM - Puppet freshness on mw1077 is CRITICAL: Puppet has not run in the last 10 hours [14:11:16] PROBLEM - Puppet freshness on mw1104 is CRITICAL: Puppet has not run in the last 10 hours [14:12:55] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -2.7027 [14:14:15] PROBLEM - Puppet freshness on mw1043 is CRITICAL: Puppet has not run in the last 10 hours [14:15:15] PROBLEM - Puppet freshness on mw1080 is CRITICAL: Puppet has not run in the last 10 hours [14:17:15] PROBLEM - Puppet freshness on mw1003 is CRITICAL: Puppet has not run in the last 10 hours [14:20:15] PROBLEM - Puppet freshness on virt5 is CRITICAL: Puppet has not run in the last 10 hours [14:21:15] PROBLEM - Puppet freshness on mw1089 is CRITICAL: Puppet has not run in the last 10 hours [14:22:15] PROBLEM - Puppet freshness on mw1129 is CRITICAL: Puppet has not run in the last 10 hours [14:25:16] PROBLEM - Puppet freshness on mw1141 is CRITICAL: Puppet has not run in the last 10 hours [14:29:15] PROBLEM - Puppet freshness on gadolinium is CRITICAL: Puppet has not run in the last 10 hours [14:33:15] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [14:33:15] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [14:33:15] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [14:33:15] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [14:33:15] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [15:31:15] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [15:31:16] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [15:31:16] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [15:31:16] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [15:38:59] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [15:42:58] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -2.63158 [16:10:20] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: Puppet has not run in the last 10 hours [16:10:20] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [16:10:20] PROBLEM - Puppet freshness on cerium is CRITICAL: Puppet has not run in the last 10 hours [16:10:20] PROBLEM - Puppet freshness on cp1028 is CRITICAL: Puppet has not run in the last 10 hours [16:10:20] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [16:13:20] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [16:18:20] PROBLEM - Puppet freshness on mw1093 is CRITICAL: Puppet has not run in the last 10 hours [16:28:21] PROBLEM - Puppet freshness on db1049 is CRITICAL: Puppet has not run in the last 10 hours [16:30:20] PROBLEM - Puppet freshness on mw1065 is CRITICAL: Puppet has not run in the last 10 hours [17:38:20] PROBLEM - Puppet freshness on constable is CRITICAL: Puppet has not run in the last 10 hours [17:42:20] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [17:43:30] RECOVERY - Puppet freshness on db1021 is OK: puppet ran at Sat Mar 23 17:43:26 UTC 2013 [17:48:30] RECOVERY - Puppet freshness on db1024 is OK: puppet ran at Sat Mar 23 17:48:24 UTC 2013 [17:50:40] RECOVERY - Puppet freshness on sq81 is OK: puppet ran at Sat Mar 23 17:50:31 UTC 2013 [17:53:32] RECOVERY - Puppet freshness on cp1001 is OK: puppet ran at Sat Mar 23 17:53:26 UTC 2013 [17:53:40] RECOVERY - Puppet freshness on sq49 is OK: puppet ran at Sat Mar 23 17:53:32 UTC 2013 [17:54:40] RECOVERY - Puppet freshness on db1018 is OK: puppet ran at Sat Mar 23 17:54:36 UTC 2013 [17:57:31] RECOVERY - Puppet freshness on sq53 is OK: puppet ran at Sat Mar 23 17:57:21 UTC 2013 [17:57:32] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Sat Mar 23 17:57:21 UTC 2013 [18:00:32] RECOVERY - Puppet freshness on sq80 is OK: puppet ran at Sat Mar 23 18:00:25 UTC 2013 [18:01:30] RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Sat Mar 23 18:01:29 UTC 2013 [18:04:30] RECOVERY - Puppet freshness on sq83 is OK: puppet ran at Sat Mar 23 18:04:21 UTC 2013 [18:04:30] RECOVERY - Puppet freshness on db40 is OK: puppet ran at Sat Mar 23 18:04:23 UTC 2013 [18:06:40] RECOVERY - Puppet freshness on db53 is OK: puppet ran at Sat Mar 23 18:06:35 UTC 2013 [18:08:30] RECOVERY - Puppet freshness on db45 is OK: puppet ran at Sat Mar 23 18:08:20 UTC 2013 [18:12:34] RECOVERY - Puppet freshness on db1033 is OK: puppet ran at Sat Mar 23 18:12:26 UTC 2013 [18:16:20] PROBLEM - Puppet freshness on mw1158 is CRITICAL: Puppet has not run in the last 10 hours [18:16:20] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [18:20:21] PROBLEM - Puppet freshness on mw1118 is CRITICAL: Puppet has not run in the last 10 hours [18:20:58] poor poor icinga-wm, all by itself [18:21:08] it's prison is reporting to a channel, all alone [18:26:20] PROBLEM - Puppet freshness on mw1085 is CRITICAL: Puppet has not run in the last 10 hours [18:28:20] PROBLEM - Puppet freshness on cp1034 is CRITICAL: Puppet has not run in the last 10 hours [18:43:21] PROBLEM - Puppet freshness on mw35 is CRITICAL: Puppet has not run in the last 10 hours [20:12:21] PROBLEM - Puppet freshness on mw44 is CRITICAL: Puppet has not run in the last 10 hours [20:14:20] PROBLEM - Puppet freshness on mw1092 is CRITICAL: Puppet has not run in the last 10 hours [20:15:20] PROBLEM - Puppet freshness on colby is CRITICAL: Puppet has not run in the last 10 hours [20:20:20] PROBLEM - Puppet freshness on mw62 is CRITICAL: Puppet has not run in the last 10 hours [21:29:00] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [21:33:01] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.44828 [21:48:21] PROBLEM - Puppet freshness on db67 is CRITICAL: Puppet has not run in the last 10 hours [21:49:21] PROBLEM - Puppet freshness on ms-fe2 is CRITICAL: Puppet has not run in the last 10 hours [22:10:21] PROBLEM - Puppet freshness on mw1126 is CRITICAL: Puppet has not run in the last 10 hours [22:11:21] PROBLEM - Puppet freshness on mw1099 is CRITICAL: Puppet has not run in the last 10 hours [22:13:23] PROBLEM - Puppet freshness on mw1152 is CRITICAL: Puppet has not run in the last 10 hours [22:13:23] PROBLEM - Puppet freshness on mw69 is CRITICAL: Puppet has not run in the last 10 hours [22:14:21] PROBLEM - Puppet freshness on mw1103 is CRITICAL: Puppet has not run in the last 10 hours [22:14:21] PROBLEM - Puppet freshness on mw1001 is CRITICAL: Puppet has not run in the last 10 hours [22:15:21] PROBLEM - Puppet freshness on mw1025 is CRITICAL: Puppet has not run in the last 10 hours [22:15:21] PROBLEM - Puppet freshness on mw1109 is CRITICAL: Puppet has not run in the last 10 hours [22:16:21] PROBLEM - Puppet freshness on grosley is CRITICAL: Puppet has not run in the last 10 hours [22:16:21] PROBLEM - Puppet freshness on locke is CRITICAL: Puppet has not run in the last 10 hours [22:16:21] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [22:16:21] PROBLEM - Puppet freshness on mw1116 is CRITICAL: Puppet has not run in the last 10 hours [22:16:21] PROBLEM - Puppet freshness on mw1155 is CRITICAL: Puppet has not run in the last 10 hours [22:16:22] PROBLEM - Puppet freshness on tola is CRITICAL: Puppet has not run in the last 10 hours [22:18:21] PROBLEM - Puppet freshness on hooft is CRITICAL: Puppet has not run in the last 10 hours [22:18:21] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [22:18:21] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [22:19:21] PROBLEM - Puppet freshness on mw48 is CRITICAL: Puppet has not run in the last 10 hours [22:19:21] PROBLEM - Puppet freshness on mw1012 is CRITICAL: Puppet has not run in the last 10 hours [22:20:21] PROBLEM - Puppet freshness on mw1063 is CRITICAL: Puppet has not run in the last 10 hours [22:32:21] PROBLEM - Puppet freshness on cp3022 is CRITICAL: Puppet has not run in the last 10 hours [22:49:01] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [22:53:01] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.0303 [23:05:22] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours