[00:03:42] (03CR) 10JanZerebecki: "Found this by accident. (Thanks to an outage ;) I saw this in the IRC channel.)" [puppet] - 10https://gerrit.wikimedia.org/r/170879 (owner: 10BBlack) [00:05:24] hi, I'm needing (if possible) to import an article from the simple english wiki onto english wiki, to resolve a copyright issue [00:05:48] is it possible to have simple added to the list of projects on Special:Import ? [00:13:58] PROBLEM - puppet last run on db2023 is CRITICAL: CRITICAL: puppet fail [00:33:19] RECOVERY - puppet last run on db2023 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:36:19] (03CR) 10Yuvipanda: "Works fine on shinken-server-01" [puppet] - 10https://gerrit.wikimedia.org/r/173080 (owner: 10Yuvipanda) [00:36:25] (03PS17) 10Yuvipanda: shinken: Setup IRC notification for shinken [puppet] - 10https://gerrit.wikimedia.org/r/173080 [00:48:28] (03CR) 10Yuvipanda: [C: 032] shinken: Setup IRC notification for shinken [puppet] - 10https://gerrit.wikimedia.org/r/173080 (owner: 10Yuvipanda) [00:49:22] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: puppet fail [00:56:17] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [01:03:51] PROBLEM - HHVM busy threads on mw1228 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [115.2] [01:07:42] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [01:09:11] RECOVERY - HHVM busy threads on mw1228 is OK: OK: Less than 1.00% above the threshold [76.8] [02:13:31] !log l10nupdate Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) [02:13:36] !log LocalisationUpdate completed (1.25wmf9) at 2014-11-29 02:13:35+00:00 [02:13:43] Logged the message, Master [02:13:45] Logged the message, Master [02:25:50] !log l10nupdate Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s) [02:25:53] !log LocalisationUpdate completed (1.25wmf10) at 2014-11-29 02:25:53+00:00 [02:25:55] Logged the message, Master [02:25:56] Logged the message, Master [04:17:41] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov 29 04:17:41 UTC 2014 (duration 17m 40s) [04:17:45] Logged the message, Master [06:33:16] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 3 failures [06:33:28] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 3 failures [06:33:45] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:18] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 2 failures [06:36:15] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:17] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: puppet fail [06:36:25] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:46:14] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:46:58] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:47:18] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:15] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:49:30] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:32] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:49:46] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:51:46] PROBLEM - HHVM busy threads on mw1222 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [115.2] [06:55:02] RECOVERY - HHVM busy threads on mw1222 is OK: OK: Less than 1.00% above the threshold [76.8] [06:58:38] PROBLEM - puppet last run on rhenium is CRITICAL: CRITICAL: Puppet has 1 failures [06:58:55] PROBLEM - puppet last run on labsdb1006 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:14] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:31] PROBLEM - puppet last run on mw1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:31] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:35] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:45] PROBLEM - puppet last run on mw1081 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:48] PROBLEM - puppet last run on search1005 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:51] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Puppet has 1 failures [06:59:56] PROBLEM - puppet last run on snapshot1004 is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:05] PROBLEM - puppet last run on search1017 is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:07] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:11] PROBLEM - puppet last run on mw1181 is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:37] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Puppet has 1 failures [07:00:42] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures [07:01:06] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 1 failures [07:01:27] PROBLEM - puppet last run on cp4005 is CRITICAL: CRITICAL: Puppet has 1 failures [07:01:28] PROBLEM - puppet last run on ssl3002 is CRITICAL: CRITICAL: Puppet has 1 failures [07:01:39] PROBLEM - puppet last run on cp1063 is CRITICAL: CRITICAL: Puppet has 1 failures [07:09:06] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:10:01] RECOVERY - puppet last run on labsdb1006 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:10:21] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:10:33] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:10:33] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:10:41] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:10:41] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:10:54] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:11:00] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:11:07] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:11:07] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:11:07] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:11:08] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:11:38] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:11:40] PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 1 failures [07:12:05] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:12:18] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:12:28] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:12:47] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:15:22] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:25:34] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:38:02] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: puppet fail [07:54:36] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:48:28] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 1 failures [09:00:48] PROBLEM - puppet last run on mw1074 is CRITICAL: CRITICAL: Puppet has 1 failures [09:01:29] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: Puppet has 1 failures [09:01:47] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 1 failures [09:01:48] PROBLEM - puppet last run on search1023 is CRITICAL: CRITICAL: Puppet has 1 failures [09:01:48] PROBLEM - puppet last run on protactinium is CRITICAL: CRITICAL: Puppet has 1 failures [09:01:48] PROBLEM - puppet last run on wtp1013 is CRITICAL: CRITICAL: Puppet has 1 failures [09:02:00] PROBLEM - puppet last run on ssl1008 is CRITICAL: CRITICAL: Puppet has 1 failures [09:02:03] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 1 failures [09:02:17] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: Puppet has 1 failures [09:03:18] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 1 failures [09:05:02] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:11:49] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [09:12:42] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:12:42] RECOVERY - puppet last run on protactinium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:13:04] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:13:11] RECOVERY - puppet last run on mw1239 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:14:12] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:15:15] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [09:15:32] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [09:15:42] RECOVERY - puppet last run on ssl1008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:17:56] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:06:11] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: puppet fail [11:22:32] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:09:24] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Puppet has 1 failures [12:10:01] (03CR) 10Nikerabbit: [C: 04-1] Add ContentTranslation database in wikishared (034 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175979 (owner: 10KartikMistry) [12:25:43] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:50:27] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: Puppet has 1 failures [12:50:27] PROBLEM - puppet last run on pc1001 is CRITICAL: CRITICAL: Puppet has 1 failures [12:50:34] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: Puppet has 1 failures [12:50:45] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: Puppet has 1 failures [12:50:45] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: Puppet has 1 failures [12:50:54] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 1 failures [12:50:54] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: Puppet has 1 failures [12:51:17] PROBLEM - puppet last run on mw1072 is CRITICAL: CRITICAL: Puppet has 1 failures [12:51:36] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: Puppet has 1 failures [12:51:36] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Puppet has 1 failures [12:51:47] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: Puppet has 1 failures [12:51:47] PROBLEM - puppet last run on amssq44 is CRITICAL: CRITICAL: Puppet has 1 failures [12:51:47] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:04] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:04] PROBLEM - puppet last run on mw1048 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:05] PROBLEM - puppet last run on mw1059 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:05] PROBLEM - puppet last run on es1001 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:05] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:05] PROBLEM - puppet last run on mc1016 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:05] PROBLEM - puppet last run on mc1009 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:06] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:06] PROBLEM - puppet last run on mw1233 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:19] PROBLEM - puppet last run on ms-be2003 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:20] PROBLEM - puppet last run on mw1041 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:35] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:36] PROBLEM - puppet last run on lvs4002 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:44] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:59] PROBLEM - puppet last run on ms-be1004 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:59] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:59] PROBLEM - puppet last run on rdb1004 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:59] PROBLEM - puppet last run on mw1256 is CRITICAL: CRITICAL: Puppet has 1 failures [12:52:59] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:00] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:05] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:05] PROBLEM - puppet last run on mw1063 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:05] PROBLEM - puppet last run on dbproxy1002 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:14] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:15] PROBLEM - puppet last run on mw1145 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:24] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:27] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:27] PROBLEM - puppet last run on mw1007 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:27] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:27] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:35] PROBLEM - puppet last run on db1053 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:36] PROBLEM - puppet last run on logstash1001 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:44] PROBLEM - puppet last run on amssq59 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:45] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 1 failures [12:53:57] PROBLEM - puppet last run on magnesium is CRITICAL: CRITICAL: Puppet has 1 failures [12:54:04] PROBLEM - puppet last run on mw1140 is CRITICAL: CRITICAL: Puppet has 1 failures [12:54:17] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 1 failures [12:54:24] PROBLEM - puppet last run on amslvs2 is CRITICAL: CRITICAL: Puppet has 1 failures [12:54:24] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 1 failures [12:54:24] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: Puppet has 1 failures [12:54:44] PROBLEM - puppet last run on mw1045 is CRITICAL: CRITICAL: Puppet has 1 failures [12:58:06] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: puppet fail [13:01:37] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:01:37] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [13:01:49] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [13:02:00] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:02:05] RECOVERY - puppet last run on elastic1031 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:02:09] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:02:09] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:02:20] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:02:29] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [13:02:42] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:02:42] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:02:50] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [13:02:52] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:02:53] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [13:03:09] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [13:03:09] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [13:03:10] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:03:19] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:03:21] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:03:25] RECOVERY - puppet last run on mw1233 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:03:39] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [13:03:42] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:04:00] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:04:01] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:04:09] RECOVERY - puppet last run on neptunium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:04:11] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:04:11] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:04:11] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:04:11] RECOVERY - puppet last run on mw1256 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:04:19] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:04:21] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:04:21] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:04:21] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:04:21] RECOVERY - puppet last run on dbproxy1002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:04:29] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [13:04:29] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:04:30] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:04:40] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:04:40] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [13:04:41] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:04:41] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:05:02] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:05:12] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:05:12] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:05:31] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:05:32] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:05:32] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:05:40] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:06:00] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:06:02] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:06:02] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:06:02] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:06:10] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:06:29] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:13:33] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: puppet fail [13:17:17] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:18:19] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:11] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 58823 bytes in 0.086 second response time [13:29:49] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [15:16:00] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.067 second response time [15:19:06] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.010 second response time [15:43:59] PROBLEM - puppet last run on ssl1006 is CRITICAL: CRITICAL: Puppet has 1 failures [15:44:08] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: Puppet has 1 failures [15:45:11] PROBLEM - puppet last run on elastic1003 is CRITICAL: CRITICAL: Puppet has 1 failures [15:45:22] PROBLEM - puppet last run on mw1209 is CRITICAL: CRITICAL: Puppet has 1 failures [15:54:46] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:57:38] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:58:26] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:04:14] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Puppet has 1 failures [16:04:14] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Puppet has 1 failures [16:04:17] PROBLEM - puppet last run on mw1143 is CRITICAL: CRITICAL: Puppet has 1 failures [16:04:26] PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Puppet has 1 failures [16:04:33] PROBLEM - puppet last run on search1004 is CRITICAL: CRITICAL: Puppet has 1 failures [16:04:34] PROBLEM - puppet last run on mw1016 is CRITICAL: CRITICAL: Puppet has 1 failures [16:05:07] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Puppet has 1 failures [16:05:19] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 1 failures [16:05:22] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet has 1 failures [16:05:22] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 1 failures [16:05:37] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 1 failures [16:06:06] PROBLEM - puppet last run on cp1044 is CRITICAL: CRITICAL: Puppet has 1 failures [16:06:25] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: Puppet has 1 failures [16:07:10] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 1 failures [16:07:50] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 1 failures [16:13:25] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:14:06] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:15:10] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:15:11] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:15:11] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:15:17] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:15:26] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:16:08] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:16:08] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:16:16] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:17:53] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:17:54] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:18:08] RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:18:40] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:19:08] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:19:47] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:21:46] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: puppet fail [16:24:10] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: puppet fail [16:38:04] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:43:17] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:22:44] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: Puppet has 1 failures [19:22:52] PROBLEM - puppet last run on es1010 is CRITICAL: CRITICAL: Puppet has 1 failures [19:24:14] PROBLEM - puppet last run on db1033 is CRITICAL: CRITICAL: Puppet has 1 failures [19:33:14] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:33:36] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [19:34:54] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:04:24] PROBLEM - MySQL Processlist on db1054 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:04:35] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: Puppet has 1 failures [20:06:54] RECOVERY - MySQL Processlist on db1054 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 1 statistics [20:19:56] PROBLEM - HTTPS on antimony is CRITICAL: SSL_CERT CRITICAL svn.wikimedia.org: certificate will expire on Jan 31 10:53:05 2015 GMT [20:21:48] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:34:15] PROBLEM - puppet last run on amssq36 is CRITICAL: CRITICAL: puppet fail [20:50:29] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures