[00:07:56] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 00:07:48 UTC 2013 [00:08:02] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 00:07:55 UTC 2013 [00:08:13] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 00:08:04 UTC 2013 [00:08:21] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 00:08:13 UTC 2013 [00:08:21] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 00:08:16 UTC 2013 [00:08:31] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:31] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 00:08:23 UTC 2013 [00:08:31] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 00:08:26 UTC 2013 [00:08:44] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 00:08:31 UTC 2013 [00:08:44] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 00:08:32 UTC 2013 [00:08:44] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 00:08:36 UTC 2013 [00:08:44] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:44] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:44] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:51] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:51] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:51] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 00:08:45 UTC 2013 [00:08:51] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 00:08:45 UTC 2013 [00:08:51] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:02] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:02] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 00:08:54 UTC 2013 [00:09:02] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 00:08:54 UTC 2013 [00:09:11] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:11] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:12] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 00:09:08 UTC 2013 [00:09:12] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:24] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:25] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 00:09:18 UTC 2013 [00:09:31] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 00:09:21 UTC 2013 [00:09:31] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 00:09:21 UTC 2013 [00:09:31] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:41] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:41] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 00:09:39 UTC 2013 [00:09:41] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:51] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 00:09:46 UTC 2013 [00:09:51] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 00:09:46 UTC 2013 [00:09:51] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 00:09:47 UTC 2013 [00:09:51] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 00:09:49 UTC 2013 [00:09:51] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:01] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:01] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 00:09:57 UTC 2013 [00:10:11] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 00:10:02 UTC 2013 [00:10:11] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 00:10:03 UTC 2013 [00:10:11] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:11] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 00:10:05 UTC 2013 [00:10:11] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 00:10:05 UTC 2013 [00:10:12] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:21] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:21] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 00:10:17 UTC 2013 [00:10:21] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 00:10:17 UTC 2013 [00:10:31] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:31] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 00:10:27 UTC 2013 [00:10:31] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:31] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:41] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 00:10:35 UTC 2013 [00:10:41] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 00:10:38 UTC 2013 [00:10:41] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:41] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 00:10:40 UTC 2013 [00:10:51] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:51] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 00:10:50 UTC 2013 [00:10:51] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 00:10:50 UTC 2013 [00:10:51] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:51] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:52] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:01] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:01] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 00:10:56 UTC 2013 [00:11:02] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:02] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:11] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:11] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 00:11:07 UTC 2013 [00:11:11] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 00:11:08 UTC 2013 [00:11:21] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 00:11:11 UTC 2013 [00:11:21] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 00:11:14 UTC 2013 [00:11:21] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 00:11:14 UTC 2013 [00:11:31] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:31] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 00:11:26 UTC 2013 [00:11:31] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 00:11:26 UTC 2013 [00:11:41] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:41] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:41] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 00:11:40 UTC 2013 [00:11:51] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:51] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:51] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 00:11:47 UTC 2013 [00:12:01] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 00:11:50 UTC 2013 [00:12:01] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 00:11:51 UTC 2013 [00:12:01] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:01] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:01] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 00:11:58 UTC 2013 [00:12:11] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:11] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:11] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 00:12:06 UTC 2013 [00:12:11] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 00:12:07 UTC 2013 [00:12:11] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 00:12:10 UTC 2013 [00:12:12] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:12] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:21] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:21] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 00:12:17 UTC 2013 [00:12:31] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 00:12:20 UTC 2013 [00:12:31] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:31] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:31] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:41] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 00:12:37 UTC 2013 [00:12:41] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 00:12:37 UTC 2013 [00:12:41] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:41] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:51] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 00:12:41 UTC 2013 [00:12:51] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 00:12:42 UTC 2013 [00:12:51] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:52] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:52] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:01] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 00:12:53 UTC 2013 [00:13:01] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 00:12:54 UTC 2013 [00:13:01] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 00:12:56 UTC 2013 [00:13:01] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:01] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:11] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 00:13:02 UTC 2013 [00:13:11] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 00:13:06 UTC 2013 [00:13:11] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 00:13:10 UTC 2013 [00:13:21] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:21] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 00:13:18 UTC 2013 [00:13:21] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 00:13:18 UTC 2013 [00:13:21] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 00:13:20 UTC 2013 [00:13:31] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:31] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:31] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 00:13:26 UTC 2013 [00:13:31] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 00:13:27 UTC 2013 [00:13:41] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 00:13:32 UTC 2013 [00:13:42] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 00:13:35 UTC 2013 [00:13:42] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 00:13:35 UTC 2013 [00:13:42] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:42] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:42] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:51] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 00:13:43 UTC 2013 [00:13:51] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 00:13:43 UTC 2013 [00:13:51] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:51] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:51] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:01] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 00:13:52 UTC 2013 [00:14:01] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:01] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:01] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:01] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 00:13:58 UTC 2013 [00:14:11] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 00:14:02 UTC 2013 [00:14:11] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 00:14:03 UTC 2013 [00:14:11] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:11] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:11] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:22] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:23] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 00:14:12 UTC 2013 [00:14:23] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:23] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 00:14:18 UTC 2013 [00:14:31] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 00:14:23 UTC 2013 [00:14:31] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 00:14:23 UTC 2013 [00:14:31] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:31] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 00:14:27 UTC 2013 [00:14:31] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:32] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:42] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 00:14:31 UTC 2013 [00:14:42] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:42] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 00:14:36 UTC 2013 [00:14:42] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 00:14:37 UTC 2013 [00:14:42] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:42] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 00:14:39 UTC 2013 [00:14:42] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 00:14:39 UTC 2013 [00:14:43] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:51] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:52] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 00:14:49 UTC 2013 [00:14:52] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 00:14:49 UTC 2013 [00:14:52] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:52] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [00:14:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:02] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 00:14:52 UTC 2013 [00:15:02] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:02] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:12] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 00:15:02 UTC 2013 [00:15:12] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 00:15:07 UTC 2013 [00:15:12] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:22] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 00:15:13 UTC 2013 [00:15:22] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:22] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 00:15:17 UTC 2013 [00:15:22] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 00:15:17 UTC 2013 [00:15:31] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 00:15:23 UTC 2013 [00:15:31] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:31] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:42] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 00:15:32 UTC 2013 [00:15:42] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 00:15:33 UTC 2013 [00:15:42] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 00:15:35 UTC 2013 [00:15:42] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:42] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 00:15:36 UTC 2013 [00:15:42] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 00:15:36 UTC 2013 [00:15:42] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:52] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 00:15:43 UTC 2013 [00:15:52] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:52] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [00:15:52] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 00:15:46 UTC 2013 [00:15:52] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:02] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:12] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 00:16:01 UTC 2013 [00:16:12] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:12] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 00:16:12 UTC 2013 [00:16:21] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:21] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:21] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 00:16:17 UTC 2013 [00:16:32] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 00:16:20 UTC 2013 [00:16:32] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 00:16:21 UTC 2013 [00:16:33] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 00:16:23 UTC 2013 [00:16:33] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 00:16:23 UTC 2013 [00:16:33] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 00:16:23 UTC 2013 [00:16:33] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:33] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:34] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 00:16:25 UTC 2013 [00:16:34] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:42] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:43] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:43] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 00:16:37 UTC 2013 [00:16:43] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:43] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [00:16:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:02] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 00:16:51 UTC 2013 [00:17:03] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 00:16:52 UTC 2013 [00:17:03] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:03] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 00:16:55 UTC 2013 [00:17:03] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 00:16:58 UTC 2013 [00:17:13] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:14] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 00:17:06 UTC 2013 [00:17:14] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 00:17:06 UTC 2013 [00:17:14] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 00:17:07 UTC 2013 [00:17:14] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:14] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 00:17:09 UTC 2013 [00:17:23] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:31] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:31] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:31] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 00:17:29 UTC 2013 [00:17:44] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 00:17:31 UTC 2013 [00:17:44] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 00:17:33 UTC 2013 [00:17:44] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 00:17:33 UTC 2013 [00:17:44] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:44] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 00:17:39 UTC 2013 [00:17:44] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:51] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:51] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:51] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:52] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [00:17:52] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:01] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 00:17:51 UTC 2013 [00:18:01] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 00:17:53 UTC 2013 [00:18:01] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:01] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:01] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:02] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 00:17:54 UTC 2013 [00:18:02] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 00:17:56 UTC 2013 [00:18:03] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:11] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:11] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 00:18:05 UTC 2013 [00:18:11] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 00:18:05 UTC 2013 [00:18:11] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 00:18:06 UTC 2013 [00:18:21] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:21] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:21] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 00:18:15 UTC 2013 [00:18:21] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 00:18:19 UTC 2013 [00:18:33] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 00:18:26 UTC 2013 [00:18:33] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 00:18:26 UTC 2013 [00:18:41] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:42] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:42] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 00:18:35 UTC 2013 [00:18:42] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 00:18:37 UTC 2013 [00:18:42] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 00:18:37 UTC 2013 [00:18:42] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 00:18:37 UTC 2013 [00:18:42] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:43] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [00:18:43] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 00:18:40 UTC 2013 [00:18:53] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:02] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:02] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 00:18:54 UTC 2013 [00:19:02] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:02] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 00:18:57 UTC 2013 [00:19:02] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 00:18:58 UTC 2013 [00:19:11] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 00:19:01 UTC 2013 [00:19:11] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 00:19:01 UTC 2013 [00:19:11] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:11] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:11] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:21] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 00:19:11 UTC 2013 [00:19:21] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 00:19:11 UTC 2013 [00:19:21] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 00:19:14 UTC 2013 [00:19:21] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:21] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 00:19:17 UTC 2013 [00:19:22] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 00:19:19 UTC 2013 [00:19:22] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 00:19:20 UTC 2013 [00:19:31] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:31] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:31] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:31] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:31] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:32] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:32] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:33] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 00:19:25 UTC 2013 [00:19:33] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 00:19:25 UTC 2013 [00:19:34] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:34] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 00:19:26 UTC 2013 [00:19:35] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 00:19:30 UTC 2013 [00:19:35] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 00:19:30 UTC 2013 [00:19:41] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:41] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 00:19:36 UTC 2013 [00:19:41] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 00:19:39 UTC 2013 [00:19:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:51] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 00:19:43 UTC 2013 [00:19:51] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 00:19:43 UTC 2013 [00:19:51] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 00:19:43 UTC 2013 [00:19:51] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:51] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:52] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [00:19:52] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:01] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 00:19:52 UTC 2013 [00:20:01] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:01] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:01] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:01] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:02] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:03] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:11] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 00:20:03 UTC 2013 [00:20:11] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 00:20:04 UTC 2013 [00:20:11] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 00:20:04 UTC 2013 [00:20:11] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:11] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:12] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:12] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 00:20:07 UTC 2013 [00:20:13] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 00:20:09 UTC 2013 [00:20:21] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 00:20:11 UTC 2013 [00:20:21] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:21] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:31] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 00:20:21 UTC 2013 [00:20:31] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 00:20:21 UTC 2013 [00:20:31] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 00:20:23 UTC 2013 [00:20:31] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:31] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:32] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:32] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 00:20:27 UTC 2013 [00:20:33] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 00:20:27 UTC 2013 [00:20:33] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 00:20:28 UTC 2013 [00:20:34] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 00:20:29 UTC 2013 [00:20:42] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:42] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:42] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 00:20:37 UTC 2013 [00:20:42] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 00:20:37 UTC 2013 [00:20:42] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:43] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:51] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 00:20:41 UTC 2013 [00:20:51] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [00:20:51] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 00:20:48 UTC 2013 [00:20:51] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 00:20:49 UTC 2013 [00:20:51] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 00:20:50 UTC 2013 [00:21:01] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 00:20:54 UTC 2013 [00:21:01] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:01] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:01] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:01] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:02] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 00:20:55 UTC 2013 [00:21:02] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 00:20:56 UTC 2013 [00:21:03] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:11] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 00:21:00 UTC 2013 [00:21:11] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 00:21:01 UTC 2013 [00:21:11] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 00:21:02 UTC 2013 [00:21:11] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 00:21:03 UTC 2013 [00:21:11] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:12] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 00:21:07 UTC 2013 [00:21:12] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 00:21:07 UTC 2013 [00:21:13] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 00:21:08 UTC 2013 [00:21:13] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 00:21:09 UTC 2013 [00:21:14] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:21] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 00:21:12 UTC 2013 [00:21:21] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 00:21:12 UTC 2013 [00:21:21] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 00:21:12 UTC 2013 [00:21:21] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 00:21:16 UTC 2013 [00:21:21] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:22] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:22] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:23] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 00:21:17 UTC 2013 [00:21:23] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 00:21:18 UTC 2013 [00:21:24] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 00:21:18 UTC 2013 [00:21:24] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 00:21:20 UTC 2013 [00:21:31] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 00:21:21 UTC 2013 [00:21:31] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 00:21:23 UTC 2013 [00:21:31] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 00:21:24 UTC 2013 [00:21:31] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 00:21:24 UTC 2013 [00:21:31] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 00:21:24 UTC 2013 [00:21:32] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:32] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:33] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:33] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:34] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:34] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 00:21:26 UTC 2013 [00:21:35] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 00:21:26 UTC 2013 [00:21:35] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 00:21:27 UTC 2013 [00:21:41] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:41] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:42] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:42] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:42] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 00:21:39 UTC 2013 [00:21:42] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:42] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 00:21:42 UTC 2013 [00:21:53] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 00:21:43 UTC 2013 [00:21:53] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 00:21:43 UTC 2013 [00:21:53] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 00:21:45 UTC 2013 [00:21:53] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:53] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:53] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [00:21:54] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:01] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:02] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:02] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:02] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:02] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:02] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:11] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:11] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:11] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:11] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:11] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:21] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:21] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:21] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:33] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:33] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:33] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:33] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:41] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:41] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [00:23:11] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 00:23:01 UTC 2013 [00:23:32] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 00:23:22 UTC 2013 [00:23:32] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 00:23:27 UTC 2013 [00:23:41] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [00:24:11] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [00:24:31] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [00:25:43] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 00:25:35 UTC 2013 [00:26:01] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [00:26:41] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 00:26:40 UTC 2013 [00:26:41] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 00:26:40 UTC 2013 [00:26:51] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 00:26:45 UTC 2013 [00:27:11] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [00:27:31] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [00:27:43] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:27:53] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 00:27:44 UTC 2013 [00:28:31] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [00:29:41] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 00:29:38 UTC 2013 [00:30:21] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [00:31:41] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 00:31:38 UTC 2013 [00:31:41] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 00:31:38 UTC 2013 [00:32:01] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [00:32:01] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [00:32:33] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 00:32:26 UTC 2013 [00:32:33] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 00:32:26 UTC 2013 [00:32:52] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [00:33:01] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [01:05:39] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 01:05:31 UTC 2013 [01:05:59] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [01:06:09] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [01:06:49] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 01:06:40 UTC 2013 [01:07:19] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [01:07:29] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 01:07:22 UTC 2013 [01:08:20] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [01:08:39] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 01:08:34 UTC 2013 [01:08:50] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:30] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 01:10:24 UTC 2013 [01:10:49] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 01:10:45 UTC 2013 [01:11:19] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [01:11:19] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [01:11:39] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 01:11:32 UTC 2013 [01:12:19] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [01:13:59] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 01:13:49 UTC 2013 [01:14:09] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 01:13:59 UTC 2013 [01:14:39] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [01:14:49] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [01:14:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 01:14:45 UTC 2013 [01:15:39] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 01:15:36 UTC 2013 [01:15:39] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 01:15:36 UTC 2013 [01:15:49] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [01:15:49] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [01:16:19] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [01:17:29] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 01:17:26 UTC 2013 [01:17:49] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [01:17:49] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 01:17:46 UTC 2013 [01:18:19] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [01:20:39] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 01:20:31 UTC 2013 [01:21:19] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [01:22:39] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 01:22:34 UTC 2013 [01:22:49] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [01:23:29] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 01:23:25 UTC 2013 [01:23:39] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 01:23:30 UTC 2013 [01:23:49] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [01:24:09] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [01:26:01] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 01:25:50 UTC 2013 [01:26:49] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [01:27:00] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 01:26:57 UTC 2013 [01:27:00] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 01:26:57 UTC 2013 [01:27:00] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 01:26:57 UTC 2013 [01:27:09] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [01:27:39] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 01:27:32 UTC 2013 [01:27:49] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [01:27:49] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [01:28:09] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [01:31:59] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 01:31:54 UTC 2013 [01:31:59] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 01:31:55 UTC 2013 [01:32:09] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 01:32:05 UTC 2013 [01:32:19] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [01:32:20] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.004076838493 secs [01:32:31] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 01:32:26 UTC 2013 [01:32:39] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 01:32:31 UTC 2013 [01:32:49] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [01:32:49] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [01:32:59] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.003705859184 secs [01:33:19] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [01:33:20] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [01:34:49] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 01:34:40 UTC 2013 [01:34:49] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 01:34:40 UTC 2013 [01:34:49] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [01:34:49] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 01:34:41 UTC 2013 [01:35:09] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [01:36:35] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [02:04:49] !log LocalisationUpdate completed (1.22wmf4) at Sat May 25 02:04:49 UTC 2013 [02:05:03] Logged the message, Master [02:08:05] PROBLEM - Puppet freshness on lanthanum is CRITICAL: No successful Puppet run in the last 10 hours [02:13:05] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [02:17:03] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat May 25 02:17:03 UTC 2013 [02:17:13] Logged the message, Master [02:22:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [02:27:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [02:43:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [02:56:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time [03:02:36] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [03:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [03:43:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [03:52:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [03:57:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [04:07:46] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 04:07:38 UTC 2013 [04:07:46] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 04:07:44 UTC 2013 [04:07:56] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 04:07:50 UTC 2013 [04:08:06] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 04:07:56 UTC 2013 [04:08:06] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:06] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 04:08:02 UTC 2013 [04:08:06] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 04:08:02 UTC 2013 [04:08:06] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 04:08:04 UTC 2013 [04:08:16] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 04:08:11 UTC 2013 [04:08:17] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 04:08:12 UTC 2013 [04:08:17] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 04:08:14 UTC 2013 [04:08:17] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 04:08:15 UTC 2013 [04:08:17] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 04:08:15 UTC 2013 [04:08:26] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:26] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:26] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 04:08:25 UTC 2013 [04:08:36] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:36] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:36] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:36] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:36] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:37] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:46] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 04:08:36 UTC 2013 [04:08:46] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:46] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:46] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:46] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 04:08:42 UTC 2013 [04:08:56] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:56] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 04:08:50 UTC 2013 [04:09:06] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 04:08:57 UTC 2013 [04:09:06] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 04:08:57 UTC 2013 [04:09:06] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:26] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 04:09:15 UTC 2013 [04:09:26] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 04:09:21 UTC 2013 [04:09:26] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 04:09:21 UTC 2013 [04:09:26] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 04:09:21 UTC 2013 [04:09:26] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:36] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 04:09:26 UTC 2013 [04:09:36] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:36] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:36] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:36] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 04:09:32 UTC 2013 [04:09:46] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 04:09:41 UTC 2013 [04:09:46] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 04:09:41 UTC 2013 [04:09:46] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 04:09:42 UTC 2013 [04:09:46] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:46] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 04:09:43 UTC 2013 [04:09:56] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 04:09:46 UTC 2013 [04:09:56] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 04:09:47 UTC 2013 [04:09:56] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:56] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:56] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 04:09:54 UTC 2013 [04:10:06] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:06] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:06] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:06] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:06] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 04:09:58 UTC 2013 [04:10:07] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:16] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:16] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 04:10:12 UTC 2013 [04:10:16] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 04:10:13 UTC 2013 [04:10:26] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 04:10:15 UTC 2013 [04:10:26] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 04:10:18 UTC 2013 [04:10:26] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 04:10:20 UTC 2013 [04:10:26] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:26] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 04:10:22 UTC 2013 [04:10:36] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 04:10:27 UTC 2013 [04:10:36] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:36] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:36] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:36] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:37] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:37] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:38] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 04:10:31 UTC 2013 [04:10:38] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 04:10:31 UTC 2013 [04:10:39] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 04:10:31 UTC 2013 [04:10:39] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:40] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:40] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:46] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 04:10:35 UTC 2013 [04:10:46] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 04:10:35 UTC 2013 [04:10:46] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:46] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:46] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:56] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 04:10:46 UTC 2013 [04:10:56] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:56] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 04:10:54 UTC 2013 [04:11:06] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 04:10:56 UTC 2013 [04:11:06] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 04:10:57 UTC 2013 [04:11:06] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:06] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 04:11:03 UTC 2013 [04:11:06] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 04:11:05 UTC 2013 [04:11:07] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 04:11:05 UTC 2013 [04:11:07] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 04:11:05 UTC 2013 [04:11:16] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:27] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 04:11:15 UTC 2013 [04:11:27] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:27] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:27] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:27] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 04:11:22 UTC 2013 [04:11:28] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:36] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:36] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:36] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:36] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 04:11:28 UTC 2013 [04:11:36] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 04:11:28 UTC 2013 [04:11:37] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 04:11:34 UTC 2013 [04:11:37] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 04:11:35 UTC 2013 [04:11:48] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 04:11:38 UTC 2013 [04:11:48] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 04:11:38 UTC 2013 [04:11:48] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 04:11:40 UTC 2013 [04:11:48] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:48] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 04:11:42 UTC 2013 [04:11:49] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 04:11:44 UTC 2013 [04:11:56] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 04:11:48 UTC 2013 [04:11:56] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:56] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:56] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 04:11:52 UTC 2013 [04:11:56] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 04:11:54 UTC 2013 [04:12:07] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 04:11:56 UTC 2013 [04:12:07] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:07] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:07] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:07] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:07] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 04:11:59 UTC 2013 [04:12:07] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 04:11:59 UTC 2013 [04:12:08] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:08] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 04:12:04 UTC 2013 [04:12:16] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 04:12:08 UTC 2013 [04:12:17] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 04:12:09 UTC 2013 [04:12:27] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 04:12:18 UTC 2013 [04:12:27] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:27] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:36] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 04:12:27 UTC 2013 [04:12:36] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:36] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:36] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:36] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:37] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:37] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:38] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 04:12:29 UTC 2013 [04:12:38] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:39] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:39] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:40] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 04:12:35 UTC 2013 [04:12:47] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 04:12:36 UTC 2013 [04:12:47] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 04:12:36 UTC 2013 [04:12:47] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:47] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:47] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:48] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 04:12:43 UTC 2013 [04:12:56] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 04:12:47 UTC 2013 [04:12:56] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 04:12:48 UTC 2013 [04:12:56] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 04:12:48 UTC 2013 [04:12:56] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:58] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:58] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:08] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:08] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:08] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 04:13:05 UTC 2013 [04:13:16] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:16] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 04:13:10 UTC 2013 [04:13:16] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 04:13:12 UTC 2013 [04:13:16] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 04:13:13 UTC 2013 [04:13:16] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 04:13:13 UTC 2013 [04:13:17] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:26] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 04:13:16 UTC 2013 [04:13:26] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 04:13:16 UTC 2013 [04:13:26] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 04:13:17 UTC 2013 [04:13:26] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 04:13:17 UTC 2013 [04:13:26] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 04:13:19 UTC 2013 [04:13:27] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:27] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 04:13:24 UTC 2013 [04:13:28] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 04:13:25 UTC 2013 [04:13:37] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:37] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:37] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:37] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:37] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:38] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:38] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:39] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 04:13:28 UTC 2013 [04:13:39] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 04:13:28 UTC 2013 [04:13:40] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 04:13:30 UTC 2013 [04:13:40] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:46] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 04:13:36 UTC 2013 [04:13:46] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 04:13:37 UTC 2013 [04:13:46] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:46] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:46] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:47] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 04:13:39 UTC 2013 [04:13:49] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 04:13:40 UTC 2013 [04:13:49] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 04:13:40 UTC 2013 [04:13:49] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:49] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 04:13:44 UTC 2013 [04:13:49] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 04:13:44 UTC 2013 [04:13:56] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 04:13:49 UTC 2013 [04:13:56] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:56] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 04:13:50 UTC 2013 [04:13:56] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 04:13:54 UTC 2013 [04:13:56] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 04:13:55 UTC 2013 [04:14:06] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 04:13:56 UTC 2013 [04:14:06] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:06] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:06] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:06] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:07] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:07] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 04:14:00 UTC 2013 [04:14:08] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 04:14:00 UTC 2013 [04:14:08] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 04:14:00 UTC 2013 [04:14:09] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 04:14:01 UTC 2013 [04:14:09] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 04:14:01 UTC 2013 [04:14:10] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:10] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 04:14:04 UTC 2013 [04:14:16] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 04:14:06 UTC 2013 [04:14:16] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 04:14:06 UTC 2013 [04:14:16] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 04:14:07 UTC 2013 [04:14:16] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 04:14:07 UTC 2013 [04:14:16] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:17] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 04:14:09 UTC 2013 [04:14:17] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 04:14:09 UTC 2013 [04:14:18] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 04:14:10 UTC 2013 [04:14:18] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 04:14:14 UTC 2013 [04:14:19] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 04:14:15 UTC 2013 [04:14:19] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:26] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 04:14:16 UTC 2013 [04:14:26] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:26] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:26] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 04:14:19 UTC 2013 [04:14:26] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 04:14:19 UTC 2013 [04:14:27] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:36] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 04:14:26 UTC 2013 [04:14:36] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:36] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:36] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:36] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:37] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:37] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:38] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:38] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:39] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:39] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:40] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 04:14:29 UTC 2013 [04:14:40] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 04:14:29 UTC 2013 [04:14:41] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 04:14:30 UTC 2013 [04:14:41] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:42] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:42] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 04:14:34 UTC 2013 [04:14:43] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 04:14:34 UTC 2013 [04:14:46] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:46] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:46] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 04:14:38 UTC 2013 [04:14:46] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:46] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 04:14:40 UTC 2013 [04:14:47] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:47] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 04:14:42 UTC 2013 [04:14:48] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 04:14:43 UTC 2013 [04:14:48] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 04:14:43 UTC 2013 [04:14:56] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 04:14:45 UTC 2013 [04:14:56] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 04:14:48 UTC 2013 [04:14:56] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 04:14:48 UTC 2013 [04:14:56] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:56] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:57] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:57] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 04:14:51 UTC 2013 [04:14:58] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 04:14:51 UTC 2013 [04:14:58] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 04:14:53 UTC 2013 [04:15:06] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 04:14:59 UTC 2013 [04:15:06] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 04:14:59 UTC 2013 [04:15:06] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:06] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:06] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:07] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:07] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:08] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:08] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 04:15:04 UTC 2013 [04:15:09] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 04:15:04 UTC 2013 [04:15:09] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 04:15:05 UTC 2013 [04:15:16] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 04:15:06 UTC 2013 [04:15:16] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 04:15:06 UTC 2013 [04:15:16] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:16] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:26] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [04:15:26] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:26] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:26] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:36] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:36] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:37] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:37] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:37] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:37] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:37] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:38] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:38] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:39] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:39] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:40] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:46] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:46] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:46] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:46] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 04:15:39 UTC 2013 [04:15:46] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 04:15:39 UTC 2013 [04:15:47] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [04:16:26] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [04:16:26] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [04:17:56] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 04:17:55 UTC 2013 [04:17:56] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 04:17:55 UTC 2013 [04:18:06] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [04:18:26] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:46] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 04:20:40 UTC 2013 [04:20:57] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [04:23:08] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 04:23:05 UTC 2013 [04:23:37] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [04:23:37] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 04:23:33 UTC 2013 [04:23:37] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 04:23:33 UTC 2013 [04:23:57] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [04:24:06] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [04:25:56] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 04:25:51 UTC 2013 [04:26:36] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [04:26:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:26:46] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 04:26:43 UTC 2013 [04:26:46] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 04:26:43 UTC 2013 [04:26:56] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 04:26:54 UTC 2013 [04:27:16] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [04:27:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.231 second response time [04:27:36] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 04:27:26 UTC 2013 [04:27:36] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [04:27:57] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [04:28:07] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [04:29:46] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 04:29:36 UTC 2013 [04:30:18] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [04:31:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:32:08] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 04:32:00 UTC 2013 [04:32:08] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 04:32:00 UTC 2013 [04:32:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [04:32:36] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 04:32:27 UTC 2013 [04:32:36] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 04:32:28 UTC 2013 [04:32:36] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [04:32:46] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [04:33:08] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [04:33:08] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [04:34:56] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 04:34:47 UTC 2013 [04:35:37] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [04:35:37] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 04:35:30 UTC 2013 [04:36:09] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [04:36:49] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 04:36:42 UTC 2013 [04:37:09] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 04:37:07 UTC 2013 [04:37:29] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [04:37:39] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 04:37:38 UTC 2013 [04:37:49] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [04:38:39] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [04:38:40] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 04:38:31 UTC 2013 [04:38:40] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:29] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 04:40:20 UTC 2013 [04:40:29] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 04:40:28 UTC 2013 [04:40:29] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [04:40:49] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [04:43:29] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 04:43:28 UTC 2013 [04:43:29] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [04:43:49] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 04:43:38 UTC 2013 [04:44:00] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 04:43:49 UTC 2013 [04:44:39] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [04:44:49] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [04:45:49] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 04:45:43 UTC 2013 [04:45:59] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 04:45:48 UTC 2013 [04:46:19] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [04:46:19] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [04:47:09] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 04:46:58 UTC 2013 [04:47:29] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [04:47:39] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 04:47:29 UTC 2013 [04:47:59] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 04:47:50 UTC 2013 [04:48:14] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [04:48:29] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [04:50:39] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 04:50:36 UTC 2013 [04:50:59] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [04:52:59] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 04:52:51 UTC 2013 [04:53:42] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [04:53:59] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 04:53:55 UTC 2013 [04:54:10] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 04:54:00 UTC 2013 [04:54:10] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [04:54:49] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [04:55:50] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 04:55:39 UTC 2013 [04:56:29] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [04:56:49] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 04:56:47 UTC 2013 [04:56:49] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 04:56:47 UTC 2013 [04:56:59] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 04:56:52 UTC 2013 [04:57:19] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:29] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:29] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [04:57:39] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 04:57:30 UTC 2013 [04:58:09] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [04:59:40] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 04:59:35 UTC 2013 [05:00:19] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [05:01:59] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 05:01:53 UTC 2013 [05:01:59] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 05:01:53 UTC 2013 [05:02:29] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [05:02:29] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [05:02:39] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 05:02:30 UTC 2013 [05:03:01] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 05:02:57 UTC 2013 [05:03:09] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [05:03:09] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [05:04:39] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 05:04:37 UTC 2013 [05:04:40] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 05:04:37 UTC 2013 [05:05:31] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 05:05:26 UTC 2013 [05:05:32] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [05:05:32] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [05:06:10] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [05:06:40] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 05:06:39 UTC 2013 [05:06:50] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [05:07:50] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 05:07:42 UTC 2013 [05:08:31] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [05:08:50] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 05:08:40 UTC 2013 [05:09:40] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [05:10:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:10:50] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 05:10:42 UTC 2013 [05:10:51] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 05:10:42 UTC 2013 [05:10:51] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [05:11:30] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [05:11:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [05:11:30] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 05:11:23 UTC 2013 [05:11:30] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [05:13:30] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 05:13:29 UTC 2013 [05:13:50] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 05:13:35 UTC 2013 [05:13:50] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [05:14:42] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [05:15:00] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 05:14:51 UTC 2013 [05:15:30] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [05:15:50] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 05:15:46 UTC 2013 [05:15:50] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 05:15:46 UTC 2013 [05:16:20] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [05:16:20] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [05:17:50] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 05:17:45 UTC 2013 [05:17:50] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 05:17:45 UTC 2013 [05:18:10] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [05:18:30] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [05:20:30] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 05:20:29 UTC 2013 [05:20:40] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [05:23:00] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 05:22:54 UTC 2013 [05:23:30] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 05:23:25 UTC 2013 [05:23:30] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 05:23:25 UTC 2013 [05:23:30] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [05:23:50] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [05:23:50] PROBLEM - SSH on mc15 is CRITICAL: Connection timed out [05:24:10] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [05:24:42] RECOVERY - SSH on mc15 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [05:25:40] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 05:25:34 UTC 2013 [05:26:30] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [05:26:50] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 05:26:48 UTC 2013 [05:26:50] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 05:26:48 UTC 2013 [05:26:50] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 05:26:48 UTC 2013 [05:27:20] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [05:27:30] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [05:27:30] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [05:27:50] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 05:27:40 UTC 2013 [05:28:11] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [05:28:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:29:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [05:29:30] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 05:29:27 UTC 2013 [05:30:20] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [05:31:41] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 05:31:36 UTC 2013 [05:32:30] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [05:32:40] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 05:32:33 UTC 2013 [05:33:11] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 05:33:00 UTC 2013 [05:33:12] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [05:33:12] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [05:33:30] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [05:34:00] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 05:33:53 UTC 2013 [05:34:30] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [05:35:01] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 05:34:57 UTC 2013 [05:35:02] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 05:34:57 UTC 2013 [05:35:30] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [05:35:30] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [05:35:40] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 05:35:34 UTC 2013 [05:35:50] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [05:36:10] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [05:36:40] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 05:36:32 UTC 2013 [05:36:50] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [06:12:32] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [06:12:32] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [06:12:32] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [06:14:33] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [06:26:32] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:54] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 06:32:49 UTC 2013 [06:33:03] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 06:32:54 UTC 2013 [06:33:04] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 06:32:54 UTC 2013 [06:33:04] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 06:32:55 UTC 2013 [06:33:04] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 06:32:56 UTC 2013 [06:33:04] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 06:32:59 UTC 2013 [06:33:04] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 06:32:59 UTC 2013 [06:33:12] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 06:33:03 UTC 2013 [06:33:12] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 06:33:05 UTC 2013 [06:33:12] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 06:33:05 UTC 2013 [06:33:12] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 06:33:08 UTC 2013 [06:33:12] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 06:33:08 UTC 2013 [06:33:13] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:22] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 06:33:12 UTC 2013 [06:33:22] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 06:33:14 UTC 2013 [06:33:22] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 06:33:15 UTC 2013 [06:33:22] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 06:33:16 UTC 2013 [06:33:22] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 06:33:18 UTC 2013 [06:33:23] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 06:33:19 UTC 2013 [06:33:23] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 06:33:19 UTC 2013 [06:33:24] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 06:33:19 UTC 2013 [06:33:32] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 06:33:22 UTC 2013 [06:33:32] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:33] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:33] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 06:33:24 UTC 2013 [06:33:33] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 06:33:26 UTC 2013 [06:33:33] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 06:33:26 UTC 2013 [06:33:33] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 06:33:27 UTC 2013 [06:33:34] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 06:33:27 UTC 2013 [06:33:34] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 06:33:29 UTC 2013 [06:33:35] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 06:33:29 UTC 2013 [06:33:35] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:36] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:36] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:37] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:37] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:38] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:38] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:39] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:39] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 06:33:30 UTC 2013 [06:33:40] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:40] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:42] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 06:33:32 UTC 2013 [06:33:42] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 06:33:34 UTC 2013 [06:33:42] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:42] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:43] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:43] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 06:33:39 UTC 2013 [06:33:52] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 06:33:41 UTC 2013 [06:33:52] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 06:33:42 UTC 2013 [06:33:52] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 06:33:43 UTC 2013 [06:33:52] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 06:33:44 UTC 2013 [06:33:52] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 06:33:44 UTC 2013 [06:33:53] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:53] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:54] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:54] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:55] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:55] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 06:33:48 UTC 2013 [06:33:56] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:56] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:57] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 06:33:50 UTC 2013 [06:33:57] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [06:33:58] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 06:33:51 UTC 2013 [06:33:58] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 06:33:51 UTC 2013 [06:34:04] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 06:33:54 UTC 2013 [06:34:04] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:04] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:04] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 06:33:55 UTC 2013 [06:34:04] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 06:33:55 UTC 2013 [06:34:05] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 06:33:58 UTC 2013 [06:34:05] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:12] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 06:34:04 UTC 2013 [06:34:13] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 06:34:04 UTC 2013 [06:34:13] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:13] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:13] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 06:34:05 UTC 2013 [06:34:13] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 06:34:05 UTC 2013 [06:34:14] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:32] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:32] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:32] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:32] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:33] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:33] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:34] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:34] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:34] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:35] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:35] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:42] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:42] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:43] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 06:34:39 UTC 2013 [06:34:52] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:55] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [06:34:55] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 06:34:50 UTC 2013 [06:34:55] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:02] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:02] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:02] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:32] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [06:35:32] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:39] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.003035545349 secs [08:07:39] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 08:07:34 UTC 2013 [08:07:39] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 08:07:35 UTC 2013 [08:07:39] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 08:07:37 UTC 2013 [08:07:40] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 08:07:38 UTC 2013 [08:07:49] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [08:07:49] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 08:07:41 UTC 2013 [08:07:49] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 08:07:41 UTC 2013 [08:07:49] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 08:07:42 UTC 2013 [08:07:49] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 08:07:45 UTC 2013 [08:07:50] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 08:07:45 UTC 2013 [08:07:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 08:07:48 UTC 2013 [08:07:59] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [08:07:59] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 08:07:50 UTC 2013 [08:07:59] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 08:07:50 UTC 2013 [08:07:59] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 08:07:52 UTC 2013 [08:08:00] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 08:07:52 UTC 2013 [08:08:00] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:01] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 08:07:56 UTC 2013 [08:08:09] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 08:08:00 UTC 2013 [08:08:09] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 08:08:01 UTC 2013 [08:08:09] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 08:08:01 UTC 2013 [08:08:09] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 08:08:05 UTC 2013 [08:08:10] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 08:08:07 UTC 2013 [08:08:10] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 08:08:07 UTC 2013 [08:08:10] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 08:08:07 UTC 2013 [08:08:19] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:19] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 08:08:10 UTC 2013 [08:08:19] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:19] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 08:08:14 UTC 2013 [08:08:19] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:20] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:20] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 08:08:18 UTC 2013 [08:08:29] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 08:08:18 UTC 2013 [08:08:29] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 08:08:19 UTC 2013 [08:08:29] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 08:08:19 UTC 2013 [08:08:29] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:29] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 08:08:22 UTC 2013 [08:08:30] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 08:08:22 UTC 2013 [08:08:30] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:31] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:31] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:32] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 08:08:24 UTC 2013 [08:08:32] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:33] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:39] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 08:08:28 UTC 2013 [08:08:39] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 08:08:31 UTC 2013 [08:08:39] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:39] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:39] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:40] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:40] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:41] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:41] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:42] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:42] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:43] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:43] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 08:08:33 UTC 2013 [08:08:44] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:44] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:45] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 08:08:35 UTC 2013 [08:08:45] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:46] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 08:08:36 UTC 2013 [08:08:46] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 08:08:38 UTC 2013 [08:08:50] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 08:08:40 UTC 2013 [08:08:50] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:50] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:50] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:50] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 08:08:44 UTC 2013 [08:08:50] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 08:08:44 UTC 2013 [08:08:59] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 08:08:49 UTC 2013 [08:08:59] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:59] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:59] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:00] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 08:08:52 UTC 2013 [08:09:00] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:00] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 08:08:55 UTC 2013 [08:09:01] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 08:08:55 UTC 2013 [08:09:09] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 08:09:00 UTC 2013 [08:09:09] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 08:09:01 UTC 2013 [08:09:09] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 08:09:02 UTC 2013 [08:09:09] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 08:09:03 UTC 2013 [08:09:09] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 08:09:04 UTC 2013 [08:09:10] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 08:09:05 UTC 2013 [08:09:10] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 08:09:07 UTC 2013 [08:09:11] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 08:09:07 UTC 2013 [08:09:11] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 08:09:08 UTC 2013 [08:09:12] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 08:09:08 UTC 2013 [08:09:19] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:19] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 08:09:10 UTC 2013 [08:09:19] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 08:09:10 UTC 2013 [08:09:19] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 08:09:11 UTC 2013 [08:09:20] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 08:09:12 UTC 2013 [08:09:20] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 08:09:13 UTC 2013 [08:09:20] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:21] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 08:09:14 UTC 2013 [08:09:21] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:22] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:22] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 08:09:17 UTC 2013 [08:09:23] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 08:09:17 UTC 2013 [08:09:29] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:29] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 08:09:20 UTC 2013 [08:09:29] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 08:09:21 UTC 2013 [08:09:29] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:29] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 08:09:22 UTC 2013 [08:09:30] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:30] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:31] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:31] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:32] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 08:09:25 UTC 2013 [08:09:39] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 08:09:31 UTC 2013 [08:09:39] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 08:09:32 UTC 2013 [08:09:39] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:39] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:39] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:40] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:40] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:41] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:41] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:42] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:42] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:43] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:43] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:44] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:44] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:49] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 08:09:39 UTC 2013 [08:09:50] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 08:09:39 UTC 2013 [08:09:50] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 08:09:40 UTC 2013 [08:09:50] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 08:09:40 UTC 2013 [08:09:50] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:50] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:50] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:51] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 08:09:41 UTC 2013 [08:09:51] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 08:09:41 UTC 2013 [08:09:52] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 08:09:45 UTC 2013 [08:09:52] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 08:09:46 UTC 2013 [08:09:53] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 08:09:46 UTC 2013 [08:09:59] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 08:09:49 UTC 2013 [08:09:59] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:59] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:59] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:59] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 08:09:51 UTC 2013 [08:10:00] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 08:09:51 UTC 2013 [08:10:00] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 08:09:52 UTC 2013 [08:10:01] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 08:09:52 UTC 2013 [08:10:01] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 08:09:53 UTC 2013 [08:10:02] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:09] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 08:09:59 UTC 2013 [08:10:20] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:20] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:20] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:20] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:29] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:29] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:29] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:29] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:29] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:30] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 08:10:28 UTC 2013 [08:10:30] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 08:10:28 UTC 2013 [08:10:39] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:39] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:39] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:39] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:39] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:40] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:40] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:41] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:41] PROBLEM - Puppet freshness on amssq57 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:42] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:51] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:51] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:59] PROBLEM - Puppet freshness on amssq55 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:59] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [08:11:39] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 08:11:29 UTC 2013 [08:12:29] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:49] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 08:13:44 UTC 2013 [08:13:49] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 08:13:44 UTC 2013 [08:14:19] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:40] PROBLEM - Puppet freshness on amssq47 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:59] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 08:14:51 UTC 2013 [08:15:39] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:49] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 08:15:47 UTC 2013 [08:15:49] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 08:15:47 UTC 2013 [08:15:59] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:19] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [08:17:39] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 08:17:31 UTC 2013 [08:17:39] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [08:19:20] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 08:19:16 UTC 2013 [08:19:29] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:39] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 08:20:32 UTC 2013 [08:20:49] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:49] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 08:23:41 UTC 2013 [08:23:49] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 08:23:41 UTC 2013 [08:23:49] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 08:23:41 UTC 2013 [08:24:19] PROBLEM - Puppet freshness on amssq50 is CRITICAL: No successful Puppet run in the last 10 hours [08:24:19] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [08:24:39] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [08:25:29] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 08:25:25 UTC 2013 [08:25:39] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [08:26:49] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 08:26:44 UTC 2013 [08:26:49] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 08:26:44 UTC 2013 [08:27:29] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [08:27:39] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [08:27:59] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 08:27:52 UTC 2013 [08:28:29] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [08:28:41] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 08:28:38 UTC 2013 [08:28:49] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [08:28:49] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 08:28:43 UTC 2013 [08:29:49] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [08:31:40] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 08:31:37 UTC 2013 [08:32:01] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.0009427070618 secs [08:32:01] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 08:31:58 UTC 2013 [08:32:01] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 08:31:58 UTC 2013 [08:32:29] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [08:32:42] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 08:32:31 UTC 2013 [08:32:42] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 08:32:31 UTC 2013 [08:32:42] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [08:32:42] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [08:32:59] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [08:33:29] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [08:34:41] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 08:34:36 UTC 2013 [08:34:59] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 08:34:51 UTC 2013 [08:35:56] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [08:35:56] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [08:36:47] New patchset: Sanja pavlovic; "Per bug #48012. Changed code to give summary message about conf errors." [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/64095 [08:40:26] PROBLEM - SSH on lvs5 is CRITICAL: Server answer: [08:41:26] RECOVERY - SSH on lvs5 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [08:54:26] New patchset: coren; "Tool Labs: User requested package" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65403 [08:56:34] New review: coren; "Trivial addition" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/65403 [08:56:34] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65403 [08:58:25] New patchset: coren; "Tool Labs: Revert, mc was already there!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65404 [08:59:13] New review: coren; "I be silly" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/65404 [08:59:33] New review: coren; "I be silly" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/65404 [08:59:34] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65404 [09:02:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65306 [09:05:35] New review: Catrope; "What's the deal with 3128? I copied port 81 from cache::mobile, which also uses 81." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/63890 [09:07:07] New patchset: Mark Bergsma; "Remove amssq* from decommissioning" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65405 [09:08:01] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65405 [09:10:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:43] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65341 [09:12:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [09:17:50] New patchset: RobH; "adding oliver's secondary key" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65406 [09:22:11] New patchset: RobH; "adding oliver's secondary key" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65406 [09:22:32] damn it [09:22:40] i forgot to save and amended exact same thing. [09:22:41] New patchset: RobH; "adding oliver's secondary key" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65406 [09:22:41] heh. [09:26:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [09:31:03] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65406 [09:36:13] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [09:42:50] New patchset: Ottomata; "Adding CDH4 Zookeeper puppetization" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/65408 [09:45:29] New patchset: Mwalker; "Add AppServer Symlink for Special:Version" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65409 [09:48:59] New patchset: Mwalker; "Add AppServer Symlink for Special:Version" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65409 [09:49:44] Ryan_Lane: when you have a free moment -- I would highly appreciate you taking a look at ^ :) [09:51:53] PROBLEM - Host lanthanum is DOWN: CRITICAL - Host Unreachable (208.80.154.13) [09:52:48] New review: Diederik; "Let's merge this." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/54116 [09:57:57] mwalker: soooooo…. :) [09:58:03] mwalker: talked with RoanKattouw about this [09:58:21] /a is used randomly for things [09:58:24] damnit [09:58:27] in some cases it's a mounted filesystem [09:58:27] on the app servers? [09:58:49] I'd rather not put this in puppet as it's possibly inconsistent [09:59:17] that's fair -- any other ideas then? [09:59:18] when we switch deployment systems this'll fixitself [09:59:27] but that's 6 months away! [09:59:36] and I'm impatient :) [10:00:05] RoanKattouw: do we ever mount something in /a on an appserver? [10:00:32] On app servers, not sure [10:00:44] There might be some remnants of notpeter's work that resulted in mounted /a's [10:00:47] And scalers use /a [10:01:08] hurm [10:03:36] RoanKattouw: yeah. I didn't want to delete it all [10:03:41] so I just split it [10:03:51] and the scalers use /a a little still, ithink [10:03:57] Isn't /tmp in it? [10:04:04] no [10:04:17] it's like /a/magick-tmp [10:04:46] notpeter: I noticed that /a isn't a mounted filesystem on some of these [10:05:29] root@mw75:~# ls /a [10:05:29] magick-tmp [10:05:56] oh, yeah [10:06:00] nvm, I'm being dumb [10:06:08] the scalers still have a /a [10:06:15] none have a /a partition [10:07:29] !log Gracefully reloading Zuul to deploy I8c95150a688973 [10:07:38] Logged the message, Master [10:08:12] do the image scalers use the applicationserver manifest? [10:10:05] hexmode: Reedy already on it ? or i can copy the files for 1.21.0 [10:10:34] notpeter: why use /a if there's no partition? [10:10:46] mutante: I could use your help... there is an RT ticket. direct url 1s [10:11:04] mutante: http://mah.everybody.org/upload-1.21.0.tar [10:15:05] Ryan_Lane: because just creating a /a for imagescalers was easier than changing the scaler config to use a different tem folder [10:15:12] because legacy [10:15:15] * Ryan_Lane nods [10:15:32] I have a feeling I asked about this in the past :D [10:15:48] seems likely. feel free to change it :) [10:17:17] heh [10:17:18] indeed [10:17:23] I may at some point [10:17:53] "Because Legacy". Hysterical raisins cause so much trouble. [10:20:05] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [10:20:05] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [10:20:05] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [10:23:05] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [10:23:05] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [10:32:44] mutante: any luck? [10:33:00] hexmode: re.. sorry got distracted but done right the second [10:33:23] http://dumps.wikimedia.org/mediawiki/1.21/ [10:33:27] heh [10:33:32] ty, will check [10:37:49] Reedy, Ryan_Lane: so I was to change the image magick dir -- I would modify the puppet manifest to ensure that both /a/magick-tmp and /tmp/magic-tmp are present and created; wait 3 hours, change the MW config, push that; and then modify the puppet manifest to no longer create /a/magick-tmp? [10:37:57] *if I [10:43:57] mutante: verified [10:48:38] hexmode: :) [10:49:49] sanity test installation worked. Now to send out the email [10:50:19] hexmode: what's the conclusion on job queue? [10:50:51] I think I left my comments on the ticket.... let me check. [10:51:36] but the conclusion is essentiall "this isn't enough to block release' [10:51:51] Nemo_bis: you broke it [10:52:14] Reedy: and I will beheaded? [10:52:19] oh, no! [10:52:33] I just found a minor bug :( [10:52:42] but I'm still shipping :P [10:52:49] hexmode: then you should close the bug invalid... [10:52:54] or at least update mileston [10:52:56] e [10:52:59] yes [10:53:11] I'll look at them in a bit [10:55:24] New patchset: Mwalker; "Starting ImageScaler tmp dir move from /a to /tmp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65423 [10:58:21] wrong channel, moving to -dev [11:06:30] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:23] Can someone please do g+w on /a/common/php-1.22wmf4/.git/objects/31 and /a/common/php-1.22wmf4/.git/objects/9e on tin please? [11:10:24] New patchset: Ottomata; "Adding classes to install hive, pig and sqoop. (very simple!)" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/65267 [11:16:30] PROBLEM - RAID on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:17:20] RECOVERY - RAID on mc15 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [11:42:24] New patchset: RobH; "bringing racktables stuff to style guide standard" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65429 [11:42:28] PROBLEM - Varnish HTCP daemon on cp1029 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 997 (varnishhtcpd), args varnishhtcpd worker [11:43:51] hashar: so zuul/jenkins used to run puppet-lint in puppet/operations [11:43:53] but now seems it doesnt [11:44:03] know about this? (was wondering why it was changed) [11:44:32] mwalker: we'd want to verify the change actually occurred everywhere first [11:45:05] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65429 [11:47:05] RobH: what is the change ? [11:47:20] RobH: it might skip it in some cases, like when you only change .erb templates [11:47:30] ahh, my change above was actual file.pp changes [11:47:33] and it didnt run lint [11:47:38] 65429 [11:48:38] [12:09:23] Can someone please do g+w on /a/common/php-1.22wmf4/.git/objects/31 and /a/common/php-1.22wmf4/.git/objects/9e on tin please? [11:49:56] Reedy: On it [11:50:00] Ta [11:50:05] Ryan_Lane: certainly -- but there's only 8 image scalers [11:50:10] Need to kick kaldari to fix his usermask again [11:50:29] Actually, can you fix it at the same time? ;) [11:50:35] ie edit his .bashrc [11:51:01] Reedy: Done [11:51:07] Let me see about that [11:51:10] New patchset: Ottomata; "Adding CDH4 Zookeeper puppetization" [operations/puppet/cdh4] (master) - https://gerrit.wikimedia.org/r/65408 [11:51:22] Aha [11:51:24] umask 022 in his .bashrc [11:51:26] we did apply the fix in puppet.. unless it didnt work on tin/terbium [11:51:35] ah, that would conflict [11:51:35] I think he already had a user account at that point [11:51:37] mutante: His ~/.bashrc overrides it [11:51:38] so it wouldn't take effect [11:51:39] ack [11:51:47] yea [11:51:59] I have root so I could edit it out, but I'm not gonna do that without permission [11:52:11] Fetching submodule extensions/CentralAuth [11:52:11] error: The requested URL returned error: 403 while accessing https://gerrit.wikimedia.org/r/p/mediawiki/extensions/CentralAuth.git/info/refs [11:52:11] fatal: HTTP request failed [11:52:12] Rage [11:52:22] wtf [11:52:28] How the hell does that 403 [11:52:32] I've no idea [11:52:42] It just started [11:52:44] gets a 406 [11:52:47] Some extensions were ok, some don't [11:52:54] WFM on fenari [11:52:56] " Not Acceptable" [11:52:57] hah [11:53:09] Sounds right for CA [11:53:27] mutante: That's probably because you're using HTTP GET rather than the weird-ass crap git uses [11:54:58] all i did was open Reedys link [11:55:17] Yeah, it's not designed for web browsing [11:56:28] oh, slow , gotcha now [11:56:48] It's starting to get rather annoying [11:57:16] currently having to update on fenari, and move across to tin, sync-dir.. [12:03:33] !log cp1029.eqiad : manually testing s/varnishhtcpd/vhtcpd/ [12:03:42] Logged the message, Master [12:07:00] !log reedy synchronized php-1.22wmf4/extensions/CentralAuth [12:07:08] Logged the message, Master [12:07:47] RECOVERY - Puppet freshness on amssq57 is OK: puppet ran at Sat May 25 12:07:35 UTC 2013 [12:07:47] RECOVERY - Puppet freshness on amssq55 is OK: puppet ran at Sat May 25 12:07:37 UTC 2013 [12:07:47] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 12:07:38 UTC 2013 [12:07:47] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 12:07:40 UTC 2013 [12:07:56] RECOVERY - Puppet freshness on amssq53 is OK: puppet ran at Sat May 25 12:07:51 UTC 2013 [12:07:56] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:07:51 UTC 2013 [12:08:07] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 12:07:57 UTC 2013 [12:08:16] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:16] RECOVERY - Puppet freshness on amssq47 is OK: puppet ran at Sat May 25 12:08:10 UTC 2013 [12:08:16] RECOVERY - Puppet freshness on amssq48 is OK: puppet ran at Sat May 25 12:08:11 UTC 2013 [12:08:16] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 12:08:13 UTC 2013 [12:08:26] RECOVERY - Puppet freshness on amssq56 is OK: puppet ran at Sat May 25 12:08:16 UTC 2013 [12:08:26] RECOVERY - Puppet freshness on amssq51 is OK: puppet ran at Sat May 25 12:08:16 UTC 2013 [12:08:26] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 12:08:21 UTC 2013 [12:08:26] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 12:08:21 UTC 2013 [12:08:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:27] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:27] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:38] RECOVERY - Puppet freshness on amssq59 is OK: puppet ran at Sat May 25 12:08:27 UTC 2013 [12:08:46] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:47] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:47] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 12:08:40 UTC 2013 [12:08:47] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:57] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:57] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 12:08:48 UTC 2013 [12:08:57] RECOVERY - Puppet freshness on amssq50 is OK: puppet ran at Sat May 25 12:08:48 UTC 2013 [12:09:06] RECOVERY - Puppet freshness on amssq60 is OK: puppet ran at Sat May 25 12:09:00 UTC 2013 [12:09:06] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 12:09:02 UTC 2013 [12:09:06] RECOVERY - Puppet freshness on amssq58 is OK: puppet ran at Sat May 25 12:09:03 UTC 2013 [12:09:18] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 12:09:09 UTC 2013 [12:09:18] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 12:09:11 UTC 2013 [12:09:18] RECOVERY - Puppet freshness on amssq62 is OK: puppet ran at Sat May 25 12:09:11 UTC 2013 [12:09:27] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:27] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:27] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:27] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:37] RECOVERY - Puppet freshness on amssq54 is OK: puppet ran at Sat May 25 12:09:27 UTC 2013 [12:09:38] RECOVERY - Puppet freshness on amssq49 is OK: puppet ran at Sat May 25 12:09:27 UTC 2013 [12:09:38] RECOVERY - Puppet freshness on amssq61 is OK: puppet ran at Sat May 25 12:09:33 UTC 2013 [12:09:38] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 12:09:33 UTC 2013 [12:09:46] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 12:09:43 UTC 2013 [12:09:47] RECOVERY - Puppet freshness on amssq52 is OK: puppet ran at Sat May 25 12:09:43 UTC 2013 [12:09:57] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 12:09:51 UTC 2013 [12:09:57] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 12:09:52 UTC 2013 [12:10:07] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:10:01 UTC 2013 [12:10:07] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 12:10:03 UTC 2013 [12:10:18] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:29] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:29] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:36] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:36] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 12:10:32 UTC 2013 [12:10:46] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 12:10:38 UTC 2013 [12:10:46] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 12:10:38 UTC 2013 [12:10:46] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:46] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:46] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 12:10:39 UTC 2013 [12:10:47] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:06] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 12:10:56 UTC 2013 [12:11:06] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 12:11:04 UTC 2013 [12:11:26] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:26] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 12:11:21 UTC 2013 [12:11:26] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:26] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:27] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 12:11:24 UTC 2013 [12:11:27] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:36] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 12:11:30 UTC 2013 [12:11:46] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:57] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:06] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 12:11:59 UTC 2013 [12:12:06] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 12:12:04 UTC 2013 [12:12:16] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 12:12:12 UTC 2013 [12:12:16] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 12:12:15 UTC 2013 [12:12:26] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:12:18 UTC 2013 [12:12:26] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:26] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:26] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 12:12:25 UTC 2013 [12:12:26] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:36] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:48] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:48] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:49] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 12:12:44 UTC 2013 [12:13:06] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 12:12:57 UTC 2013 [12:13:06] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 12:12:57 UTC 2013 [12:13:16] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:26] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:26] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 12:13:24 UTC 2013 [12:13:37] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 12:13:28 UTC 2013 [12:13:46] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:46] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:46] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 12:13:39 UTC 2013 [12:13:46] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 12:13:41 UTC 2013 [12:13:46] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 12:13:42 UTC 2013 [12:13:56] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:17] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 12:14:13 UTC 2013 [12:14:17] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 12:14:14 UTC 2013 [12:14:26] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:26] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 12:14:21 UTC 2013 [12:14:26] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:26] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 12:14:23 UTC 2013 [12:14:26] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:27] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:27] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:36] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:14:27 UTC 2013 [12:14:36] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 12:14:29 UTC 2013 [12:14:36] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:36] RECOVERY - Puppet freshness on erzurumi is OK: puppet ran at Sat May 25 12:14:30 UTC 2013 [12:14:47] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 12:14:36 UTC 2013 [12:14:47] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:47] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 12:14:41 UTC 2013 [12:14:47] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 12:14:41 UTC 2013 [12:14:47] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:00] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 12:14:52 UTC 2013 [12:15:06] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 12:14:56 UTC 2013 [12:15:17] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 12:15:07 UTC 2013 [12:15:17] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 12:15:07 UTC 2013 [12:15:17] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:17] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 12:15:09 UTC 2013 [12:15:27] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 12:15:19 UTC 2013 [12:15:27] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:27] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:27] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:27] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:28] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 12:15:22 UTC 2013 [12:15:28] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:29] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:29] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:30] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:37] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:37] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 12:15:31 UTC 2013 [12:15:43] whoo puppet! [12:15:47] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 12:15:37 UTC 2013 [12:15:48] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:48] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:48] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:15:41 UTC 2013 [12:15:48] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 12:15:42 UTC 2013 [12:15:48] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:56] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 12:15:47 UTC 2013 [12:15:56] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:56] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 12:15:51 UTC 2013 [12:15:56] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 12:15:51 UTC 2013 [12:16:10] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 12:15:59 UTC 2013 [12:16:10] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 12:16:00 UTC 2013 [12:16:18] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:18] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 12:16:11 UTC 2013 [12:16:18] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 12:16:11 UTC 2013 [12:16:18] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 12:16:12 UTC 2013 [12:16:27] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:27] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:27] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:27] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:28] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:28] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 12:16:25 UTC 2013 [12:16:28] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:29] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:36] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 12:16:28 UTC 2013 [12:16:36] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:36] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 12:16:34 UTC 2013 [12:16:46] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:46] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:16:39 UTC 2013 [12:16:46] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 12:16:39 UTC 2013 [12:16:46] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:46] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 12:16:43 UTC 2013 [12:16:56] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 12:16:46 UTC 2013 [12:16:57] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 12:16:49 UTC 2013 [12:16:57] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 12:16:49 UTC 2013 [12:16:57] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:08] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 12:16:56 UTC 2013 [12:17:08] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 12:16:57 UTC 2013 [12:17:08] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 12:17:03 UTC 2013 [12:17:08] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 12:17:03 UTC 2013 [12:17:18] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 12:17:06 UTC 2013 [12:17:18] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:26] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 12:17:19 UTC 2013 [12:17:26] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:26] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:26] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:26] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:27] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:27] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 12:17:23 UTC 2013 [12:17:28] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:28] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:29] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:37] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:17:26 UTC 2013 [12:17:37] RECOVERY - Puppet freshness on knsq18 is OK: puppet ran at Sat May 25 12:17:28 UTC 2013 [12:17:37] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:37] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 12:17:32 UTC 2013 [12:17:37] RECOVERY - Puppet freshness on knsq26 is OK: puppet ran at Sat May 25 12:17:34 UTC 2013 [12:17:46] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:47] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:47] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:47] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 12:17:42 UTC 2013 [12:17:56] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 12:17:48 UTC 2013 [12:17:56] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 12:17:48 UTC 2013 [12:17:56] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:56] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 12:17:51 UTC 2013 [12:18:06] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 12:17:56 UTC 2013 [12:18:06] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 12:18:01 UTC 2013 [12:18:16] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 12:18:10 UTC 2013 [12:18:16] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:16] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:18:12 UTC 2013 [12:18:27] RECOVERY - Puppet freshness on knsq27 is OK: puppet ran at Sat May 25 12:18:17 UTC 2013 [12:18:27] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:27] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:27] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:27] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:27] PROBLEM - Puppet freshness on knsq26 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:27] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:28] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:28] PROBLEM - Puppet freshness on knsq18 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:37] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:46] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:46] PROBLEM - Puppet freshness on knsq27 is CRITICAL: No successful Puppet run in the last 10 hours [12:18:46] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:20:00] who killed puppet :) [12:20:47] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:20:37 UTC 2013 [12:20:48] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:20:59] see above [12:21:14] icinga snmp hook being broken? [12:23:37] RECOVERY - Puppet freshness on knsq21 is OK: puppet ran at Sat May 25 12:23:32 UTC 2013 [12:23:37] RECOVERY - Puppet freshness on knsq22 is OK: puppet ran at Sat May 25 12:23:32 UTC 2013 [12:23:56] PROBLEM - Puppet freshness on knsq21 is CRITICAL: No successful Puppet run in the last 10 hours [12:24:08] LeslieCarr: ping [12:24:27] PROBLEM - Puppet freshness on knsq22 is CRITICAL: No successful Puppet run in the last 10 hours [12:26:56] RECOVERY - Puppet freshness on knsq17 is OK: puppet ran at Sat May 25 12:26:49 UTC 2013 [12:26:56] RECOVERY - Puppet freshness on knsq19 is OK: puppet ran at Sat May 25 12:26:54 UTC 2013 [12:27:26] PROBLEM - Puppet freshness on knsq19 is CRITICAL: No successful Puppet run in the last 10 hours [12:27:26] PROBLEM - Puppet freshness on knsq17 is CRITICAL: No successful Puppet run in the last 10 hours [12:27:37] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 12:27:25 UTC 2013 [12:28:27] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:31:29] paravoid: pong [12:31:49] hrm, hrm [12:32:14] oh decommissioned [12:32:24] that is why [12:32:29] stupid decomissioned [12:32:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:06] RECOVERY - Puppet freshness on knsq23 is OK: puppet ran at Sat May 25 12:33:02 UTC 2013 [12:33:16] RECOVERY - Puppet freshness on knsq29 is OK: puppet ran at Sat May 25 12:33:07 UTC 2013 [12:33:27] PROBLEM - Puppet freshness on knsq29 is CRITICAL: No successful Puppet run in the last 10 hours [12:33:27] PROBLEM - Puppet freshness on knsq23 is CRITICAL: No successful Puppet run in the last 10 hours [12:33:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [12:34:36] RECOVERY - Puppet freshness on knsq20 is OK: puppet ran at Sat May 25 12:34:32 UTC 2013 [12:35:36] PROBLEM - Puppet freshness on knsq20 is CRITICAL: No successful Puppet run in the last 10 hours [12:36:36] or wait [12:36:39] they were once decom'ed [12:37:18] LeslieCarr: https://gerrit.wikimedia.org/r/#/q/c7501d4178b5d1339c2ad84eea89b9ec6a4c4dc8,n,z [12:37:25] https://gerrit.wikimedia.org/r/#/c/65405/ [12:37:34] oh, that's knsq [12:37:36] never mind [12:37:40] they still are decommissioned [12:38:18] knsq23 and knsq29 aren't [12:38:20] RECOVERY - Puppet freshness on knsq24 is OK: puppet ran at Sat May 25 12:38:13 UTC 2013 [12:38:21] or aren't in decom.pp [12:38:27] New patchset: Faidon; "Rsync public datasets for visualization to stat1001" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116 [12:38:28] mark, paravoid want to do it now? [12:38:29] neither is 17 or 19 [12:38:38] huh [12:38:41] RECOVERY - Puppet freshness on knsq28 is OK: puppet ran at Sat May 25 12:38:34 UTC 2013 [12:38:41] PROBLEM - Puppet freshness on knsq28 is CRITICAL: No successful Puppet run in the last 10 hours [12:38:43] i put them in yesterday [12:38:45] oh wait, i didn't git pull [12:38:51] that may explain it :) [12:39:00] PROBLEM - Puppet freshness on knsq24 is CRITICAL: No successful Puppet run in the last 10 hours [12:39:09] look at that [12:39:12] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54116 [12:40:45] mark - meeting with MaxSem [12:40:54] yes ok [12:41:10] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 12:41:00 UTC 2013 [12:41:40] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [12:42:47] !log authdns-update [12:42:56] Logged the message, Master [12:45:51] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 12:45:47 UTC 2013 [12:45:58] !log reedy synchronized php-1.22wmf4/extensions/CentralAuth [12:46:06] Logged the message, Master [12:46:33] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [12:50:15] New patchset: Dzahn; "add virtual language subdomain redirects for wikidata" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [12:52:23] New review: Dzahn; "dzahn@fenari:~$ apache-fast-test wikidata.url mw1044" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [13:03:20] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [13:09:39] Can someone please on fenari rm -rf /home/wikipedia/common/php-1.21wmf12 [13:10:24] Reedy: okay [13:10:40] New patchset: Reedy; "Point php to 1.22wmf4" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65445 [13:10:49] Reedy: done [13:10:55] thanks [13:11:00] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/65445 [13:11:01] !log rm -rf /home/wikipedia/common/php-1.21wmf12 on fenari [13:11:09] Logged the message, Mistress of the network gear. [13:12:00] New patchset: JanZerebecki; "add virtual language subdomain redirects for wikidata" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [13:29:56] Hi, I am new _here_ [13:30:09] I have a question, kind request, reminder... [13:30:26] some years ago, I solicited and became maintainer of https://www.mediawiki.org/wiki/Extension:UserMerge [13:30:35] But when that was moved to git from svn, [13:30:54] it was apparently forgotten to set me up as project owner there [13:31:15] I already https://www.mediawiki.org/wiki/Gerrit/Project_ownership#Wikinaut_for_extension_UserMerge_and_Delete [13:31:23] filed my request, but nothing happened. [13:31:47] Can someone in ops please set me up as maintainer, so that this request can be closed on https://www.mediawiki.org/wiki/Gerrit/Project_ownership#Wikinaut_for_extension_UserMerge_and_Delete [13:31:48] ? [13:32:07] PROBLEM - Auth DNS on ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:35:22] !log restarting pdns on ns0 [13:35:30] Logged the message, Master [13:36:02] RECOVERY - Auth DNS on ns0.wikimedia.org is OK: DNS OK: 0.483 seconds response time. www.wikipedia.org returns 208.80.154.225 [13:47:23] thanks mutante [13:47:23] New patchset: Dzahn; "Fold wikiquote rewrites into one." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/53403 [13:49:18] mutante: try not to redirect all the cluster to wikiquote ^^ [13:49:20] SCNR [13:53:31] New patchset: Faidon; "Varnish: fixes to opera-mini client.ip replacement" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65454 [13:55:57] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65454 [13:57:10] Nemo_bis: looks pretty good though:) [13:57:29] that was a rebase [14:01:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [14:18:31] mutante: still around? [14:35:07] Ryan_Lane: ping [14:35:10] pong [14:35:14] yessir [14:35:39] Ryan_Lane: could you reupload my tarball? I found a problem with it [14:36:21] Ryan_Lane: so http://mah.everybody.org/upload-1.21.0.tar has new stuff that needs to move to dumps.wikimedia.org [14:39:08] Ryan_Lane: ?? [14:39:18] ummm [14:39:31] chris told me you could do it [14:39:34] did he lie? [14:39:38] ah [14:39:50] I'd like someone to approve this :) [14:40:10] do you want robla or chris to come smack you? [14:44:36] hexmode: ? [14:45:01] mutante: Ryan_Lane helped me [14:45:16] ok,cool [14:45:21] mutante: I found a brown paper bag problem with the tarball [14:45:27] New patchset: Dzahn; "add "/entity/" redirects for wikidata per" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65463 [14:46:16] hexmode: heh, i looked that up in jargon file, gotcha [14:50:35] !log authdns-update [14:50:44] Logged the message, Master [15:00:50] mutante: still around? [15:04:13] hexmode: done [15:07:34] yay! [15:17:59] New patchset: Ottomata; "test commit, do not merge." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65470 [15:19:15] Change abandoned: Ottomata; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/65470 [16:08:45] RECOVERY - Puppet freshness on db44 is OK: puppet ran at Sat May 25 16:08:37 UTC 2013 [16:08:45] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 16:08:42 UTC 2013 [16:08:55] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 16:08:54 UTC 2013 [16:09:29] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:45] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 16:11:39 UTC 2013 [16:12:07] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 16:11:58 UTC 2013 [16:12:25] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:37] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:27] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:27] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:27] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:15] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 16:15:05 UTC 2013 [16:15:26] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:26] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:35] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 16:15:28 UTC 2013 [16:15:35] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:15] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 16:17:14 UTC 2013 [16:17:25] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 16:17:22 UTC 2013 [16:17:38] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:40] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:57] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 16:18:52 UTC 2013 [16:19:15] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 16:19:06 UTC 2013 [16:19:36] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:36] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:25] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 16:20:16 UTC 2013 [16:20:25] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 16:20:22 UTC 2013 [16:20:36] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:20:36] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:27] RECOVERY - Puppet freshness on knsq16 is OK: puppet ran at Sat May 25 16:21:19 UTC 2013 [16:21:37] PROBLEM - Puppet freshness on knsq16 is CRITICAL: No successful Puppet run in the last 10 hours [16:21:37] RECOVERY - Puppet freshness on ms2 is OK: puppet ran at Sat May 25 16:21:29 UTC 2013 [16:21:37] PROBLEM - Puppet freshness on ms2 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:25] PROBLEM - Puppet freshness on db1017 is CRITICAL: No successful Puppet run in the last 10 hours [16:54:06] RECOVERY - Puppet freshness on db1017 is OK: puppet ran at Sat May 25 16:54:00 UTC 2013 [17:21:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [17:26:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [17:39:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:40:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.021 second response time [18:21:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [18:38:21] RECOVERY - search indices - check lucene status page on search19 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.111 second response time [19:43:05] New review: Odder; "As a test commit, this looks OK, but as MZMcBride point out, you left a typo behind ;-)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/64635 [19:51:34] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:51:35] PROBLEM - DPKG on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:52:34] RECOVERY - DPKG on searchidx1001 is OK: All packages OK [19:54:24] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [20:20:32] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [20:20:32] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [20:20:32] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [20:23:32] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [20:23:32] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [20:56:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [21:07:13] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [21:27:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:29:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.151 second response time [21:47:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [21:56:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [21:59:14] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:00:03] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [22:00:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.272 second response time [22:14:12] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [22:15:11] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [22:18:11] PROBLEM - Puppet freshness on cp1029 is CRITICAL: No successful Puppet run in the last 10 hours [22:47:55] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [23:32:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:33:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.180 second response time [23:35:36] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [23:42:42] New patchset: Hazard-SJ; "I fixed a few typographical errors." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/64635 [23:57:09] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [23:57:40] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.56 ms