[00:09:50] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [00:10:55] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:24] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [01:00:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [01:07:26] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [01:07:56] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [01:08:06] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:56] PROBLEM - RAID on analytics1010 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:11:56] RECOVERY - RAID on analytics1010 is OK: OK: Active: 6, Working: 6, Failed: 0, Spare: 0 [01:15:06] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [01:22:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [01:39:33] PROBLEM - Host mw72 is DOWN: PING CRITICAL - Packet loss = 100% [01:40:34] RECOVERY - Host mw72 is UP: PING OK - Packet loss = 0%, RTA = 26.56 ms [01:52:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [02:05:10] !log LocalisationUpdate completed (1.22wmf8) at Sun Jul 7 02:05:09 UTC 2013 [02:05:22] Logged the message, Master [02:06:39] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [02:06:59] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [02:09:01] !log LocalisationUpdate completed (1.22wmf9) at Sun Jul 7 02:09:01 UTC 2013 [02:09:14] Logged the message, Master [02:14:42] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 7 02:14:42 UTC 2013 [02:14:59] Logged the message, Master [02:22:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [03:08:12] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [03:08:32] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [03:32:42] PROBLEM - RAID on ms-be3 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:06:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:06:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:07:50] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:07:48 UTC 2013 [04:08:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:40] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:08:39 UTC 2013 [04:09:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:09:50] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:09:42 UTC 2013 [04:10:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:40] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:10:33 UTC 2013 [04:11:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:11:40] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:11:33 UTC 2013 [04:12:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:12:30] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:12:20 UTC 2013 [04:13:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:13:20] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:13:13 UTC 2013 [04:14:00] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:13:56 UTC 2013 [04:14:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:15:00] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:14:56 UTC 2013 [04:15:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:15:40] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:15:31 UTC 2013 [04:16:01] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:16:21] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:16:19 UTC 2013 [04:17:00] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:16:54 UTC 2013 [04:17:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:17:50] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:17:42 UTC 2013 [04:18:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:18:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:18:20] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:18:18 UTC 2013 [04:19:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:19:10] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:19:03 UTC 2013 [04:19:30] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:19:29 UTC 2013 [04:20:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:20:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:10] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:20:09 UTC 2013 [04:20:40] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:20:36 UTC 2013 [04:21:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:21:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:20] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:21:13 UTC 2013 [04:21:40] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:21:39 UTC 2013 [04:22:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:22:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:22:10] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:22:08 UTC 2013 [04:22:40] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:22:32 UTC 2013 [04:23:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:23:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:23:21] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:23:19 UTC 2013 [04:23:50] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:23:41 UTC 2013 [04:24:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:24:00] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:23:57 UTC 2013 [04:24:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:24:30] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 04:24:21 UTC 2013 [04:25:01] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:25:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:30:10] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 04:30:04 UTC 2013 [04:31:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [05:05:40] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [05:06:10] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [05:21:30] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [05:21:30] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [05:21:30] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [05:21:30] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [05:21:30] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [05:21:31] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [05:21:31] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [05:21:32] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [05:58:00] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:58:50] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [06:06:14] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [06:06:44] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:22:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [06:26:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:27:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [06:30:44] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 06:30:40 UTC 2013 [06:31:04] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 06:31:02 UTC 2013 [06:31:14] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [06:31:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:44] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:54] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 06:31:46 UTC 2013 [06:32:14] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [06:32:14] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 06:32:08 UTC 2013 [06:32:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [06:32:34] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 06:32:25 UTC 2013 [06:32:44] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:44] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 06:32:43 UTC 2013 [06:33:14] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [06:33:44] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [07:06:36] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [07:06:46] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [07:22:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:24:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [07:31:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [08:06:35] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [08:06:44] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [08:07:54] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 08:07:47 UTC 2013 [08:08:14] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 08:08:07 UTC 2013 [08:08:34] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:34] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 08:08:31 UTC 2013 [08:08:44] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [08:08:54] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 08:08:47 UTC 2013 [08:09:34] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:44] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [08:16:34] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 08:16:25 UTC 2013 [08:17:34] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [08:29:54] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 08:29:50 UTC 2013 [08:30:44] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [08:31:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:32:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [09:16:26] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 09:16:25 UTC 2013 [09:16:46] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [09:29:56] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 09:29:48 UTC 2013 [09:30:56] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [09:41:31] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:42:21] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [09:47:11] New patchset: Amire80; "Remove unnecessary ULS IME selector from VE headings menu" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/72356 [10:01:31] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:02:21] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [10:11:08] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [10:11:08] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [10:11:08] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [10:23:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.147 second response time [10:24:28] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:34:18] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [10:35:08] PROBLEM - Disk space on wtp1021 is CRITICAL: DISK CRITICAL - free space: / 355 MB (3% inode=78%): [11:06:18] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [11:06:28] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [11:08:58] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [11:15:58] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [11:17:05] Reedy: was reading now bug 30413, and the next to last comment now really shows why I can be a bit frustrated: "Thanks to Platonides using the email interface, there is now RT #4076 about publicly documenting RT usage. There hasn't been much progress yet though." [11:17:31] didn't find that bug before [11:17:49] anyway, imo RT is a too much of a black hole for non-ops [11:18:47] I can't fathom why something like documenting RT usage would need to have to be kept from public eyes... [11:19:16] does Andre Klapper hang out on IRC? [11:22:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [11:27:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.148 second response time [11:33:41] he does. [11:33:59] AzaToth: under andre__ [11:34:28] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:35:18] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [11:39:07] ok [11:52:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [12:01:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [12:07:09] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [12:07:39] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [12:07:59] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 12:07:54 UTC 2013 [12:08:29] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 12:08:22 UTC 2013 [12:08:39] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [12:08:59] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 12:08:51 UTC 2013 [12:09:09] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [12:09:19] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 12:09:16 UTC 2013 [12:09:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:39] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:49] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 12:09:43 UTC 2013 [12:10:09] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [12:10:09] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 12:10:03 UTC 2013 [12:10:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.146 second response time [12:10:39] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:09] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [12:11:09] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 12:11:05 UTC 2013 [12:11:30] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 12:11:19 UTC 2013 [12:11:39] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:09] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [12:19:10] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 12:19:07 UTC 2013 [12:19:39] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [12:22:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [12:27:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [12:29:59] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 12:29:50 UTC 2013 [12:30:09] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [12:42:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [12:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [12:57:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [13:06:07] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [13:21:31] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:22:21] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [14:01:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [14:06:34] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [14:07:04] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [14:36:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:37:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.142 second response time [14:55:18] could someone please kick stafford in da but? [14:56:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [15:06:10] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [15:06:50] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [15:22:00] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [15:22:00] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [15:22:00] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [15:22:00] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [15:22:00] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [15:22:01] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [15:22:01] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [15:22:02] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:57] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:08:17] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:38] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 16:08:36 UTC 2013 [16:08:57] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:09:27] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 16:09:17 UTC 2013 [16:09:57] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 16:09:48 UTC 2013 [16:10:17] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:27] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 16:10:25 UTC 2013 [16:10:57] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:10:57] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 16:10:53 UTC 2013 [16:11:17] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:37] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 16:11:27 UTC 2013 [16:11:57] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:11:57] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 16:11:54 UTC 2013 [16:12:17] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:27] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 16:12:22 UTC 2013 [16:12:57] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:13:17] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:28] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 16:13:26 UTC 2013 [16:13:57] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 16:13:47 UTC 2013 [16:13:57] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:14:07] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 16:14:00 UTC 2013 [16:14:17] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:57] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:16:37] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 16:16:31 UTC 2013 [16:17:17] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:47] PROBLEM - DPKG on mc15 is CRITICAL: Timeout while attempting connection [16:28:47] RECOVERY - DPKG on mc15 is OK: All packages OK [16:29:57] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 16:29:48 UTC 2013 [16:30:57] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:47:04] andre__: I read your comment on https://bugzilla.wikimedia.org/show_bug.cgi?id=30413 and I like the irony ツ [16:48:08] AzaToth, I don't expect much progress either but I'm happy to be proven wrong. [16:48:59] andre__: I assume all eventual progress will occur inside RT, so I will probably never know :( [16:49:09] catch 22 [16:49:17] so I think I could CC you on that ticket. [16:49:48] sure [16:50:35] PROBLEM - Disk space on wtp1021 is CRITICAL: DISK CRITICAL - free space: / 348 MB (3% inode=78%): [16:51:35] it can be rather distressful when a bug refers to a RT ticket and you don't know if that ticket is buried or taken care of [16:51:54] AzaToth, CC'ed you [16:53:26] havn't got any [16:53:58] AzaToth: s/and you don't know if that ticket is buried or taken care of// [16:54:22] MatmaRex: true [16:54:58] MatmaRex: I would want to be able to assume that all tickets are taken care of asap without any delay and to the fullest [16:55:32] and that no tickets are ignored or invalidated [16:55:33] but we all know that's unrealistic in any software project :P [16:55:51] andre__: heh [16:56:34] andre__: I assume you just set me up on cc, not that it would cc current data to me right? [16:57:29] yes [16:57:38] I know it's not a solution to the fundamental problem, but I think I do have rt access and can help check the status of things if people want to [16:57:43] guessing other people would too [16:57:44] andre__: is there much secret and sensitive information flowing through the RT? [16:58:13] in general? Some I'd say. It's not that I follow tickets closely. [16:58:17] Procurement etc. [16:58:23] YuviPanda: that's nice (at least when we know the RT number) [16:59:13] AzaToth: true. [16:59:22] andre__: wont things like procurement be publicized anyway? [16:59:26] RT's interface is... something though :) [16:59:34] YuviPanda: I know [16:59:43] have used RT a lot [16:59:59] poor you [17:00:11] ツ [17:00:34] AzaToth, results maybe. but not the process. [17:00:48] hmm [17:01:29] andre__: if we are barred to know the process, we must assume it's illegal :-P [17:02:56] YuviPanda: RT #5394 any progress? [17:03:09] looking [17:03:26] für bug 36443 [17:03:40] AzaToth: initial creation by hashar, and then ntohing [17:03:41] *nothing [17:03:58] oh [17:04:24] initial creation has a fair amount of detail about jenkins-debian-glue [17:04:33] good to know nothing has been made... [17:04:52] YuviPanda: too much information for the poor sys-admins? [17:05:02] it is possible that you will get access to RT, but not now. RT is only the first and least private of internal ticketing systems. I have caught glimpses of the second and know of the third only by rumor. [17:05:27] ori-l: there are three levels? [17:05:38] AzaToth: the title is actionable, and I suppose it is standard enough (import into wmf repos) [17:05:39] (4 if including the bugzilla) [17:05:54] not including random on wiki bashing / complaints? [17:06:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [17:06:08] AzaToth: i'm just making a prague joke for andre__ [17:06:09] and IRC poking to death [17:06:19] ori-l: ah [17:06:19] AzaToth: it also complains about 'lintian' errors [17:06:25] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:41] AzaToth: http://ace.home.xs4all.nl/Literaria/Txt-Kafka.html [17:06:42] clearly fight club deaths are too mainstream now [17:06:47] YuviPanda: it will run lintian on the packages, which checks for common errors [17:06:54] ehehe [17:07:05] then it will use piuparts to test install/uninstall the package to make sure it's fine [17:07:30] i think it's errors on jenkins-debian-glue itself :D [17:07:32] ori-l: sorry, I can't speak German [17:07:42] YuviPanda: it? [17:07:48] YuviPanda: you said no one repsoned [17:07:54] unsure how much of RT I can paste publicly, but I can PM you a private paste [17:08:00] AzaToth: this is in the original creation itself [17:08:03] AzaToth: oh, I don't know why I thought that. Maybe the 'für' earlier. [17:08:13] I don't know what hashar has written, so I can't know for sure what he wrote [17:08:22] ori-l: heh [17:08:27] AzaToth: pm'd [17:09:01] YuviPanda: ah, yea, some warnings in the packages [17:09:31] changelog-should-mention-nmu - common when building packages and you are not maintainer/uploader [17:10:12] newer-standards-version 3.9.4 - only policy things, binary-without-manpage - you shall always have a manpage!; [17:10:23] nothing major [17:10:40] I'm not sure how you can respond to it, though. I'm not sure if emailing in lets you add comments [17:10:53] I can offer to act like a human proxy though [17:10:54] probably not [17:11:11] if you want me to [17:11:41] sure [17:12:00] none of the warnings are important [17:12:05] write up a reply, email yuvipanda@gmail.com, and I'll post it? :) [17:12:08] and reply when they reply? [17:12:12] easier than having this on IRC [17:12:22] well, dunno if they will reply [17:12:34] usually poking whoever is on ops duty helps [17:12:39] poking hashar will too [17:12:47] Ryan aint online [17:12:59] has been up since 2013-06-28 [17:13:12] weekend, etc [17:13:20] excuses... :-P [17:13:30] I know [17:13:31] :) [17:15:03] ori-l: well, now we at least know where the _real_ cabal channel is ツ [17:15:20] in the SF office? :D [17:15:37] AzaToth: #wikimedia-e3, naturally [17:15:45] heh [17:16:24] could someone setup #wikipedia-en-admins-cabal and make it locked? [17:16:38] just to drive the conspirationista mad [17:17:22] make it password protected and then forget the password? [17:17:47] sort of [17:18:04] *evil grin* [17:18:22] :) [17:18:32] so it'll be a cabal where literally nobody can get in [17:18:38] yea ツ [17:23:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:24:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [18:06:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [18:06:50] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [19:06:25] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [19:06:55] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [19:30:55] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:31:45] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [19:31:45] Elsie: can you restart deployment-parsoid2? [19:54:07] AzaToth: Not cleanly. [19:54:51] nevermind [19:55:26] I thought parsoid.wmflabs.org was resolving to that one [20:06:52] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:07:02] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:08:02] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 20:08:00 UTC 2013 [20:08:42] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 20:08:34 UTC 2013 [20:08:52] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:09:02] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:22] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 20:09:15 UTC 2013 [20:09:52] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 20:09:45 UTC 2013 [20:10:02] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:23] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 20:10:20 UTC 2013 [20:10:52] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:10:52] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 20:10:48 UTC 2013 [20:11:02] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:23] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 20:11:20 UTC 2013 [20:11:52] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:11:52] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sun Jul 7 20:11:43 UTC 2013 [20:12:02] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:02] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [20:12:22] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sun Jul 7 20:12:13 UTC 2013 [20:12:52] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:13:02]