[00:08:01] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:08:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:08:20] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 00:08:13 UTC 2013 [00:08:50] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 00:08:42 UTC 2013 [00:09:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:09:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:30] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 00:09:21 UTC 2013 [00:09:50] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 00:09:49 UTC 2013 [00:10:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:10:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:30] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 00:10:24 UTC 2013 [00:10:50] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 00:10:42 UTC 2013 [00:11:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:11:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:11:10] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 00:11:07 UTC 2013 [00:11:40] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 00:11:30 UTC 2013 [00:12:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:12:00] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 00:11:58 UTC 2013 [00:12:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:13:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:16:30] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 00:16:28 UTC 2013 [00:17:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [00:30:00] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 00:29:50 UTC 2013 [00:30:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [00:32:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [01:02:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [02:05:49] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [02:06:19] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [02:14:25] !log LocalisationUpdate completed (1.22wmf8) at Sat Jul 6 02:14:24 UTC 2013 [02:14:35] Logged the message, Master [02:27:28] !log LocalisationUpdate completed (1.22wmf9) at Sat Jul 6 02:27:28 UTC 2013 [02:27:37] Logged the message, Master [02:38:04] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 6 02:38:04 UTC 2013 [02:38:13] Logged the message, Master [02:52:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [03:06:50] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [03:06:50] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [03:22:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [03:31:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [03:52:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:53:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.718 second response time [04:01:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [04:06:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:06:50] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:00] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 04:07:53 UTC 2013 [04:08:20] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 04:08:15 UTC 2013 [04:08:50] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:08:50] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [04:08:50] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 04:08:43 UTC 2013 [04:09:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:09:10] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 04:09:04 UTC 2013 [04:09:50] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:10:10] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 04:10:04 UTC 2013 [04:10:20] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 04:10:19 UTC 2013 [04:10:50] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:16:40] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 04:16:33 UTC 2013 [04:16:50] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [04:22:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.750 second response time [04:29:50] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 04:29:48 UTC 2013 [04:30:00] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [04:52:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.207 second response time [05:06:22] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [05:06:52] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [05:07:12] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [05:11:29] New patchset: ArielGlenn; "mwbzutils: version to 0.0.4, clean up in prep for debian packaging" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/72005 [05:14:12] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [05:22:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:23:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [05:52:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [06:02:34] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:04:24] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [06:06:15] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:07:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [06:10:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:11:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [06:22:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [06:29:35] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.122 second response time [06:30:05] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 06:30:00 UTC 2013 [06:30:15] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:26] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 06:30:23 UTC 2013 [06:30:55] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 06:30:48 UTC 2013 [06:31:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [06:31:15] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:15] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 06:31:07 UTC 2013 [06:31:36] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 06:31:31 UTC 2013 [06:32:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [06:32:15] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:26] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 06:32:20 UTC 2013 [06:32:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [06:33:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [07:00:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:01:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.158 second response time [07:09:42] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [07:09:42] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [07:11:42] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:32] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [07:17:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:18:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [08:09:35] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:35] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [08:51:21] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [08:52:21] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.78 ms [09:08:29] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [09:08:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [09:18:49] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:19:49] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [09:20:59] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [09:20:59] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [09:20:59] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [09:20:59] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [09:20:59] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [09:21:00] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [09:21:00] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [09:21:01] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [10:09:06] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:09:45] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [10:10:15] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [10:11:55] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [10:27:05] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:28:05] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [10:39:02] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:40:02] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [11:06:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [11:06:39] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [11:34:41] New patchset: Odder; "(bug 50802) Create new user groups on sh.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/72252 [11:52:41] PROBLEM - search indices - check lucene status page on search20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern found - 60051 bytes in 0.109 second response time [12:08:39] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [12:08:49] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [12:15:59] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:16:49] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [12:22:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [12:31:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [12:52:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [13:09:46] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [13:10:16] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [13:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [13:24:01] New patchset: Odder; "(bug 50156) Set import sources 'w' and 'en' for ml.wikibooks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/72254 [13:33:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [13:52:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [14:07:31] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [14:08:01] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [14:09:11] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [14:27:31] PROBLEM - Disk space on analytics1006 is CRITICAL: DISK CRITICAL - free space: / 705 MB (3% inode=84%): [14:32:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [15:05:43] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [15:06:23] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [15:07:23] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [15:13:25] LeslieCarr: filed https://bugzilla.wikimedia.org/show_bug.cgi?id=50866 fyi [15:14:23] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:06:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:06:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:08:29] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:08:20 UTC 2013 [16:08:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:09:49] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:09:48 UTC 2013 [16:10:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:59] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:10:51 UTC 2013 [16:11:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:12:19] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:12:13 UTC 2013 [16:13:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:19] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:13:18 UTC 2013 [16:13:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:14:40] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:14:37 UTC 2013 [16:15:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:39] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:15:37 UTC 2013 [16:16:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:16:59] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:16:49 UTC 2013 [16:17:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:17:49] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:17:46 UTC 2013 [16:18:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:18:59] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:18:58 UTC 2013 [16:19:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:19:59] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:19:54 UTC 2013 [16:20:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:21:09] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:21:04 UTC 2013 [16:21:59] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:21:55 UTC 2013 [16:22:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:22:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:22:59] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:22:54 UTC 2013 [16:23:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:23:49] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:23:41 UTC 2013 [16:24:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:24:40] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:24:35 UTC 2013 [16:25:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:25:19] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:25:14 UTC 2013 [16:25:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:26:09] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:26:05 UTC 2013 [16:26:49] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:26:42 UTC 2013 [16:27:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:27:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:27:30] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:27:27 UTC 2013 [16:28:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:28:09] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:28:04 UTC 2013 [16:28:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:28:49] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:28:45 UTC 2013 [16:28:59] LeslieCarr: puppet [16:29:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:29:19] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:29:16 UTC 2013 [16:29:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:29:59] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:29:55 UTC 2013 [16:30:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:30:29] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:30:23 UTC 2013 [16:31:09] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:30:59 UTC 2013 [16:31:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:31:29] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:31:25 UTC 2013 [16:32:09] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:31:58 UTC 2013 [16:32:09] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:32:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:32:59] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:32:49 UTC 2013 [16:32:59] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:33:13] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:33:08 UTC 2013 [16:33:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:33:39] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:33:32 UTC 2013 [16:33:59] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:33:49 UTC 2013 [16:33:59] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:34:09] MatmaRex: puppet is doing the do again [16:34:19] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 16:34:12 UTC 2013 [16:34:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:34:29] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 16:34:25 UTC 2013 [16:34:59] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [16:35:29] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [16:35:38] AzaToth: well, then it's good that i don't care [16:35:59] you are such a careless person :-P [16:35:59] i'm only sitting here because #-tech is actually useless when something is actually broken, since everybody sits in here [16:36:31] the wikimedia experience has left me all out of damns to give [17:06:10] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [17:06:10] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [18:06:01] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [18:06:01] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [18:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [19:04:35] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [19:05:15] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [19:21:25] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [19:21:25] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [19:21:25] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [19:21:25] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [19:21:25] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [19:21:26] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [19:21:26] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [19:21:27] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [20:03:30] New patchset: ArielGlenn; "mwbzutils: version to 0.0.4, clean up in prep for debian packaging" [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/72005 [20:06:05] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:06:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:08:05] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 20:08:02 UTC 2013 [20:08:35] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 20:08:29 UTC 2013 [20:09:05] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:09:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:09:15] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 20:09:09 UTC 2013 [20:09:45] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 20:09:38 UTC 2013 [20:10:05] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:10:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:10:25] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 20:10:18 UTC 2013 [20:10:45] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 20:10:40 UTC 2013 [20:11:06] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:11:06] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:11:15] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 20:11:13 UTC 2013 [20:11:45] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 20:11:37 UTC 2013 [20:12:05] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:12:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:12:15] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 20:12:07 UTC 2013 [20:12:35] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 20:12:25 UTC 2013 [20:13:05] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:13:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:13:15] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 20:13:07 UTC 2013 [20:13:18] New review: AzaToth; "as I said, doing it manually invites problems :)" [operations/dumps] (ariel) C: -1; - https://gerrit.wikimedia.org/r/72005 [20:13:35] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 20:13:28 UTC 2013 [20:13:47] apergos: there [20:14:05] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:14:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:14:14] also I must point to the fact you should make it an own repo [20:16:57] 14 comments? that's worse than the last time [20:17:10] if I make another patchset I'll have 30 comments [20:17:21] aye [20:17:35] and yet I need to have this package before en wp dumps run this month [20:18:03] and when is that? [20:18:33] well a few days ago :-D [20:19:05] RECOVERY - Puppet freshness on cp3001 is OK: puppet ran at Sat Jul 6 20:18:56 UTC 2013 [20:19:13] I assume it's too late now then ツ [20:19:17] I don't want to remove the generated man pages unless we do reallyclean [20:19:27] reason for this is that the man pages should be pregenerated ahead of time [20:19:29] ok [20:19:35] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [20:19:40] as I noted in the commit, this permits cross-platform package builds to succeed [20:19:55] then ignore the manpage issue; I assume it was meant to generate the man pages when building [20:20:05] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [20:20:18] cross-platform package builds? [20:20:23] uh huh [20:20:24] dunno what that is [20:20:25] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [20:20:40] because help2man relies on running the executable [20:20:47] ok [20:20:47] to generate the man pages [20:21:09] there's a note about that either in help2man docs or debian package docs, I forget which [20:21:15] anyway, do not place gz in the repo unless you really must [20:21:52] I guess I can gzip them at the time [20:22:09] debian will compress them if needed when building debian packages (the right way) [20:22:35] it would but of course I get to override a bunch of crap because we do in fact put our stuff in /usr/local [20:24:02] CPPFLAGS must at least have VERSION, that's mandatory, and given that this code requires 64-bit it needs the other flag too [20:24:05] those are requirements [20:24:14] now I can append instead of just set if you like [20:24:25] yea [20:24:35] ok [20:24:39] just saying you shouldn't replace them unconditionally [20:24:44] uh huh [20:25:01] when building debian packages, a lot of other flags might arrive ツ [20:25:38] for cppflags I didn't really see that (I spent yesterday building.. and rebuilding.. and tweaking.. and rebuilding) [20:26:19] when building debian packages, and hardening is enabled, following flags will be added: -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security --std=c++0x -D_FORTIFY_SOURCE=2 [20:26:20] anyways next patch set will be tomorrow I guess [20:26:25] pretty crispy tonight [20:26:35] in CPPFLAGS? or CFLAGS? [20:26:37] except the std=c++0x [20:26:56] $ dpkg-buildflags [20:26:56] CFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security [20:26:56] CPPFLAGS=-D_FORTIFY_SOURCE=2 [20:26:56] CXXFLAGS=-g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security [20:26:56] FFLAGS=-g -O2 [20:26:57] LDFLAGS=-Wl,-z,relro [20:27:05] * apergos looks up fortify_source [20:27:27] actually [20:27:29] * apergos lies [20:27:33] fyi, your code doesn't compile with these flags atm [20:27:36] * apergos opens the tab and leaves it for tomorrow [20:28:26] hopefully it's only format-security related that makes it not compile [20:28:41] it's gonig to get ignored right now [20:29:01] apergos: not really if you are planning to have it as a debian package [20:29:09] it's not meant to go into debian [20:29:15] it's meant for us to deploy [20:29:18] yea [20:29:37] via a deb instead of how it has been which is manually [20:29:42] the flags are default when building nowadays [20:29:55] RECOVERY - Puppet freshness on dysprosium is OK: puppet ran at Sat Jul 6 20:29:49 UTC 2013 [20:30:05] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [20:30:06] mm we'll see [20:32:50] thanks for loking at it again, as I say next version will be tomorrow sometime [20:32:50] *looking [21:07:05] New patchset: Odder; "(bug 50357) Set $wgSitename for te.wikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/72318 [21:07:26] RECOVERY - search indices - check lucene status page on search19 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.110 second response time [21:07:56] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [21:21:56] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [21:22:02] wow thanks Dereckson [21:23:06] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.54 ms [22:02:59] New patchset: Odder; "(bug 50357) Set $wgSitename for te.wikisource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/72318 [22:07:59] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:08:29] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [22:08:59] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [22:11:39] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [22:22:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [22:29:39] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:31:29] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [22:39:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [22:52:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:53:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [23:06:23] PROBLEM - Puppet freshness on dysprosium is CRITICAL: No successful Puppet run in the last 10 hours [23:06:53] PROBLEM - Puppet freshness on cp3001 is CRITICAL: No successful Puppet run in the last 10 hours [23:12:33] RECOVERY - search indices - check lucene status page on search20 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.109 second response time [23:22:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.423 second response time [23:39:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:40:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.062 second response time