[00:01:25] !log rebuilding mailman archives …. very slowly [00:01:56] maybe we can move mailing lists to google groups :PPPP :) [00:02:08] LeslieCarr: better than mailmans other speed [00:02:10] 'broke' [00:02:16] anything else we can outsource to the big G? [00:03:02] haha [00:03:06] en.wikipedia.google.doc/article [00:03:12] hehe [00:03:26] knolpedia [00:03:31] haha [00:06:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:07:25] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [00:07:55] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 26261 MB (3% inode=99%): [00:07:55] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 00:07:52 UTC 2013 [00:08:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:08:55] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 00:08:52 UTC 2013 [00:09:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:09:45] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Sat Mar 30 00:09:44 UTC 2013 [00:09:53] LeslieCarr: OMG I HAVENT GOTTEN A NEW TROLL-L MESSAGE IN TWO MINUTES [00:09:55] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 00:09:47 UTC 2013 [00:10:06] WHY IS MAILMAN NOT RUNNING,WWWWWWHHHHHYYYYY [00:10:13] i wanted to give you the 2pm experience at 5pm. [00:10:13] hahaha [00:10:14] :) [00:10:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:10:45] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 00:10:35 UTC 2013 [00:11:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:11:25] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 00:11:17 UTC 2013 [00:12:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:12:35] now it's just db11, icinga is mocking us [00:17:05] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 00:17:04 UTC 2013 [00:17:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:20:15] PROBLEM - Puppet freshness on mw1160 is CRITICAL: Puppet has not run in the last 10 hours [00:25:05] RECOVERY - Puppet freshness on mw1160 is OK: puppet ran at Sat Mar 30 00:24:56 UTC 2013 [00:40:41] jeremyb_: What's up [00:42:17] Krinkle: i think i was just wondering about https://rt.wikimedia.org/Ticket/Display.html?id=4851 [00:42:50] but i guess on second thought that's not related to jenkins [00:43:27] anyway, will find out when hashar's back or someone decides to check the puppet log :) [00:46:46] k [00:47:23] * jeremyb_ equated hashar with jenkins and then extended jenkins to you :) [00:47:28] New review: Krinkle; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56640 [00:47:35] New patchset: Krinkle; "gerrit: Change repeat from 8-40 to 7-40 for hash in commentlink" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56640 [00:49:45] ahhh, you rebased [00:51:11] "Patch Set 1 was rebased" [00:51:42] * jeremyb_ read the diff first [01:01:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [01:05:21] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [01:07:31] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [01:08:01] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25802 MB (3% inode=99%): [01:09:28] New patchset: Pyoungmeister; "really basic module for multiple instance of mysql on a node" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56710 [01:26:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.159 second response time [02:05:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [02:07:09] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25394 MB (3% inode=99%): [02:07:39] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [02:13:49] !log LocalisationUpdate completed (1.21wmf12) at Sat Mar 30 02:13:49 UTC 2013 [02:13:57] Logged the message, Master [02:22:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.141 second response time [02:56:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time [03:00:53] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [03:04:51] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [03:06:31] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24930 MB (3% inode=99%): [03:07:01] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [03:22:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.138 second response time [03:31:11] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [03:32:11] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [03:32:32] * jeremyb_ wonders if frack has its own nagios/icinga and if frack alerts come to this channel [04:00:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [04:06:19] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:07:29] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [04:07:59] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25584 MB (3% inode=99%): [04:07:59] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 04:07:53 UTC 2013 [04:08:19] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:09:09] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 04:08:59 UTC 2013 [04:09:19] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:10:09] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 04:10:01 UTC 2013 [04:10:19] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:10:59] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 04:10:56 UTC 2013 [04:11:19] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:11:49] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 04:11:44 UTC 2013 [04:12:19] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:12:29] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 04:12:25 UTC 2013 [04:13:19] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:14:39] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 04:14:31 UTC 2013 [04:15:19] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:58:24] fuck did lvs1001 die again ? [05:04:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [05:06:39] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [05:07:09] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25180 MB (3% inode=99%): [06:00:16] RECOVERY - Disk space on snapshot2 is OK: DISK OK [06:01:46] RECOVERY - Disk space on snapshot1 is OK: DISK OK [06:02:46] RECOVERY - Disk space on snapshot3 is OK: DISK OK [06:04:36] RECOVERY - Disk space on snapshot4 is OK: DISK OK [06:05:57] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:08:07] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [06:08:37] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24780 MB (3% inode=99%): [06:30:17] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 06:30:07 UTC 2013 [06:30:57] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:31:17] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 06:31:14 UTC 2013 [06:31:57] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [07:05:22] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [07:07:32] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [07:08:02] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24345 MB (2% inode=99%): [07:39:59] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [07:39:59] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [07:39:59] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [08:05:35] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:07:15] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23937 MB (2% inode=99%): [08:07:45] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [08:07:55] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 08:07:48 UTC 2013 [08:08:35] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:08:35] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 08:08:28 UTC 2013 [08:09:35] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:15:05] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 08:14:57 UTC 2013 [08:15:35] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:24:05] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [08:44:10] New review: QChris; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56640 [08:45:27] New review: QChris; "Unrelated/Unmotivated change of regular expression for Change-Ids as described in inline comments in..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/56640 [09:06:47] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [09:08:27] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23555 MB (2% inode=99%): [09:08:57] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [09:18:27] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 09:18:23 UTC 2013 [09:18:47] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [11:05:57] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [11:07:37] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25758 MB (3% inode=99%): [11:08:07] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [12:06:54] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:07:54] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 12:07:45 UTC 2013 [12:08:34] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25305 MB (3% inode=99%): [12:08:54] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:09:04] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [12:14:34] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 12:14:32 UTC 2013 [12:14:54] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [13:01:11] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [13:04:34] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [13:05:44] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [13:06:14] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24838 MB (3% inode=99%): [14:04:35] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [14:06:45] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [14:07:15] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24375 MB (2% inode=99%): [15:05:59] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [15:07:39] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24947 MB (3% inode=99%): [15:08:09] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [16:05:18] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:06:58] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 26446 MB (3% inode=99%): [16:07:28] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [16:07:58] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 16:07:54 UTC 2013 [16:08:18] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:09:08] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 16:08:59 UTC 2013 [16:09:18] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:10:08] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 16:09:59 UTC 2013 [16:10:18] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:10:58] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 16:10:52 UTC 2013 [16:11:18] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:11:48] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 16:11:38 UTC 2013 [16:12:18] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:12:28] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 16:12:18 UTC 2013 [16:13:18] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:14:38] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 16:14:33 UTC 2013 [16:15:18] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:17:14] paulproteus: https://github.com/puppetlabs/puppetlabs-mongodb/blob/master/manifests/sources/apt.pp#L14 :-( [16:22:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [16:26:10] jeremyb_: Should we clone the key for the lulz? [16:26:15] heh [16:26:35] Could clone it but not send it to keyservers. [16:26:51] https://github.com/puppetlabs/puppetlabs-mongodb/issues/28 [16:30:08] b [16:30:14] (thumbs up) [16:31:19] paulproteus: in other news, can you help me find an alioth admin? :) see #alioth and #debian-devel [16:32:07] Not any faster than your pings, I'm afraid [16:32:34] rats :( [16:34:39] New review: Jeremyb; "filed upstream: https://github.com/puppetlabs/puppetlabs-mongodb/issues/28" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27195 [16:37:46] csteipp: see above fyi :-) [16:38:21] * jeremyb_ runs away [17:08:12] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [17:10:22] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [17:10:52] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25985 MB (3% inode=99%): [17:40:03] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [17:40:03] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [17:40:03] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [18:04:43] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [18:06:23] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25564 MB (3% inode=99%): [18:06:53] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [18:24:53] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [18:35:25] New review: Krinkle; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56640 [19:01:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [19:05:12] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [19:07:22] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [19:07:52] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25095 MB (3% inode=99%): [19:13:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [20:00:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [20:04:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:06:30] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25609 MB (3% inode=99%): [20:07:00] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [20:08:00] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 20:07:54 UTC 2013 [20:08:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:09:00] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 20:08:59 UTC 2013 [20:09:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:10:00] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 20:09:58 UTC 2013 [20:10:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:11:00] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 20:10:51 UTC 2013 [20:11:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:12:20] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 20:12:18 UTC 2013 [20:12:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:13:00] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 20:12:51 UTC 2013 [20:13:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:14:40] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sat Mar 30 20:14:32 UTC 2013 [20:14:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:25:13] New review: QChris; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56640 [21:05:21] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [21:07:31] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [21:08:01] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 26125 MB (3% inode=99%): [21:17:35] New review: PleaseStand; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/56408 [21:42:00] New review: PleaseStand; "(1 comment)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/56408 [21:56:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:57:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.139 second response time [22:06:41] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [22:08:21] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25540 MB (3% inode=99%): [22:08:51] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [23:02:00] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [23:08:24] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [23:09:34] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [23:10:04] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25132 MB (3% inode=99%): [23:26:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:27:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.162 second response time