[00:05:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:07:25] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24741 MB (3% inode=99%): [00:07:55] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [00:08:05] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 00:07:55 UTC 2013 [00:08:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:09:05] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 00:09:00 UTC 2013 [00:09:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:10:05] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 00:10:00 UTC 2013 [00:10:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:10:55] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 00:10:52 UTC 2013 [00:11:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:11:45] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 00:11:39 UTC 2013 [00:12:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:12:55] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 00:12:52 UTC 2013 [00:13:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [00:14:35] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 00:14:33 UTC 2013 [00:14:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [01:04:40] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [01:06:20] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25286 MB (3% inode=99%): [01:06:50] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [02:04:20] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [02:06:30] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [02:07:00] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25910 MB (3% inode=99%): [02:10:19] !log LocalisationUpdate completed (1.21wmf12) at Sun Mar 31 02:10:18 UTC 2013 [02:10:26] Logged the message, Master [02:10:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:13:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [02:57:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:58:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [03:04:25] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [03:06:05] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25468 MB (3% inode=99%): [03:06:35] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [03:32:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [03:40:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:40:45] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [03:40:45] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [03:40:45] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [03:41:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.288 second response time [03:56:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:59:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [04:06:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:07:59] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 04:07:57 UTC 2013 [04:08:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:08:39] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [04:09:09] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24989 MB (3% inode=99%): [04:09:09] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 04:09:03 UTC 2013 [04:09:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:10:09] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 04:10:03 UTC 2013 [04:10:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:10:59] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 04:10:56 UTC 2013 [04:11:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:11:49] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 04:11:44 UTC 2013 [04:12:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:12:29] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 04:12:25 UTC 2013 [04:13:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:14:39] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 04:14:30 UTC 2013 [04:15:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [04:25:19] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [04:32:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [05:02:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [05:06:24] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [05:08:04] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24552 MB (2% inode=99%): [05:08:34] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [05:13:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:14:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.172 second response time [05:31:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:32:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [05:40:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [05:52:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:53:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [06:05:17] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:07:27] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [06:07:57] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 25020 MB (3% inode=99%): [06:26:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.123 second response time [06:30:07] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 06:29:57 UTC 2013 [06:30:17] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:30:47] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 06:30:39 UTC 2013 [06:31:17] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [06:31:18] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 06:31:16 UTC 2013 [06:32:17] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [07:03:24] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [07:05:34] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [07:06:04] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24637 MB (3% inode=99%): [08:03:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:05:25] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [08:05:55] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24283 MB (2% inode=99%): [08:07:55] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 08:07:48 UTC 2013 [08:08:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:08:35] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 08:08:27 UTC 2013 [08:09:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:10:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:11:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.148 second response time [08:14:35] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 08:14:27 UTC 2013 [08:15:15] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [08:26:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:27:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [08:28:45] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:29:35] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [09:02:06] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [09:05:52] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [09:07:32] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23751 MB (2% inode=99%): [09:08:02] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [09:14:42] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 09:14:34 UTC 2013 [09:14:52] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [09:26:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [09:30:38] New patchset: MaxSem; "Add device detection to mobile ResourceLoader requests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56774 [09:31:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [09:34:23] New patchset: MaxSem; "Add device detection to mobile ResourceLoader requests" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56774 [10:04:26] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [10:06:06] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24343 MB (2% inode=99%): [10:06:36] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [11:04:08] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [11:06:18] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [11:06:48] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24818 MB (3% inode=99%): [12:05:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:07:30] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24352 MB (2% inode=99%): [12:08:00] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 12:07:50 UTC 2013 [12:08:00] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [12:08:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:09:20] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 12:09:16 UTC 2013 [12:09:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:10:00] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 12:09:50 UTC 2013 [12:10:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [12:14:40] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 12:14:31 UTC 2013 [12:14:50] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [13:04:20] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [13:06:30] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [13:07:00] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23903 MB (2% inode=99%): [13:40:50] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [13:40:50] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [13:40:50] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [14:07:58] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [14:10:08] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [14:10:38] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23389 MB (2% inode=99%): [14:25:38] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [15:05:23] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [15:06:33] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [15:07:03] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24675 MB (3% inode=99%): [16:05:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:07:26] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [16:07:56] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24153 MB (2% inode=99%): [16:08:06] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 16:07:59 UTC 2013 [16:08:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:09:06] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 16:09:05 UTC 2013 [16:09:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:10:06] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 16:10:04 UTC 2013 [16:10:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:11:06] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 16:10:56 UTC 2013 [16:11:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:11:46] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 16:11:43 UTC 2013 [16:12:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:12:27] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 16:12:23 UTC 2013 [16:13:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [16:14:36] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 16:14:32 UTC 2013 [16:15:16] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [17:07:29] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [17:08:39] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [17:09:09] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23391 MB (2% inode=99%): [17:23:19] PROBLEM - RAID on ms-be7 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:03:42] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [18:05:52] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [18:06:22] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23876 MB (2% inode=99%): [19:02:20] PROBLEM - Puppet freshness on virt3 is CRITICAL: Puppet has not run in the last 10 hours [19:05:01] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [19:07:11] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [19:07:41] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 24274 MB (2% inode=99%): [19:23:38] New patchset: ArielGlenn; "revoke tfinc's keys" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56876 [19:25:28] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/56876 [19:27:31] looks like puppet runs regularly right now, so that's a plus [19:35:37] i just removed his root keys [19:45:18] ok [19:47:32] ct called my cell, that's the only reason I heard about it (I was off line doing other things) [19:48:24] would you have done the root keys as well? [19:52:11] tfinc doesn't work for wmf anymore? [19:57:07] mark, I don't even know what you removed, so no, I wouldn't have done it [20:04:07] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:06:17] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [20:06:47] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23716 MB (2% inode=99%): [20:07:57] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 20:07:53 UTC 2013 [20:08:07] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:09:07] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 20:09:02 UTC 2013 [20:10:07] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:10:07] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 20:10:03 UTC 2013 [20:11:07] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:11:47] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 20:11:44 UTC 2013 [20:12:07] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:12:27] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 20:12:26 UTC 2013 [20:13:07] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:13:07] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 20:13:01 UTC 2013 [20:14:07] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [20:14:37] RECOVERY - Puppet freshness on db11 is OK: puppet ran at Sun Mar 31 20:14:32 UTC 2013 [20:15:07] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [21:03:32] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [21:05:12] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23110 MB (2% inode=99%): [21:05:42] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [21:22:32] PROBLEM - LVS HTTPS IPv4 on wikivoyage-lb.pmtpa.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:42] PROBLEM - LVS HTTP IPv4 on appservers.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:12] PROBLEM - Apache HTTP on mw1058 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:23:22] RECOVERY - LVS HTTPS IPv4 on wikivoyage-lb.pmtpa.wikimedia.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 757 bytes in 0.196 second response time [21:24:02] RECOVERY - Apache HTTP on mw1058 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 6.717 second response time [21:24:12] PROBLEM - Apache HTTP on mw1023 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:12] PROBLEM - Apache HTTP on mw1166 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:12] PROBLEM - Apache HTTP on mw1098 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:12] PROBLEM - Apache HTTP on mw1052 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:32] RECOVERY - LVS HTTP IPv4 on appservers.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 61914 bytes in 1.434 second response time [21:25:02] RECOVERY - Apache HTTP on mw1166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [21:25:02] RECOVERY - Apache HTTP on mw1098 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.191 second response time [21:25:03] RECOVERY - Apache HTTP on mw1023 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.684 second response time [21:25:03] RECOVERY - Apache HTTP on mw1052 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.264 second response time [21:33:37] ugh, bouncy lvs [22:04:33] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [22:06:43] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [22:07:13] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23529 MB (2% inode=99%): [23:04:45] PROBLEM - Puppet freshness on db11 is CRITICAL: Puppet has not run in the last 10 hours [23:06:25] PROBLEM - Disk space on db11 is CRITICAL: DISK CRITICAL - free space: /a 23981 MB (2% inode=99%): [23:06:55] PROBLEM - RAID on db11 is CRITICAL: CRITICAL: Defunct disk drive count: 1 [23:41:48] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [23:41:48] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [23:41:48] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours