[00:04:29] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.997 second response time on port 11211 [00:07:48] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [00:07:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.998 second response time on port 11211 [00:08:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.003 second response time on port 6379 [00:11:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [00:19:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [00:21:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.000 second response time on port 11211 [00:24:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [00:29:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.008 second response time on port 6379 [00:30:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.998 second response time on port 11211 [00:32:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [00:38:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.001 second response time on port 6379 [00:41:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [00:42:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.001 second response time on port 6379 [00:42:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [00:45:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [00:46:53] New review: TTO; "Thanks" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/73568 [00:46:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 3.001 second response time on port 11211 [00:49:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [00:50:48] PROBLEM - Puppet freshness on grosley is CRITICAL: No successful Puppet run in the last 10 hours [00:54:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.997 second response time on port 6379 [00:57:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [00:58:48] PROBLEM - Puppet freshness on mw56 is CRITICAL: No successful Puppet run in the last 10 hours [01:03:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.998 second response time on port 11211 [01:06:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.001 second response time on port 6379 [01:11:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [01:13:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [01:15:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.000 second response time on port 11211 [01:18:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [01:19:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.001 second response time on port 6379 [01:22:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [01:24:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.997 second response time on port 6379 [01:25:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.000 second response time on port 11211 [01:27:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [01:29:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [01:30:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.997 second response time on port 11211 [01:32:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.997 second response time on port 6379 [01:35:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [01:39:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [01:46:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 1.000 second response time on port 11211 [01:49:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [01:50:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.000 second response time on port 6379 [01:52:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 1.004 second response time on port 11211 [01:53:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [01:58:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [01:59:58] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [02:01:43] !log LocalisationUpdate completed (1.22wmf9) at Sun Jul 14 02:01:43 UTC 2013 [02:01:57] Logged the message, Master [02:02:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.000 second response time on port 6379 [02:02:31] !log LocalisationUpdate completed (1.22wmf10) at Sun Jul 14 02:02:31 UTC 2013 [02:02:43] Logged the message, Master [02:02:58] RECOVERY - Memcached on mc1005 is OK: TCP OK - 1.000 second response time on port 11211 [02:07:10] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 14 02:07:10 UTC 2013 [02:07:25] Logged the message, Master [02:08:02] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [02:09:52] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.000 second response time on port 11211 [02:11:22] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [02:14:02] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [02:15:52] RECOVERY - Memcached on mc1005 is OK: TCP OK - 3.004 second response time on port 11211 [02:19:02] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [02:20:52] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [02:23:22] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.001 second response time on port 6379 [02:24:52] RECOVERY - Memcached on mc1005 is OK: TCP OK - 3.004 second response time on port 11211 [02:26:22] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [02:27:22] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.004 second response time on port 6379 [02:30:22] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [02:40:02] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [02:40:22] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.002 second response time on port 6379 [02:43:11] New patchset: TTO; "(bug 48381) remove obsolete js/css entries from $wgWhitelistRead" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73603 [02:43:22] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [02:50:52] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.000 second response time on port 11211 [02:56:22] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.004 second response time on port 6379 [03:01:22] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [03:04:02] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [03:04:52] RECOVERY - Memcached on mc1005 is OK: TCP OK - 3.002 second response time on port 11211 [03:05:46] New patchset: TTO; "(bug 30743) add namespace aliases for fawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73604 [03:07:59] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [03:09:59] RECOVERY - Memcached on mc1005 is OK: TCP OK - 3.001 second response time on port 11211 [03:12:59] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [03:14:49] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.000 second response time on port 11211 [03:15:20] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.999 second response time on port 6379 [03:18:30] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [03:29:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.003 second response time on port 6379 [03:33:29] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [03:34:59] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [03:35:49] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.000 second response time on port 11211 [03:38:59] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [03:43:59] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.999 second response time on port 11211 [03:44:20] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.001 second response time on port 6379 [03:47:29] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [03:51:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.000 second response time on port 6379 [03:53:59] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [03:54:29] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [03:55:59] RECOVERY - Memcached on mc1005 is OK: TCP OK - 3.002 second response time on port 11211 [03:58:59] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [04:09:57] RECOVERY - Memcached on mc1005 is OK: TCP OK - 3.002 second response time on port 11211 [04:12:57] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [04:21:57] RECOVERY - Memcached on mc1005 is OK: TCP OK - 3.004 second response time on port 11211 [04:26:17] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.000 second response time on port 6379 [04:27:57] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [04:29:27] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [04:30:57] RECOVERY - Memcached on mc1005 is OK: TCP OK - 1.000 second response time on port 11211 [04:35:57] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [04:37:17] RECOVERY - Redis on mc1005 is OK: [05:41:37] could use a prod DB query and maybe passwd reset if someone's on. (it can wait for a weekday but I guess it doesn't hurt to ask) [05:42:35] * apergos peeks in [05:45:27] ohai [05:45:32] hello [05:45:34] perfect timing for my wifi to break :-/ [05:45:38] (i'm back) [05:45:39] :-D [05:47:49] apergos: you has /msg [05:48:29] apergos: he claims to have created that account with that address but somehow the confirmation link didn't work and also the password reset form says he has no address [05:48:50] we mailed back and forth so i know his mail isn't spoofed [05:49:11] this is 2013071210008512 btw [05:49:40] i verified that the password reset form does in fact say that [05:49:53] so i guess first question is does the DB have any sign of that email for that user? [05:49:57] right [05:50:26] ohhhh, how did i miss that before [05:50:37] it's unattached! [05:50:43] uh huh [05:50:55] is that supposed to be unpossible for such a new account? [05:50:59] this is on en wp I guess? [05:51:09] and I don't know what's supposed to be the state for new accounts [05:51:12] yes, that's what it says at my link [05:52:07] there is such a user on en wp [05:52:20] there is no email set [05:53:22] hrmmm. so i guess we either just believe him and set a new address for the account (but then maybe there's something private in a watchlist? or other prefs?). or we tell him to go through crats to usurp [05:53:28] I am reluctant (read: I won't do it I guess) to set the email [05:53:43] I think it's better to usurp (= move this account out of the way, etc) [05:54:06] right, ok [05:54:08] but, it would be good to find ut if new accounts that aren't sul are a bug or not [05:54:16] right [05:56:44] apergos: this was my test the other day: https://toolserver.org/~quentinv57/sulinfo/Testing9083020932 [05:57:59] I would bz it and flag someone that knows the current state of centralauth on it [05:58:46] ok. i can also look for other bugs about unattached accounts. i know they exist but i thought they were mostly about existing users growing unattached not brand new users [05:59:11] uh huh [05:59:46] anything too old isn't worth it because it will have been about a different state of the code, befure we said all accounts will/shuld be universal [06:00:01] right [06:00:20] i guess i talk to the people managing loginwiki transition, they should know. hopefully [06:01:11] cool [06:01:47] 14 05:56:07 < snitch> [[Wikimedia Forum]]; John Vandenberg; /* Minangkabau Wikipedia */ ping RobiH and Erik Zachte; https://meta.wikimedia.org/w/index.php?diff=5649650&oldid=5648876&rcid=4369215 [06:02:01] you think that's supposed to be RobH ? but he doesn't seem relevant [06:02:10] i don't know a RobiH [06:04:54] no [06:05:04] I think it's User:RobiH that's referenced in the diff even [06:05:11] http://meta.wikimedia.org/wiki/User:RobiH [06:05:12] this person [06:05:18] nice try though ;-) [06:06:46] well at least they don't have exactly the same name... we certainly have an abundance of /\beri[ck]+\b/ [06:06:55] here is the ... well not proof but [06:06:56] http://meta.wikimedia.org/wiki/User_talk:RobiH [06:07:17] first comment, you can see this is the person that has to do with the statistical lists [06:07:20] from 2006 even... [06:08:04] yeah, i'm looking now [06:18:14] my reply: http://dpaste.com/1299267/plain/ [06:18:37] great [06:22:26] whoa, this whole time i thought i was in #-tech [06:22:33] nope [06:22:34] * jeremyb must be sleepy [06:22:39] must be bedtime [06:22:56] * apergos is slowly waking up (it's a sunday, I'm entitled) [06:23:07] you certainly are! [06:25:19] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [06:26:18] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.999 second response time on port 11211 [06:29:18] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [07:31:48] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [07:31:48] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:31:48] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:31:48] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [07:31:48] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [07:31:48] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [07:31:48] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [10:02:49] New patchset: TTO; "(bug 51312) add rollbacker group for ckbwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73608 [10:51:05] PROBLEM - Puppet freshness on grosley is CRITICAL: No successful Puppet run in the last 10 hours [10:59:05] PROBLEM - Puppet freshness on mw56 is CRITICAL: No successful Puppet run in the last 10 hours [11:05:10] PROBLEM - DPKG on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:10] PROBLEM - DPKG on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:10] PROBLEM - RAID on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:10] PROBLEM - Apache HTTP on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:10] PROBLEM - Apache HTTP on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:10] PROBLEM - DPKG on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:10] PROBLEM - DPKG on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:11] PROBLEM - RAID on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:11] PROBLEM - RAID on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:20] PROBLEM - Apache HTTP on mw1137 is CRITICAL: Connection timed out [11:05:20] PROBLEM - Apache HTTP on mw1140 is CRITICAL: Connection timed out [11:05:20] PROBLEM - SSH on mw1114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:20] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:21] PROBLEM - DPKG on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:21] PROBLEM - twemproxy process on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:21] PROBLEM - DPKG on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:21] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:22] PROBLEM - RAID on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:22] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:23] PROBLEM - DPKG on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:23] PROBLEM - RAID on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:24] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:24] PROBLEM - Apache HTTP on mw1133 is CRITICAL: Connection timed out [11:05:25] PROBLEM - Apache HTTP on mw1139 is CRITICAL: Connection timed out [11:05:25] PROBLEM - Apache HTTP on mw1115 is CRITICAL: Connection timed out [11:05:26] PROBLEM - Apache HTTP on mw1135 is CRITICAL: Connection timed out [11:05:30] PROBLEM - RAID on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:30] PROBLEM - RAID on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:30] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:05:30] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:05:30] PROBLEM - twemproxy process on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:06:40] PROBLEM - SSH on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:06:40] RECOVERY - RAID on mw1137 is OK: OK: no RAID installed [11:06:40] RECOVERY - Disk space on mw1146 is OK: DISK OK [11:06:40] RECOVERY - Apache HTTP on mw1148 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.313 second response time [11:07:10] RECOVERY - Apache HTTP on mw1208 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 7.275 second response time [11:07:10] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.700 second response time [11:07:20] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:07:20] RECOVERY - Apache HTTP on mw1120 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.051 second response time [11:07:21] RECOVERY - Disk space on mw1131 is OK: DISK OK [11:07:30] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:07:31] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:07:31] RECOVERY - DPKG on mw1148 is OK: All packages OK [11:07:40] RECOVERY - DPKG on mw1139 is OK: All packages OK [11:07:40] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:07:40] PROBLEM - Disk space on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:07:40] PROBLEM - twemproxy process on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:07:40] RECOVERY - DPKG on mw1127 is OK: All packages OK [11:07:41] RECOVERY - Apache HTTP on mw1127 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.275 second response time [11:07:41] RECOVERY - RAID on mw1127 is OK: OK: no RAID installed [11:07:42] PROBLEM - Disk space on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:07:42] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:10] RECOVERY - twemproxy process on mw1146 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:08:10] RECOVERY - Apache HTTP on mw1126 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 9.457 second response time [11:08:20] PROBLEM - twemproxy process on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:08:20] RECOVERY - twemproxy process on mw1117 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:08:21] RECOVERY - Apache HTTP on mw1134 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.080 second response time [11:08:21] RECOVERY - SSH on mw1133 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:08:30] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:08:30] PROBLEM - Apache HTTP on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:30] RECOVERY - Disk space on mw1118 is OK: DISK OK [11:08:31] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.990 second response time [11:08:31] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:08:31] RECOVERY - twemproxy process on mw1126 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:08:31] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:08:31] RECOVERY - Disk space on mw1133 is OK: DISK OK [11:08:40] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:08:40] PROBLEM - twemproxy process on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:08:40] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:08:40] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:40] RECOVERY - twemproxy process on mw1122 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:08:40] PROBLEM - twemproxy process on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:08:41] RECOVERY - twemproxy process on mw1133 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:08:41] RECOVERY - twemproxy process on mw1136 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:08:42] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:08:42] RECOVERY - DPKG on mw1141 is OK: All packages OK [11:09:21] RECOVERY - Apache HTTP on mw1145 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 6.041 second response time [11:09:30] RECOVERY - RAID on mw1145 is OK: OK: no RAID installed [11:09:30] PROBLEM - RAID on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:09:30] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:09:30] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:09:30] RECOVERY - SSH on mw1132 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:09:31] RECOVERY - Apache HTTP on mw1146 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.521 second response time [11:09:31] RECOVERY - Disk space on mw1121 is OK: DISK OK [11:09:32] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:09:32] RECOVERY - Disk space on mw1132 is OK: DISK OK [11:09:33] RECOVERY - Disk space on mw1138 is OK: DISK OK [11:09:33] RECOVERY - SSH on mw1117 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:09:40] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:09:40] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:09:40] PROBLEM - RAID on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:09:40] RECOVERY - DPKG on mw1146 is OK: All packages OK [11:09:40] PROBLEM - Apache HTTP on mw1124 is CRITICAL: Connection timed out [11:09:41] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:09:41] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:09:42] RECOVERY - Disk space on mw1114 is OK: DISK OK [11:09:50] PROBLEM - Apache HTTP on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:00] RECOVERY - RAID on mw1146 is OK: OK: no RAID installed [11:10:20] RECOVERY - SSH on mw1147 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:10:20] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:21] RECOVERY - SSH on mw1129 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:10:21] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:30] RECOVERY - RAID on mw1124 is OK: OK: no RAID installed [11:10:30] PROBLEM - twemproxy process on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:10:30] PROBLEM - Disk space on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:10:30] PROBLEM - Apache HTTP on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:31] RECOVERY - Disk space on mw1130 is OK: DISK OK [11:10:31] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:31] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:10:40] RECOVERY - twemproxy process on mw1130 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:10:40] RECOVERY - Apache HTTP on mw1119 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.180 second response time [11:10:40] PROBLEM - SSH on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:40] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:10:40] PROBLEM - DPKG on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:10:40] RECOVERY - DPKG on mw1130 is OK: All packages OK [11:10:40] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:10:41] PROBLEM - Disk space on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:10:41] PROBLEM - Apache HTTP on mw1127 is CRITICAL: Connection timed out [11:10:42] PROBLEM - RAID on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:10:42] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:50] PROBLEM - twemproxy process on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:10:50] PROBLEM - DPKG on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:20] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 7.871 second response time [11:11:20] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:11:21] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: Connection timed out [11:11:30] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:30] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:11:31] RECOVERY - Disk space on mw1148 is OK: DISK OK [11:11:31] PROBLEM - SSH on mw1133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:31] PROBLEM - SSH on mw1122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:40] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:11:40] RECOVERY - Apache HTTP on mw1148 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [11:11:40] PROBLEM - SSH on mw1115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:40] PROBLEM - Disk space on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:40] PROBLEM - Apache HTTP on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:40] PROBLEM - Disk space on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:41] PROBLEM - SSH on mw1140 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:41] PROBLEM - Disk space on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:42] PROBLEM - DPKG on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:42] PROBLEM - twemproxy process on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:43] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:43] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:44] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:44] RECOVERY - DPKG on mw1127 is OK: All packages OK [11:11:45] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:45] PROBLEM - twemproxy process on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:46] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:46] PROBLEM - SSH on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:47] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:50] PROBLEM - twemproxy process on mw1136 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:11:50] PROBLEM - DPKG on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:20] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 0.079 second response time [11:12:22] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:30] RECOVERY - Disk space on mw1137 is OK: DISK OK [11:12:30] PROBLEM - SSH on mw1136 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:30] PROBLEM - SSH on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:30] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:30] PROBLEM - Apache HTTP on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:30] PROBLEM - twemproxy process on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:31] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:31] PROBLEM - RAID on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:32] PROBLEM - Apache HTTP on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:32] RECOVERY - DPKG on mw1118 is OK: All packages OK [11:12:33] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:33] PROBLEM - Disk space on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:34] PROBLEM - Disk space on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:34] PROBLEM - SSH on mw1132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:35] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:12:40] PROBLEM - Apache HTTP on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:40] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:40] PROBLEM - SSH on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:40] PROBLEM - twemproxy process on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:40] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:40] PROBLEM - twemproxy process on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:41] PROBLEM - Disk space on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:42] PROBLEM - Disk space on mw1136 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:42] PROBLEM - SSH on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:43] PROBLEM - Disk space on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:43] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:44] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:44] PROBLEM - Disk space on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:45] PROBLEM - Disk space on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:12:45] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:10] PROBLEM - RAID on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:10] PROBLEM - Apache HTTP on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:20] RECOVERY - Apache HTTP on mw1121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.064 second response time [11:13:20] RECOVERY - RAID on mw1118 is OK: OK: no RAID installed [11:13:30] PROBLEM - Disk space on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:30] PROBLEM - SSH on mw1147 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:30] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:30] PROBLEM - DPKG on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:30] PROBLEM - Disk space on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:31] RECOVERY - Apache HTTP on mw1118 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.390 second response time [11:13:31] PROBLEM - SSH on mw1129 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:32] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:32] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:40] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:40] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:40] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:40] PROBLEM - DPKG on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:50] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:50] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:14:20] PROBLEM - twemproxy process on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:14:20] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:14:21] RECOVERY - Disk space on mw1123 is OK: DISK OK [11:14:21] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:30] RECOVERY - Disk space on mw1138 is OK: DISK OK [11:14:30] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:14:31] RECOVERY - Disk space on mw1121 is OK: DISK OK [11:14:31] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:31] RECOVERY - SSH on mw1115 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:14:40] RECOVERY - twemproxy process on mw1138 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:14:40] PROBLEM - SSH on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:40] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:14:40] PROBLEM - Disk space on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:14:40] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:14:50] PROBLEM - Apache HTTP on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:50] PROBLEM - Apache HTTP on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:50] PROBLEM - DPKG on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:14:50] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:15:10] RECOVERY - Apache HTTP on mw1123 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 7.743 second response time [11:15:20] RECOVERY - SSH on mw1147 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:15:30] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:15:40] PROBLEM - twemproxy process on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:15:40] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:15:40] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:16:21] RECOVERY - SSH on mw1128 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:16:35] PROBLEM - RAID on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:16:35] PROBLEM - RAID on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:16:35] RECOVERY - DPKG on mw1121 is OK: All packages OK [11:16:35] PROBLEM - SSH on mw1124 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:40] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:17:20] RECOVERY - SSH on mw1136 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:17:30] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:31] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:17:31] RECOVERY - Disk space on mw1130 is OK: DISK OK [11:17:31] RECOVERY - Apache HTTP on mw1130 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.361 second response time [11:17:40] PROBLEM - SSH on mw1115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:40] PROBLEM - twemproxy process on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:17:40] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:18:10] PROBLEM - Apache HTTP on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:18:30] PROBLEM - SSH on mw1147 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:18:30] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:18:40] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:18:40] PROBLEM - Disk space on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:18:40] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:19:20] RECOVERY - RAID on mw1118 is OK: OK: no RAID installed [11:19:30] PROBLEM - SSH on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:30] RECOVERY - DPKG on mw1118 is OK: All packages OK [11:19:31] PROBLEM - Disk space on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:19:40] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:40] PROBLEM - DPKG on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:19:40] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:19:40] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:19:40] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:30] PROBLEM - SSH on mw1136 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:30] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:20:30] RECOVERY - Disk space on mw1121 is OK: DISK OK [11:20:31] RECOVERY - DPKG on mw1121 is OK: All packages OK [11:20:31] RECOVERY - Disk space on mw1138 is OK: DISK OK [11:20:40] RECOVERY - twemproxy process on mw1138 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:20:40] RECOVERY - Apache HTTP on mw1148 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.324 second response time [11:20:40] PROBLEM - Apache HTTP on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:40] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:20:40] PROBLEM - twemproxy process on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:20:40] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:10] RECOVERY - Disk space on mw1139 is OK: DISK OK [11:21:30] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:40] RECOVERY - Disk space on mw1147 is OK: DISK OK [11:21:40] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:21:40] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:50] RECOVERY - twemproxy process on mw1147 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:22:20] RECOVERY - SSH on mw1147 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:22:20] RECOVERY - Apache HTTP on mw1145 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [11:22:20] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: Connection timed out [11:22:30] PROBLEM - RAID on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:22:30] RECOVERY - RAID on mw1147 is OK: OK: no RAID installed [11:22:30] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:22:40] RECOVERY - Disk space on mw1145 is OK: DISK OK [11:22:40] PROBLEM - Apache HTTP on mw1118 is CRITICAL: Connection timed out [11:22:40] RECOVERY - Disk space on mw1129 is OK: DISK OK [11:22:40] RECOVERY - RAID on mw1121 is OK: OK: no RAID installed [11:22:50] RECOVERY - twemproxy process on mw1129 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:23:21] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 0.086 second response time [11:23:30] RECOVERY - Disk space on mw1148 is OK: DISK OK [11:23:31] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:23:31] RECOVERY - RAID on mw1148 is OK: OK: no RAID installed [11:23:31] RECOVERY - DPKG on mw1148 is OK: All packages OK [11:23:40] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:23:40] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:23:41] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:23:41] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:23:41] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:41] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:20] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:24:20] RECOVERY - SSH on mw1128 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:24:21] RECOVERY - Disk space on mw1141 is OK: DISK OK [11:24:32] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.968 second response time [11:24:40] PROBLEM - twemproxy process on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:24:41] RECOVERY - Apache HTTP on mw1143 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 6.357 second response time [11:24:41] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:24:50] PROBLEM - twemproxy process on mw1147 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:25:30] PROBLEM - SSH on mw1147 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:30] PROBLEM - Apache HTTP on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:40] RECOVERY - Disk space on mw1118 is OK: DISK OK [11:25:40] PROBLEM - Disk space on mw1147 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:25:40] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:25:40] PROBLEM - RAID on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:25:40] PROBLEM - DPKG on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:25:41] PROBLEM - RAID on mw1147 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:25:41] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:25:41] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:42] PROBLEM - Disk space on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:25:42] PROBLEM - Disk space on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:25:50] PROBLEM - twemproxy process on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:25:50] PROBLEM - twemproxy process on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:26:20] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:26:31] RECOVERY - Disk space on mw1145 is OK: DISK OK [11:26:40] RECOVERY - twemproxy process on mw1138 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:26:41] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:21] RECOVERY - Apache HTTP on mw1145 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.240 second response time [11:27:30] PROBLEM - SSH on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:30] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:27:31] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:31] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:27:40] RECOVERY - Apache HTTP on mw1130 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 7.356 second response time [11:27:40] PROBLEM - Disk space on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:27:40] PROBLEM - Apache HTTP on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:27:40] PROBLEM - DPKG on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:27:41] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:27:41] PROBLEM - RAID on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:27:41] PROBLEM - Apache HTTP on mw1148 is CRITICAL: Connection timed out [11:27:41] RECOVERY - Disk space on mw1129 is OK: DISK OK [11:27:44] notpeter: EMERGENCY [11:27:50] PROBLEM - Apache HTTP on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:28:34] Ryan_Lane: ping [11:28:40] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:28:40] RECOVERY - Disk space on mw1130 is OK: DISK OK [11:29:30] PROBLEM - Disk space on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:29:31] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:31] Reedy: you? [11:29:40] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:29:40] PROBLEM - twemproxy process on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:30:10] RECOVERY - Disk space on mw1120 is OK: DISK OK [11:30:30] PROBLEM - Apache HTTP on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:30] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:30:40] RECOVERY - Disk space on mw1146 is OK: DISK OK [11:30:40] PROBLEM - Apache HTTP on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:42] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:30:42] PROBLEM - Disk space on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:30:42] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:40] RECOVERY - Disk space on mw1128 is OK: DISK OK [11:31:40] RECOVERY - Disk space on mw1140 is OK: DISK OK [11:31:40] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:40] PROBLEM - Disk space on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:31:49] MaxSem: ... [11:32:20] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:32:21] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:32:21] RECOVERY - Apache HTTP on mw1117 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.993 second response time [11:32:21] RECOVERY - Apache HTTP on mw1120 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.349 second response time [11:32:21] RECOVERY - twemproxy process on mw1145 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:32:21] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:32:30] RECOVERY - DPKG on mw1145 is OK: All packages OK [11:32:30] RECOVERY - Apache HTTP on mw1130 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [11:32:30] RECOVERY - Disk space on mw1138 is OK: DISK OK [11:32:30] RECOVERY - Disk space on mw1123 is OK: DISK OK [11:32:31] RECOVERY - twemproxy process on mw1130 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:32:31] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:32:40] RECOVERY - DPKG on mw1130 is OK: All packages OK [11:32:40] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:33:20] RECOVERY - SSH on mw1114 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:33:20] RECOVERY - RAID on mw1130 is OK: OK: no RAID installed [11:33:30] RECOVERY - SSH on mw1124 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:33:31] RECOVERY - Disk space on mw1145 is OK: DISK OK [11:34:20] RECOVERY - Apache HTTP on mw1145 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.291 second response time [11:34:31] RECOVERY - Disk space on mw1118 is OK: DISK OK [11:34:40] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:34:40] PROBLEM - Disk space on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:34:40] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:34:40] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:35:20] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:35:20] RECOVERY - RAID on mw1118 is OK: OK: no RAID installed [11:35:21] RECOVERY - RAID on mw1145 is OK: OK: no RAID installed [11:35:30] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:30] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:35:30] PROBLEM - Apache HTTP on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:31] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:35:31] PROBLEM - Disk space on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:35:31] RECOVERY - Disk space on mw1124 is OK: DISK OK [11:35:40] RECOVERY - twemproxy process on mw1124 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:35:40] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:40] RECOVERY - Disk space on mw1134 is OK: DISK OK [11:35:40] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:50] RECOVERY - DPKG on mw1146 is OK: All packages OK [11:36:10] RECOVERY - twemproxy process on mw1146 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:36:20] PROBLEM - SSH on mw1114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:36:30] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:36:30] RECOVERY - Disk space on mw1138 is OK: DISK OK [11:36:40] PROBLEM - Apache HTTP on mw1130 is CRITICAL: Connection timed out [11:36:40] PROBLEM - DPKG on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:37:10] RECOVERY - Disk space on mw1120 is OK: DISK OK [11:37:20] RECOVERY - Disk space on mw1141 is OK: DISK OK [11:37:30] RECOVERY - Disk space on mw1131 is OK: DISK OK [11:37:30] PROBLEM - RAID on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:37:31] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:37:40] RECOVERY - twemproxy process on mw1138 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:37:40] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:37:40] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:37:40] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:37:40] PROBLEM - twemproxy process on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:37:50] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:50] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:00] RECOVERY - RAID on mw1138 is OK: OK: no RAID installed [11:38:20] RECOVERY - DPKG on mw1138 is OK: All packages OK [11:38:20] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: Connection timed out [11:38:26] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:38:27] RECOVERY - Apache HTTP on mw1138 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 7.116 second response time [11:38:30] PROBLEM - RAID on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:38:30] PROBLEM - Apache HTTP on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:40] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:38:50] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:39:30] PROBLEM - Apache HTTP on mw1145 is CRITICAL: Connection timed out [11:39:30] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:30] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:39:40] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:39:40] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:39:50] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:40:20] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 0.068 second response time [11:40:22] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:40:22] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:40:30] PROBLEM - RAID on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:40:30] PROBLEM - Disk space on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:40:30] PROBLEM - DPKG on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:40:40] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:40:40] RECOVERY - Disk space on mw1118 is OK: DISK OK [11:40:41] RECOVERY - Disk space on mw1148 is OK: DISK OK [11:40:41] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:40:41] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:40:41] RECOVERY - Disk space on mw1142 is OK: DISK OK [11:40:41] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:40:50] PROBLEM - twemproxy process on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:41:20] PROBLEM - Apache HTTP on mw1138 is CRITICAL: Connection timed out [11:41:30] PROBLEM - twemproxy process on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:41:41] RECOVERY - RAID on mw1120 is OK: OK: no RAID installed [11:41:41] RECOVERY - DPKG on mw1120 is OK: All packages OK [11:41:41] RECOVERY - twemproxy process on mw1124 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:41:41] RECOVERY - SSH on mw1140 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:41:50] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:10] PROBLEM - RAID on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:42:20] PROBLEM - twemproxy process on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:42:20] RECOVERY - SSH on mw1128 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:42:30] PROBLEM - DPKG on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:42:30] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:42:30] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:42:30] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:50] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:50] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:42:50] PROBLEM - twemproxy process on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:42:50] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:20] RECOVERY - twemproxy process on mw1145 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:43:21] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:43:35] RECOVERY - Disk space on mw1131 is OK: DISK OK [11:43:35] RECOVERY - twemproxy process on mw1117 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:43:35] RECOVERY - Disk space on mw1117 is OK: DISK OK [11:43:35] RECOVERY - DPKG on mw1145 is OK: All packages OK [11:43:41] RECOVERY - SSH on mw1117 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:43:41] RECOVERY - DPKG on mw1118 is OK: All packages OK [11:43:41] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.459 second response time [11:43:41] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:43:41] RECOVERY - Disk space on mw1130 is OK: DISK OK [11:43:42] RECOVERY - twemproxy process on mw1130 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:43:50] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:43:50] PROBLEM - Disk space on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:43:50] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:43:50] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:50] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:43:50] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:00] RECOVERY - Apache HTTP on mw1123 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [11:44:10] RECOVERY - DPKG on mw1117 is OK: All packages OK [11:44:20] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:44:21] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:44:30] RECOVERY - Disk space on mw1127 is OK: DISK OK [11:44:30] RECOVERY - Disk space on mw1123 is OK: DISK OK [11:44:30] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:30] PROBLEM - Apache HTTP on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:30] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:44:40] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:44:50] RECOVERY - twemproxy process on mw1123 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:44:50] PROBLEM - RAID on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:44:50] PROBLEM - DPKG on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:44:50] PROBLEM - SSH on mw1140 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:50] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:45:10] RECOVERY - DPKG on mw1123 is OK: All packages OK [11:45:20] RECOVERY - Apache HTTP on mw1121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.195 second response time [11:45:20] RECOVERY - Apache HTTP on mw1134 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.986 second response time [11:45:21] RECOVERY - RAID on mw1145 is OK: OK: no RAID installed [11:45:21] RECOVERY - Disk space on mw1141 is OK: DISK OK [11:45:30] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:45:30] PROBLEM - SSH on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:45:40] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:45:40] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:45:41] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:45:41] RECOVERY - Disk space on mw1148 is OK: DISK OK [11:45:41] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:45:41] RECOVERY - DPKG on mw1148 is OK: All packages OK [11:45:41] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:45:50] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:45:50] RECOVERY - Disk space on mw1121 is OK: DISK OK [11:45:50] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:46:10] RECOVERY - Disk space on mw1139 is OK: DISK OK [11:46:10] RECOVERY - Disk space on mw1120 is OK: DISK OK [11:46:20] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:46:24] cmjohnson1: ping [11:46:30] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:46:30] PROBLEM - Disk space on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:46:30] PROBLEM - twemproxy process on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:46:40] RECOVERY - RAID on mw1123 is OK: OK: no RAID installed [11:46:41] RECOVERY - Disk space on mw1128 is OK: DISK OK [11:46:41] RECOVERY - Disk space on mw1142 is OK: DISK OK [11:46:50] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:46:50] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:46:50] PROBLEM - SSH on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:50] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:46:50] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:46:51] PROBLEM - Apache HTTP on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:51] PROBLEM - twemproxy process on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:46:52] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:46:52] PROBLEM - Disk space on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:46:53] PROBLEM - twemproxy process on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:46:53] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:54] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:54] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:00] RECOVERY - DPKG on mw1142 is OK: All packages OK [11:47:10] RECOVERY - SSH on mw1114 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:47:10] PROBLEM - DPKG on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:47:20] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: Connection timed out [11:47:22] RECOVERY - Disk space on mw1143 is OK: DISK OK [11:47:30] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:47:30] PROBLEM - Disk space on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:47:31] PROBLEM - Disk space on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:47:40] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:47:40] RECOVERY - RAID on mw1148 is OK: OK: no RAID installed [11:47:41] RECOVERY - Apache HTTP on mw1119 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.314 second response time [11:47:41] RECOVERY - Disk space on mw1146 is OK: DISK OK [11:47:41] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:48:20] RECOVERY - twemproxy process on mw1146 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:48:20] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 0.071 second response time [11:48:22] RECOVERY - SSH on mw1128 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:48:22] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:48:30] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:48:30] PROBLEM - Apache HTTP on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:30] RECOVERY - Disk space on mw1138 is OK: DISK OK [11:48:30] RECOVERY - Disk space on mw1126 is OK: DISK OK [11:48:30] PROBLEM - SSH on mw1124 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:40] RECOVERY - DPKG on mw1146 is OK: All packages OK [11:48:50] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:48:50] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:48:50] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:48:50] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:50] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:50] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:48:55] guillom: ping [11:49:10] RECOVERY - RAID on mw1146 is OK: OK: no RAID installed [11:49:10] PROBLEM - Apache HTTP on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:20] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:49:20] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:49:30] PROBLEM - RAID on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:49:30] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:49:30] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:30] PROBLEM - DPKG on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:49:40] RECOVERY - RAID on mw1121 is OK: OK: no RAID installed [11:49:40] RECOVERY - DPKG on mw1121 is OK: All packages OK [11:49:40] RECOVERY - Disk space on mw1124 is OK: DISK OK [11:49:40] RECOVERY - twemproxy process on mw1124 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:49:50] RECOVERY - twemproxy process on mw1126 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:49:50] PROBLEM - Disk space on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:50:10] PROBLEM - DPKG on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:50:20] PROBLEM - DPKG on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:50:20] RECOVERY - SSH on mw1124 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:50:21] RECOVERY - DPKG on mw1124 is OK: All packages OK [11:50:30] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:50:30] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:50:31] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:50:40] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:50:40] RECOVERY - Disk space on mw1118 is OK: DISK OK [11:50:41] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:50:50] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:50:50] PROBLEM - twemproxy process on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:50:50] PROBLEM - Apache HTTP on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:50:50] PROBLEM - RAID on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:51:20] RECOVERY - RAID on mw1124 is OK: OK: no RAID installed [11:51:30] PROBLEM - twemproxy process on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:51:30] PROBLEM - SSH on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:30] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:51:30] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:51:30] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:40] RECOVERY - SSH on mw1140 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:51:50] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:51:50] PROBLEM - RAID on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:51:50] PROBLEM - Disk space on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:51:50] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:51:50] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:52:10] PROBLEM - RAID on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:52:20] RECOVERY - Apache HTTP on mw1134 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.315 second response time [11:52:21] RECOVERY - Disk space on mw1117 is OK: DISK OK [11:52:21] RECOVERY - twemproxy process on mw1117 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:52:30] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:40] RECOVERY - SSH on mw1117 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:52:40] RECOVERY - SSH on mw1115 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:52:41] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:52:41] RECOVERY - twemproxy process on mw1122 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:52:41] RECOVERY - Disk space on mw1115 is OK: DISK OK [11:52:50] RECOVERY - Disk space on mw1130 is OK: DISK OK [11:52:50] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:50] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:52:50] PROBLEM - DPKG on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:52:50] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:52:50] PROBLEM - RAID on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:52:51] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:53:00] RECOVERY - DPKG on mw1117 is OK: All packages OK [11:53:10] RECOVERY - Disk space on mw1120 is OK: DISK OK [11:53:20] PROBLEM - SSH on mw1114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:20] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:53:21] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:53:21] RECOVERY - Apache HTTP on mw1120 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.066 second response time [11:53:21] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:53:40] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:53:40] RECOVERY - twemproxy process on mw1140 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:53:40] RECOVERY - RAID on mw1120 is OK: OK: no RAID installed [11:53:40] RECOVERY - DPKG on mw1120 is OK: All packages OK [11:53:41] RECOVERY - Disk space on mw1140 is OK: DISK OK [11:53:41] RECOVERY - Apache HTTP on mw1124 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.919 second response time [11:53:41] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:53:42] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:53:42] RECOVERY - twemproxy process on mw1131 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:53:50] RECOVERY - DPKG on mw1131 is OK: All packages OK [11:53:50] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:54:10] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.186 second response time [11:54:20] PROBLEM - twemproxy process on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:54:20] RECOVERY - Disk space on mw1122 is OK: DISK OK [11:54:33] RECOVERY - SSH on mw1128 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:54:33] RECOVERY - twemproxy process on mw1145 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:54:33] RECOVERY - DPKG on mw1145 is OK: All packages OK [11:54:33] PROBLEM - Disk space on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:54:33] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:54:40] RECOVERY - Disk space on mw1145 is OK: DISK OK [11:54:50] RECOVERY - Disk space on mw1114 is OK: DISK OK [11:54:50] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:50] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:54:50] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:50] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:54:51] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:54:51] PROBLEM - DPKG on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:55:10] RECOVERY - SSH on mw1114 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:55:10] RECOVERY - DPKG on mw1114 is OK: All packages OK [11:55:21] RECOVERY - Disk space on mw1126 is OK: DISK OK [11:55:30] RECOVERY - Disk space on mw1123 is OK: DISK OK [11:55:30] PROBLEM - twemproxy process on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:55:30] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:55:30] PROBLEM - Disk space on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:55:40] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:55:40] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:55:41] RECOVERY - DPKG on mw1127 is OK: All packages OK [11:55:41] RECOVERY - Disk space on mw1142 is OK: DISK OK [11:55:41] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:55:41] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:55:50] PROBLEM - SSH on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:50] PROBLEM - SSH on mw1115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:50] PROBLEM - twemproxy process on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:55:50] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:55:50] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:55:50] PROBLEM - Disk space on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:56:10] RECOVERY - twemproxy process on mw1146 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:56:10] PROBLEM - DPKG on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:56:20] RECOVERY - Disk space on mw1141 is OK: DISK OK [11:56:21] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:56:40] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:56:40] RECOVERY - SSH on mw1119 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:56:40] RECOVERY - Disk space on mw1146 is OK: DISK OK [11:56:41] RECOVERY - Disk space on mw1119 is OK: DISK OK [11:56:41] RECOVERY - Disk space on mw1118 is OK: DISK OK [11:56:41] RECOVERY - RAID on mw1127 is OK: OK: no RAID installed [11:56:41] RECOVERY - Apache HTTP on mw1119 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.822 second response time [11:56:42] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:56:42] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:56:50] RECOVERY - DPKG on mw1118 is OK: All packages OK [11:56:50] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:50] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:56:50] PROBLEM - DPKG on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:56:50] PROBLEM - SSH on mw1140 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:56:51] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:56:51] PROBLEM - RAID on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:56:52] PROBLEM - twemproxy process on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:56:52] PROBLEM - Disk space on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:56:53] PROBLEM - DPKG on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:57:00] RECOVERY - RAID on mw1119 is OK: OK: no RAID installed [11:57:00] RECOVERY - RAID on mw1146 is OK: OK: no RAID installed [11:57:10] RECOVERY - Disk space on mw1125 is OK: DISK OK [11:57:20] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:57:20] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:57:21] RECOVERY - Disk space on mw1138 is OK: DISK OK [11:57:30] RECOVERY - twemproxy process on mw1117 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:57:30] PROBLEM - Apache HTTP on mw1120 is CRITICAL: Connection timed out [11:57:30] RECOVERY - Disk space on mw1117 is OK: DISK OK [11:57:30] RECOVERY - Disk space on mw1143 is OK: DISK OK [11:57:30] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:57:30] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:57:31] PROBLEM - Disk space on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:57:31] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:57:32] PROBLEM - twemproxy process on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:57:32] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:57:33] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:57:33] PROBLEM - DPKG on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:57:40] RECOVERY - SSH on mw1117 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:57:40] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:57:41] RECOVERY - RAID on mw1121 is OK: OK: no RAID installed [11:57:41] RECOVERY - twemproxy process on mw1138 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:57:41] RECOVERY - DPKG on mw1121 is OK: All packages OK [11:57:41] RECOVERY - SSH on mw1140 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:57:50] RECOVERY - DPKG on mw1146 is OK: All packages OK [11:57:50] PROBLEM - Disk space on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:57:50] PROBLEM - Disk space on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:57:50] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:57:53] apergos: ping [11:58:00] RECOVERY - Apache HTTP on mw1126 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [11:58:00] already looking [11:58:03] ok [11:58:10] RECOVERY - DPKG on mw1119 is OK: All packages OK [11:58:10] jsut got back in [11:58:10] RECOVERY - RAID on mw1138 is OK: OK: no RAID installed [11:58:10] PROBLEM - DPKG on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:58:18] check the other channel [11:58:20] RECOVERY - DPKG on mw1138 is OK: All packages OK [11:58:21] RECOVERY - Disk space on mw1131 is OK: DISK OK [11:58:30] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:30] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:58:40] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:58:41] RECOVERY - DPKG on mw1131 is OK: All packages OK [11:58:41] RECOVERY - Disk space on mw1144 is OK: DISK OK [11:58:41] RECOVERY - Disk space on mw1145 is OK: DISK OK [11:58:41] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:58:50] PROBLEM - twemproxy process on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:58:50] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:50] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:58:50] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:50] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:59:10] RECOVERY - Disk space on mw1120 is OK: DISK OK [11:59:20] RECOVERY - RAID on mw1131 is OK: OK: no RAID installed [11:59:21] RECOVERY - twemproxy process on mw1145 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:59:21] RECOVERY - RAID on mw1145 is OK: OK: no RAID installed [11:59:21] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:59:21] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:59:21] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:30] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:59:30] PROBLEM - Apache HTTP on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:30] RECOVERY - DPKG on mw1145 is OK: All packages OK [11:59:30] RECOVERY - Disk space on mw1126 is OK: DISK OK [11:59:30] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:59:40] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:59:40] RECOVERY - Apache HTTP on mw1131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [11:59:40] RECOVERY - Disk space on mw1128 is OK: DISK OK [11:59:50] RECOVERY - twemproxy process on mw1126 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [11:59:50] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:50] PROBLEM - SSH on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:50] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:50] PROBLEM - RAID on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:59:51] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:59:51] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:59:52] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:59:52] PROBLEM - DPKG on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:00:10] PROBLEM - RAID on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:00:21] PROBLEM - twemproxy process on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:00:21] RECOVERY - DPKG on mw1126 is OK: All packages OK [12:00:21] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:00:21] RECOVERY - Apache HTTP on mw1145 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.490 second response time [12:00:30] PROBLEM - Disk space on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:00:30] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:00:30] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:00:31] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:40] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:00:41] RECOVERY - Disk space on mw1118 is OK: DISK OK [12:00:41] RECOVERY - Disk space on mw1114 is OK: DISK OK [12:00:50] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:00:50] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:50] PROBLEM - SSH on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:50] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:00:50] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:00:50] PROBLEM - SSH on mw1140 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:51] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:01:00] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:01:20] PROBLEM - SSH on mw1114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:20] RECOVERY - Disk space on mw1141 is OK: DISK OK [12:01:21] RECOVERY - Disk space on mw1127 is OK: DISK OK [12:01:21] RECOVERY - Disk space on mw1123 is OK: DISK OK [12:01:21] PROBLEM - Apache HTTP on mw1121 is CRITICAL: Connection timed out [12:01:30] RECOVERY - Disk space on mw1137 is OK: DISK OK [12:01:30] PROBLEM - twemproxy process on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:01:31] RECOVERY - Disk space on mw1143 is OK: DISK OK [12:01:31] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:01:31] PROBLEM - Disk space on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:01:40] RECOVERY - twemproxy process on mw1123 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:01:50] RECOVERY - Disk space on mw1148 is OK: DISK OK [12:01:50] PROBLEM - DPKG on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:01:50] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:01:50] PROBLEM - RAID on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:01:50] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:01:51] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:10] RECOVERY - DPKG on mw1114 is OK: All packages OK [12:02:10] RECOVERY - DPKG on mw1144 is OK: All packages OK [12:02:20] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:02:30] PROBLEM - DPKG on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:02:30] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:02:30] PROBLEM - SSH on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:30] PROBLEM - Disk space on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:02:40] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:02:40] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:02:40] RECOVERY - Disk space on mw1134 is OK: DISK OK [12:02:40] RECOVERY - Disk space on mw1146 is OK: DISK OK [12:02:40] RECOVERY - SSH on mw1140 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:02:41] RECOVERY - Disk space on mw1144 is OK: DISK OK [12:02:41] RECOVERY - SSH on mw1134 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:02:41] RECOVERY - Disk space on mw1140 is OK: DISK OK [12:02:42] RECOVERY - DPKG on mw1118 is OK: All packages OK [12:02:50] PROBLEM - Apache HTTP on mw1131 is CRITICAL: Connection timed out [12:02:50] PROBLEM - Apache HTTP on mw1119 is CRITICAL: Connection timed out [12:02:50] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:02:50] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:02:50] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:50] PROBLEM - twemproxy process on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:02:50] PROBLEM - DPKG on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:02:51] PROBLEM - Disk space on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:02:51] PROBLEM - twemproxy process on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:10] RECOVERY - twemproxy process on mw1146 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:03:10] PROBLEM - RAID on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:11] RECOVERY - Disk space on mw1125 is OK: DISK OK [12:03:20] PROBLEM - DPKG on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:20] RECOVERY - twemproxy process on mw1117 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:03:30] PROBLEM - RAID on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:30] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:30] PROBLEM - DPKG on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:30] PROBLEM - Disk space on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:30] PROBLEM - RAID on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:31] PROBLEM - twemproxy process on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:31] PROBLEM - Apache HTTP on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:32] PROBLEM - DPKG on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:40] RECOVERY - Apache HTTP on mw1114 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time [12:03:40] RECOVERY - Disk space on mw1132 is OK: DISK OK [12:03:50] PROBLEM - twemproxy process on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:50] PROBLEM - Disk space on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:50] PROBLEM - twemproxy process on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:03:50] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:50] PROBLEM - SSH on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:10] RECOVERY - Disk space on mw1120 is OK: DISK OK [12:04:10] PROBLEM - RAID on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:04:10] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [12:04:20] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:04:21] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:04:21] RECOVERY - SSH on mw1132 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:04:21] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:04:30] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:04:30] RECOVERY - Disk space on mw1138 is OK: DISK OK [12:04:30] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:04:30] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:04:30] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:04:40] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:04:40] RECOVERY - DPKG on mw1132 is OK: All packages OK [12:04:41] RECOVERY - twemproxy process on mw1132 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:04:41] RECOVERY - Apache HTTP on mw1125 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.316 second response time [12:04:41] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:04:41] RECOVERY - SSH on mw1116 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:04:41] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:04:41] RECOVERY - Disk space on mw1119 is OK: DISK OK [12:04:42] RECOVERY - twemproxy process on mw1138 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:04:50] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:04:50] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:04:50] RECOVERY - twemproxy process on mw1116 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:04:50] PROBLEM - Disk space on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:04:50] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:10] PROBLEM - DPKG on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:05:10] PROBLEM - DPKG on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:05:25] RECOVERY - DPKG on mw1138 is OK: All packages OK [12:05:30] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:05:30] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:30] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:51] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:05:51] RECOVERY - DPKG on mw1146 is OK: All packages OK [12:05:52] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:05:52] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:52] PROBLEM - SSH on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:52] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:05:52] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:05:53] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:05:53] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:06:11] PROBLEM - Apache HTTP on mw1114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:06:11] RECOVERY - DPKG on mw1137 is OK: All packages OK [12:06:12] PROBLEM - Apache HTTP on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:06:21] RECOVERY - DPKG on mw1125 is OK: All packages OK [12:06:22] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: Connection timed out [12:06:24] RECOVERY - Disk space on mw1141 is OK: DISK OK [12:06:31] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:06:31] RECOVERY - Disk space on mw1143 is OK: DISK OK [12:06:31] PROBLEM - twemproxy process on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:06:41] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:06:41] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:06:41] RECOVERY - Apache HTTP on mw1143 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.056 second response time [12:06:41] RECOVERY - twemproxy process on mw1137 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:06:41] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:06:42] RECOVERY - Disk space on mw1121 is OK: DISK OK [12:06:42] RECOVERY - Apache HTTP on mw1148 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.459 second response time [12:06:51] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:07:01] RECOVERY - RAID on mw1143 is OK: OK: no RAID installed [12:07:01] RECOVERY - Disk space on mw1142 is OK: DISK OK [12:07:01] RECOVERY - DPKG on mw1148 is OK: All packages OK [12:07:11] RECOVERY - DPKG on mw1142 is OK: All packages OK [12:07:11] PROBLEM - twemproxy process on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:07:11] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [12:07:11] RECOVERY - Disk space on mw1139 is OK: DISK OK [12:07:21] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:21] PROBLEM - SSH on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:21] PROBLEM - SSH on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:21] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:07:21] RECOVERY - SSH on mw1128 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:07:22] RECOVERY - DPKG on mw1143 is OK: All packages OK [12:07:22] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 4.429 second response time [12:07:23] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:31] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:07:31] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:07:31] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:31] PROBLEM - twemproxy process on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:07:31] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:07:31] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:07:32] PROBLEM - SSH on mw1132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:51] PROBLEM - SSH on mw1140 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:51] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:51] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:07:51] PROBLEM - DPKG on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:07:51] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:07:51] PROBLEM - Disk space on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:08:11] PROBLEM - Disk space on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:08:11] RECOVERY - Disk space on mw1134 is OK: DISK OK [12:08:11] RECOVERY - RAID on mw1138 is OK: OK: no RAID installed [12:08:11] RECOVERY - Disk space on mw1120 is OK: DISK OK [12:08:21] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:08:21] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:08:21] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:08:21] RECOVERY - twemproxy process on mw1145 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:08:22] RECOVERY - twemproxy process on mw1131 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:08:22] RECOVERY - Disk space on mw1131 is OK: DISK OK [12:08:22] RECOVERY - RAID on mw1145 is OK: OK: no RAID installed [12:08:23] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:08:23] RECOVERY - DPKG on mw1145 is OK: All packages OK [12:08:24] RECOVERY - Disk space on mw1133 is OK: DISK OK [12:08:31] RECOVERY - twemproxy process on mw1117 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:08:31] RECOVERY - Disk space on mw1117 is OK: DISK OK [12:08:31] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:08:31] RECOVERY - Disk space on mw1116 is OK: DISK OK [12:08:31] PROBLEM - twemproxy process on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:08:32] PROBLEM - Disk space on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:08:32] PROBLEM - DPKG on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:08:33] RECOVERY - Disk space on mw1126 is OK: DISK OK [12:08:41] RECOVERY - SSH on mw1117 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:08:41] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:08:41] RECOVERY - twemproxy process on mw1133 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:08:41] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:08:41] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:08:51] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:01] RECOVERY - DPKG on mw1131 is OK: All packages OK [12:09:11] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:11] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:09:11] RECOVERY - DPKG on mw1141 is OK: All packages OK [12:09:11] RECOVERY - Disk space on mw1118 is OK: DISK OK [12:09:21] PROBLEM - DPKG on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:21] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:09:21] RECOVERY - Disk space on mw1127 is OK: DISK OK [12:09:21] RECOVERY - Apache HTTP on mw1134 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [12:09:21] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:22] RECOVERY - SSH on mw1115 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:09:31] PROBLEM - DPKG on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:31] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:31] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:09:31] RECOVERY - SSH on mw1134 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:09:32] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:32] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:41] PROBLEM - twemproxy process on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:41] RECOVERY - Disk space on mw1146 is OK: DISK OK [12:09:41] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:09:51] RECOVERY - Disk space on mw1115 is OK: DISK OK [12:09:51] RECOVERY - Disk space on mw1128 is OK: DISK OK [12:09:51] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:51] PROBLEM - twemproxy process on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:09:51] PROBLEM - Apache HTTP on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:01] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:10:01] RECOVERY - DPKG on mw1118 is OK: All packages OK [12:10:01] RECOVERY - DPKG on mw1134 is OK: All packages OK [12:10:01] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:10:01] RECOVERY - Disk space on mw1119 is OK: DISK OK [12:10:02] RECOVERY - twemproxy process on mw1134 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:10:11] PROBLEM - DPKG on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:10:11] RECOVERY - RAID on mw1142 is OK: OK: no RAID installed [12:10:21] PROBLEM - Apache HTTP on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:21] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:10:21] RECOVERY - Disk space on mw1137 is OK: DISK OK [12:10:22] RECOVERY - Apache HTTP on mw1138 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.060 second response time [12:10:22] RECOVERY - RAID on mw1118 is OK: OK: no RAID installed [12:10:22] RECOVERY - SSH on mw1119 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:10:31] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:34] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:10:54] PROBLEM - Apache HTTP on mw1143 is CRITICAL: Connection timed out [12:10:54] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:01] RECOVERY - DPKG on mw1148 is OK: All packages OK [12:11:01] RECOVERY - RAID on mw1134 is OK: OK: no RAID installed [12:11:11] PROBLEM - RAID on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:11] RECOVERY - Disk space on mw1114 is OK: DISK OK [12:11:21] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:11:21] RECOVERY - Disk space on mw1139 is OK: DISK OK [12:11:21] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:21] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:21] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:21] RECOVERY - DPKG on mw1138 is OK: All packages OK [12:11:22] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:11:31] PROBLEM - twemproxy process on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:31] PROBLEM - twemproxy process on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:31] PROBLEM - RAID on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:31] PROBLEM - Disk space on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:31] PROBLEM - DPKG on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:32] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:32] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:32] PROBLEM - DPKG on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:33] PROBLEM - Disk space on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:33] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:11:41] RECOVERY - Apache HTTP on mw1148 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.618 second response time [12:11:51] RECOVERY - Disk space on mw1121 is OK: DISK OK [12:11:51] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:12:11] PROBLEM - DPKG on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:12:11] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:12:11] RECOVERY - Disk space on mw1120 is OK: DISK OK [12:12:11] RECOVERY - SSH on mw1116 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:12:11] RECOVERY - RAID on mw1146 is OK: OK: no RAID installed [12:12:12] RECOVERY - Disk space on mw1125 is OK: DISK OK [12:12:12] PROBLEM - DPKG on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:12:13] RECOVERY - DPKG on mw1120 is OK: All packages OK [12:12:21] PROBLEM - SSH on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:21] PROBLEM - DPKG on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:12:21] RECOVERY - twemproxy process on mw1116 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:12:22] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:12:22] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:12:31] RECOVERY - Disk space on mw1143 is OK: DISK OK [12:12:31] RECOVERY - Disk space on mw1131 is OK: DISK OK [12:12:31] RECOVERY - twemproxy process on mw1126 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:12:31] PROBLEM - SSH on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:31] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:12:31] PROBLEM - SSH on mw1115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:32] RECOVERY - Disk space on mw1126 is OK: DISK OK [12:12:32] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:12:33] PROBLEM - Disk space on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:12:41] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:12:51] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:12:51] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:51] PROBLEM - Disk space on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:13:11] RECOVERY - SSH on mw1114 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:13:11] RECOVERY - SSH on mw1135 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:13:11] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:13:11] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:13:11] RECOVERY - DPKG on mw1119 is OK: All packages OK [12:13:21] PROBLEM - RAID on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:13:21] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:22] RECOVERY - twemproxy process on mw1131 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:13:31] PROBLEM - Apache HTTP on mw1138 is CRITICAL: Connection timed out [12:13:31] PROBLEM - Apache HTTP on mw1134 is CRITICAL: Connection timed out [12:13:31] PROBLEM - RAID on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:13:31] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:13:31] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:32] PROBLEM - Disk space on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:13:32] RECOVERY - RAID on mw1120 is OK: OK: no RAID installed [12:13:33] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:41] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:13:41] PROBLEM - SSH on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:51] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:01] RECOVERY - DPKG on mw1131 is OK: All packages OK [12:14:01] RECOVERY - Disk space on mw1142 is OK: DISK OK [12:14:11] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:11] PROBLEM - DPKG on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:11] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:14:11] PROBLEM - RAID on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:11] PROBLEM - RAID on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:11] PROBLEM - twemproxy process on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:21] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:21] PROBLEM - Disk space on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:22] PROBLEM - Disk space on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:22] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:22] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:14:22] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:14:22] RECOVERY - twemproxy process on mw1145 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:14:31] PROBLEM - Disk space on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:31] PROBLEM - twemproxy process on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:31] PROBLEM - twemproxy process on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:31] PROBLEM - DPKG on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:32] RECOVERY - Disk space on mw1116 is OK: DISK OK [12:14:32] PROBLEM - Disk space on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:41] RECOVERY - Disk space on mw1146 is OK: DISK OK [12:14:42] apergos: found anything yet? [12:14:51] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:14:51] PROBLEM - SSH on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:51] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:11] PROBLEM - DPKG on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:11] RECOVERY - Disk space on mw1114 is OK: DISK OK [12:15:11] RECOVERY - Disk space on mw1148 is OK: DISK OK [12:15:11] PROBLEM - RAID on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:14] we're looking at it a few of us [12:15:21] still slugging along, sorry for the delay [12:15:21] PROBLEM - twemproxy process on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:21] PROBLEM - SSH on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:22] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:22] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 0.072 second response time [12:15:23] RECOVERY - RAID on mw1145 is OK: OK: no RAID installed [12:15:23] RECOVERY - Apache HTTP on mw1145 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.062 second response time [12:15:23] RECOVERY - DPKG on mw1145 is OK: All packages OK [12:15:23] RECOVERY - RAID on mw1114 is OK: OK: no RAID installed [12:15:30] apergos: no problem [12:15:31] RECOVERY - SSH on mw1128 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:15:31] RECOVERY - DPKG on mw1143 is OK: All packages OK [12:15:31] RECOVERY - twemproxy process on mw1114 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:15:31] PROBLEM - twemproxy process on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:31] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:32] PROBLEM - twemproxy process on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:32] RECOVERY - RAID on mw1148 is OK: OK: no RAID installed [12:15:33] RECOVERY - DPKG on mw1146 is OK: All packages OK [12:15:33] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:41] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:15:51] PROBLEM - Apache HTTP on mw1148 is CRITICAL: Connection timed out [12:15:51] RECOVERY - Disk space on mw1121 is OK: DISK OK [12:15:51] RECOVERY - Disk space on mw1115 is OK: DISK OK [12:15:51] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:51] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:51] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:51] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:15:52] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:16:01] RECOVERY - RAID on mw1143 is OK: OK: no RAID installed [12:16:01] RECOVERY - Apache HTTP on mw1114 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.061 second response time [12:16:01] RECOVERY - DPKG on mw1148 is OK: All packages OK [12:16:01] RECOVERY - DPKG on mw1118 is OK: All packages OK [12:16:01] RECOVERY - DPKG on mw1114 is OK: All packages OK [12:16:11] RECOVERY - DPKG on mw1128 is OK: All packages OK [12:16:11] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 6.722 second response time [12:16:22] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:16:22] RECOVERY - Disk space on mw1125 is OK: DISK OK [12:16:22] PROBLEM - SSH on mw1135 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:22] PROBLEM - DPKG on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:16:22] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:16:22] RECOVERY - twemproxy process on mw1138 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:16:22] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:16:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [12:16:22] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:16:23] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:16:23] RECOVERY - Disk space on mw1138 is OK: DISK OK [12:16:31] RECOVERY - DPKG on mw1138 is OK: All packages OK [12:16:31] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:16:31] PROBLEM - twemproxy process on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:16:31] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:31] PROBLEM - Disk space on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:16:32] RECOVERY - Disk space on mw1123 is OK: DISK OK [12:16:41] PROBLEM - RAID on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:16:41] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:16:41] RECOVERY - DPKG on mw1121 is OK: All packages OK [12:16:41] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:16:41] RECOVERY - Apache HTTP on mw1118 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.921 second response time [12:16:42] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:16:51] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:16:51] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:16:51] RECOVERY - twemproxy process on mw1139 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:16:51] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:16:51] PROBLEM - Disk space on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:16:52] PROBLEM - twemproxy process on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:17:01] RECOVERY - RAID on mw1121 is OK: OK: no RAID installed [12:17:01] RECOVERY - RAID on mw1138 is OK: OK: no RAID installed [12:17:01] RECOVERY - Apache HTTP on mw1126 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.061 second response time [12:17:01] RECOVERY - twemproxy process on mw1132 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:17:11] PROBLEM - DPKG on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:17:21] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:21] PROBLEM - DPKG on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:17:21] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:17:21] RECOVERY - twemproxy process on mw1126 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:17:21] RECOVERY - RAID on mw1126 is OK: OK: no RAID installed [12:17:22] RECOVERY - DPKG on mw1126 is OK: All packages OK [12:17:22] RECOVERY - Disk space on mw1141 is OK: DISK OK [12:17:31] RECOVERY - SSH on mw1132 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:17:31] RECOVERY - RAID on mw1141 is OK: OK: no RAID installed [12:17:32] PROBLEM - Disk space on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:17:41] RECOVERY - Disk space on mw1128 is OK: DISK OK [12:18:11] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:11] RECOVERY - SSH on mw1116 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:18:21] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:18:22] RECOVERY - Disk space on mw1144 is OK: DISK OK [12:18:31] PROBLEM - RAID on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:18:41] PROBLEM - RAID on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:18:41] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:18:41] PROBLEM - twemproxy process on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:18:41] RECOVERY - Disk space on mw1136 is OK: DISK OK [12:18:51] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:18:51] PROBLEM - Disk space on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:01] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:19:01] PROBLEM - Apache HTTP on mw1114 is CRITICAL: Connection timed out [12:19:11] PROBLEM - RAID on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:11] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:11] PROBLEM - DPKG on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:11] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:19:11] RECOVERY - twemproxy process on mw1146 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:19:11] RECOVERY - Disk space on mw1120 is OK: DISK OK [12:19:12] PROBLEM - DPKG on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:21] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:21] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:21] PROBLEM - DPKG on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:21] RECOVERY - Apache HTTP on mw1138 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.083 second response time [12:19:21] RECOVERY - Disk space on mw1127 is OK: DISK OK [12:19:22] RECOVERY - SSH on mw1133 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:19:22] RECOVERY - Disk space on mw1130 is OK: DISK OK [12:19:22] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:19:23] RECOVERY - Disk space on mw1122 is OK: DISK OK [12:19:31] RECOVERY - Disk space on mw1133 is OK: DISK OK [12:19:31] RECOVERY - Disk space on mw1117 is OK: DISK OK [12:19:31] PROBLEM - DPKG on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:31] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:31] PROBLEM - SSH on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:31] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:31] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:32] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:32] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:41] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:41] RECOVERY - SSH on mw1117 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:19:41] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:19:41] RECOVERY - twemproxy process on mw1133 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:19:51] PROBLEM - Apache HTTP on mw1118 is CRITICAL: Connection timed out [12:19:51] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:51] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:51] PROBLEM - DPKG on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:51] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:51] PROBLEM - twemproxy process on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:19:52] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:11] PROBLEM - twemproxy process on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:11] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:11] PROBLEM - RAID on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:11] PROBLEM - Apache HTTP on mw1126 is CRITICAL: Connection timed out [12:20:11] RECOVERY - Disk space on mw1118 is OK: DISK OK [12:20:21] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:21] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:22] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:22] RECOVERY - twemproxy process on mw1117 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:20:22] RECOVERY - SSH on mw1122 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:20:31] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: Connection timed out [12:20:34] PROBLEM - RAID on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:34] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:34] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:34] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:34] PROBLEM - twemproxy process on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:35] PROBLEM - DPKG on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:35] PROBLEM - twemproxy process on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:36] RECOVERY - Disk space on mw1143 is OK: DISK OK [12:20:36] PROBLEM - SSH on mw1132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:37] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:41] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:20:41] PROBLEM - RAID on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:41] RECOVERY - Disk space on mw1115 is OK: DISK OK [12:20:51] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:20:51] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:20:51] RECOVERY - twemproxy process on mw1129 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:20:51] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:20:51] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:51] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:21:01] RECOVERY - DPKG on mw1118 is OK: All packages OK [12:21:01] RECOVERY - DPKG on mw1117 is OK: All packages OK [12:21:11] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:21:11] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:21:11] RECOVERY - RAID on mw1117 is OK: OK: no RAID installed [12:21:11] RECOVERY - Disk space on mw1134 is OK: DISK OK [12:21:11] RECOVERY - Disk space on mw1139 is OK: DISK OK [12:21:21] PROBLEM - SSH on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:21] RECOVERY - SSH on mw1136 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:21:22] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:21:22] RECOVERY - Disk space on mw1137 is OK: DISK OK [12:21:22] RECOVERY - Disk space on mw1131 is OK: DISK OK [12:21:32] RECOVERY - twemproxy process on mw1131 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:21:32] RECOVERY - SSH on mw1134 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:21:32] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:21:32] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:32] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:21:32] RECOVERY - Apache HTTP on mw1127 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.924 second response time [12:21:41] RECOVERY - DPKG on mw1127 is OK: All packages OK [12:21:52] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [12:21:52] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:21:52] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:21:52] PROBLEM - Disk space on mw1136 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:21:52] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:52] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:22:01] RECOVERY - RAID on mw1127 is OK: OK: no RAID installed [12:22:11] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:11] RECOVERY - Disk space on mw1125 is OK: DISK OK [12:22:21] PROBLEM - twemproxy process on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:22:21] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:22] RECOVERY - Disk space on mw1141 is OK: DISK OK [12:22:22] RECOVERY - DPKG on mw1143 is OK: All packages OK [12:22:22] RECOVERY - Disk space on mw1144 is OK: DISK OK [12:22:22] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:22:31] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:22:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:31] PROBLEM - Disk space on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:22:31] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:22:31] RECOVERY - Disk space on mw1116 is OK: DISK OK [12:22:31] PROBLEM - SSH on mw1133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:41] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:22:51] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:51] PROBLEM - twemproxy process on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:23:11] RECOVERY - SSH on mw1116 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:23:11] RECOVERY - DPKG on mw1119 is OK: All packages OK [12:23:11] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.094 second response time [12:23:21] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 0.085 second response time [12:23:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [12:23:23] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:23:51] RECOVERY - Disk space on mw1121 is OK: DISK OK [12:23:51] RECOVERY - Disk space on mw1146 is OK: DISK OK [12:23:51] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:23:51] PROBLEM - Disk space on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:23:51] PROBLEM - twemproxy process on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:24:01] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:24:01] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:24:01] RECOVERY - RAID on mw1119 is OK: OK: no RAID installed [12:24:11] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:24:11] RECOVERY - twemproxy process on mw1146 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:24:21] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:21] PROBLEM - Disk space on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:24:21] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:24:22] PROBLEM - RAID on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:24:22] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:24:22] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:24:22] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:24:23] RECOVERY - SSH on mw1128 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:24:31] PROBLEM - twemproxy process on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:24:31] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:24:31] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:31] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:24:31] PROBLEM - SSH on mw1136 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:32] RECOVERY - DPKG on mw1146 is OK: All packages OK [12:24:41] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:24:41] RECOVERY - DPKG on mw1121 is OK: All packages OK [12:24:51] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:24:51] RECOVERY - Disk space on mw1135 is OK: DISK OK [12:25:01] RECOVERY - DPKG on mw1144 is OK: All packages OK [12:25:01] RECOVERY - RAID on mw1134 is OK: OK: no RAID installed [12:25:01] RECOVERY - twemproxy process on mw1134 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:25:11] RECOVERY - DPKG on mw1120 is OK: All packages OK [12:25:11] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:25:11] RECOVERY - DPKG on mw1130 is OK: All packages OK [12:25:21] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:25:21] PROBLEM - Disk space on mw1114 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:25:21] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:22] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:25:22] RECOVERY - RAID on mw1144 is OK: OK: no RAID installed [12:25:22] RECOVERY - SSH on mw1129 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:25:22] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:25:23] RECOVERY - Disk space on mw1130 is OK: DISK OK [12:25:23] RECOVERY - twemproxy process on mw1131 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:25:31] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:25:31] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:32] PROBLEM - DPKG on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:25:32] PROBLEM - Disk space on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:25:32] RECOVERY - Disk space on mw1123 is OK: DISK OK [12:25:32] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:32] PROBLEM - SSH on mw1122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:33] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:25:33] PROBLEM - Disk space on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:25:41] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:41] RECOVERY - Apache HTTP on mw1118 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.473 second response time [12:25:51] RECOVERY - twemproxy process on mw1139 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:25:51] RECOVERY - twemproxy process on mw1129 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:26:01] RECOVERY - DPKG on mw1131 is OK: All packages OK [12:26:11] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:26:11] RECOVERY - Disk space on mw1129 is OK: DISK OK [12:26:11] PROBLEM - DPKG on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:26:21] PROBLEM - SSH on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:21] RECOVERY - Disk space on mw1133 is OK: DISK OK [12:26:22] RECOVERY - SSH on mw1136 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:26:22] RECOVERY - RAID on mw1131 is OK: OK: no RAID installed [12:26:22] RECOVERY - Disk space on mw1126 is OK: DISK OK [12:26:22] RECOVERY - Disk space on mw1141 is OK: DISK OK [12:26:31] RECOVERY - twemproxy process on mw1126 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:26:31] PROBLEM - twemproxy process on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:26:31] RECOVERY - SSH on mw1133 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:26:31] PROBLEM - Disk space on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:26:51] PROBLEM - SSH on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:51] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:11] PROBLEM - RAID on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:21] PROBLEM - SSH on mw1114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:21] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:21] PROBLEM - DPKG on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:31] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:27:31] PROBLEM - twemproxy process on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:31] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:31] RECOVERY - twemproxy process on mw1123 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:27:32] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:27:41] PROBLEM - SSH on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:41] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:41] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:51] RECOVERY - twemproxy process on mw1147 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:27:51] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:27:51] PROBLEM - DPKG on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:51] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:51] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:51] PROBLEM - Disk space on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:27:51] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:28:11] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:28:11] RECOVERY - SSH on mw1116 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:28:11] PROBLEM - RAID on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:28:21] RECOVERY - Disk space on mw1125 is OK: DISK OK [12:28:21] RECOVERY - Disk space on mw1148 is OK: DISK OK [12:28:21] PROBLEM - DPKG on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:28:21] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:28:22] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:22] PROBLEM - DPKG on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:28:22] RECOVERY - twemproxy process on mw1115 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:28:31] RECOVERY - Disk space on mw1116 is OK: DISK OK [12:28:31] RECOVERY - twemproxy process on mw1116 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:28:31] RECOVERY - SSH on mw1134 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:28:31] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:28:32] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:28:32] PROBLEM - SSH on mw1129 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:32] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:28:41] RECOVERY - DPKG on mw1116 is OK: All packages OK [12:28:41] RECOVERY - SSH on mw1117 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:28:41] RECOVERY - DPKG on mw1121 is OK: All packages OK [12:28:41] RECOVERY - Disk space on mw1146 is OK: DISK OK [12:28:51] PROBLEM - Apache HTTP on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:51] PROBLEM - twemproxy process on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:28:51] PROBLEM - DPKG on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:28:51] PROBLEM - twemproxy process on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:01] RECOVERY - DPKG on mw1117 is OK: All packages OK [12:29:11] PROBLEM - DPKG on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:11] PROBLEM - RAID on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:11] RECOVERY - Apache HTTP on mw1125 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [12:29:11] RECOVERY - Disk space on mw1114 is OK: DISK OK [12:29:11] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:29:12] PROBLEM - DPKG on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:21] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:21] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:21] PROBLEM - SSH on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:22] PROBLEM - Disk space on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:22] RECOVERY - twemproxy process on mw1135 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:29:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [12:29:22] PROBLEM - Apache HTTP on mw1144 is CRITICAL: Connection timed out [12:29:23] RECOVERY - Disk space on mw1147 is OK: DISK OK [12:29:31] RECOVERY - SSH on mw1147 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:29:31] PROBLEM - RAID on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:31] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:31] PROBLEM - SSH on mw1136 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:31] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:31] PROBLEM - RAID on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:32] PROBLEM - Disk space on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:32] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:33] PROBLEM - twemproxy process on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:33] PROBLEM - Disk space on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:34] PROBLEM - twemproxy process on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:34] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:35] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:29:41] PROBLEM - Apache HTTP on mw1127 is CRITICAL: Connection timed out [12:29:41] RECOVERY - twemproxy process on mw1114 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:29:41] RECOVERY - Disk space on mw1135 is OK: DISK OK [12:29:51] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:51] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:29:51] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:30:01] RECOVERY - Disk space on mw1142 is OK: DISK OK [12:30:11] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:21] RECOVERY - SSH on mw1132 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:30:22] RECOVERY - SSH on mw1115 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:30:31] RECOVERY - Disk space on mw1143 is OK: DISK OK [12:30:31] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:30:31] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:31] PROBLEM - SSH on mw1128 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:31] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:30:32] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:30:41] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:41] PROBLEM - twemproxy process on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:30:41] RECOVERY - twemproxy process on mw1137 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:30:51] RECOVERY - twemproxy process on mw1130 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:30:51] PROBLEM - twemproxy process on mw1147 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:30:51] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:01] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:01] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:31:11] RECOVERY - RAID on mw1146 is OK: OK: no RAID installed [12:31:11] PROBLEM - twemproxy process on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:21] PROBLEM - SSH on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:21] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:22] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:22] RECOVERY - Apache HTTP on mw1146 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.061 second response time [12:31:22] RECOVERY - Disk space on mw1141 is OK: DISK OK [12:31:22] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:31:22] RECOVERY - Disk space on mw1127 is OK: DISK OK [12:31:22] RECOVERY - Disk space on mw1140 is OK: DISK OK [12:31:23] RECOVERY - Disk space on mw1144 is OK: DISK OK [12:31:31] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:31:31] PROBLEM - twemproxy process on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:31] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:31:31] RECOVERY - Disk space on mw1123 is OK: DISK OK [12:31:31] RECOVERY - DPKG on mw1146 is OK: All packages OK [12:31:31] RECOVERY - Disk space on mw1117 is OK: DISK OK [12:31:32] PROBLEM - Disk space on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:41] PROBLEM - SSH on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:31:41] RECOVERY - SSH on mw1140 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:31:41] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:31:41] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:31:51] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:51] PROBLEM - DPKG on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:51] PROBLEM - DPKG on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:51] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:31:51] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:32:01] RECOVERY - DPKG on mw1148 is OK: All packages OK [12:32:01] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.676 second response time [12:32:11] RECOVERY - DPKG on mw1141 is OK: All packages OK [12:32:11] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.859 second response time [12:32:11] RECOVERY - Disk space on mw1125 is OK: DISK OK [12:32:11] PROBLEM - DPKG on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:32:11] PROBLEM - Disk space on mw1114 is CRITICAL: Timeout while attempting connection [12:32:21] PROBLEM - Apache HTTP on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:21] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:22] RECOVERY - twemproxy process on mw1117 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:32:31] RECOVERY - Disk space on mw1137 is OK: DISK OK [12:32:31] PROBLEM - SSH on mw1147 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:31] PROBLEM - Disk space on mw1147 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:32:31] PROBLEM - twemproxy process on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:32:31] RECOVERY - Disk space on mw1126 is OK: DISK OK [12:32:32] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:32:44] PROBLEM - twemproxy process on mw1114 is CRITICAL: Timeout while attempting connection [12:32:44] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:33:01] RECOVERY - DPKG on mw1142 is OK: All packages OK [12:33:11] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:33:11] RECOVERY - RAID on mw1142 is OK: OK: no RAID installed [12:33:11] RECOVERY - Disk space on mw1139 is OK: DISK OK [12:33:11] RECOVERY - Apache HTTP on mw1125 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.903 second response time [12:33:11] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:33:21] RECOVERY - SSH on mw1147 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:33:21] RECOVERY - twemproxy process on mw1126 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:33:22] RECOVERY - DPKG on mw1126 is OK: All packages OK [12:33:22] RECOVERY - DPKG on mw1125 is OK: All packages OK [12:33:22] RECOVERY - Disk space on mw1131 is OK: DISK OK [12:33:31] PROBLEM - SSH on mw1115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:31] PROBLEM - Disk space on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:33:32] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:33:32] PROBLEM - SSH on mw1132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:32] PROBLEM - SSH on mw1133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:41] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:33:51] RECOVERY - RAID on mw1125 is OK: OK: no RAID installed [12:33:51] PROBLEM - twemproxy process on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:34:01] PROBLEM - Host mw1114 is DOWN: PING CRITICAL - Packet loss = 100% [12:34:11] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:34:11] RECOVERY - Disk space on mw1120 is OK: DISK OK [12:34:21] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:34:22] RECOVERY - twemproxy process on mw1131 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:34:22] RECOVERY - RAID on mw1131 is OK: OK: no RAID installed [12:34:31] RECOVERY - twemproxy process on mw1114 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:34:31] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:34:31] PROBLEM - Disk space on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:34:31] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:34:31] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:34:31] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:34:32] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:34:32] RECOVERY - SSH on mw1132 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:34:33] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:34:41] RECOVERY - Host mw1114 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [12:34:41] mark: currently dealing with users going all four loco over gadgets vanishing generally and the VE pref vanishing specifically. Can you ping me when it gets resolved so I can report as such to the various noticeboards? [12:34:51] RECOVERY - Disk space on mw1132 is OK: DISK OK [12:34:51] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:34:51] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:34:51] PROBLEM - SSH on mw1140 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:35:01] PROBLEM - Apache HTTP on mw1141 is CRITICAL: Connection timed out [12:35:01] RECOVERY - twemproxy process on mw1132 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:35:02] RECOVERY - DPKG on mw1114 is OK: All packages OK [12:35:11] RECOVERY - SSH on mw1114 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:35:11] RECOVERY - SSH on mw1135 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:35:11] RECOVERY - Disk space on mw1114 is OK: DISK OK [12:35:21] PROBLEM - DPKG on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:21] RECOVERY - RAID on mw1114 is OK: OK: no RAID installed [12:35:21] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:35:22] RECOVERY - Disk space on mw1123 is OK: DISK OK [12:35:31] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:35:31] RECOVERY - twemproxy process on mw1123 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:35:31] PROBLEM - twemproxy process on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:31] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:31] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:32] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:32] PROBLEM - Disk space on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:33] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:41] RECOVERY - RAID on mw1148 is OK: OK: no RAID installed [12:35:41] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:35:51] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:35:51] PROBLEM - twemproxy process on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:35:51] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:11] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:36:11] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:36:11] RECOVERY - RAID on mw1115 is OK: OK: no RAID installed [12:36:11] RECOVERY - Disk space on mw1134 is OK: DISK OK [12:36:11] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:36:11] RECOVERY - twemproxy process on mw1134 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:36:12] PROBLEM - DPKG on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:36:21] PROBLEM - Apache HTTP on mw1142 is CRITICAL: Connection timed out [12:36:21] PROBLEM - RAID on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:36:21] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:21] RECOVERY - SSH on mw1115 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:36:22] RECOVERY - Disk space on mw1127 is OK: DISK OK [12:36:22] RECOVERY - Disk space on mw1141 is OK: DISK OK [12:36:31] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:36:31] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:36:31] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:36:31] RECOVERY - SSH on mw1134 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:36:31] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:31] PROBLEM - twemproxy process on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:36:31] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:36:32] PROBLEM - DPKG on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:36:32] PROBLEM - SSH on mw1147 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:36:33] RECOVERY - Disk space on mw1143 is OK: DISK OK [12:36:33] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:36:34] RECOVERY - Apache HTTP on mw1127 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 6.666 second response time [12:36:41] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:36:41] RECOVERY - DPKG on mw1127 is OK: All packages OK [12:36:41] RECOVERY - Disk space on mw1121 is OK: DISK OK [12:36:51] RECOVERY - twemproxy process on mw1133 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:36:51] PROBLEM - SSH on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:01] RECOVERY - DPKG on mw1134 is OK: All packages OK [12:37:01] RECOVERY - RAID on mw1127 is OK: OK: no RAID installed [12:37:01] RECOVERY - RAID on mw1134 is OK: OK: no RAID installed [12:37:21] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:21] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:37:21] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:22] RECOVERY - Disk space on mw1140 is OK: DISK OK [12:37:22] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:37:22] RECOVERY - SSH on mw1128 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:37:27] !log Restarted pybal on lvs1006 (set api depool threshold to .4 from .6) [12:37:29] holy hell that's a lot of icinga spam [12:37:31] PROBLEM - twemproxy process on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:37:31] PROBLEM - twemproxy process on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:37:31] PROBLEM - RAID on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:37:31] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:37:31] PROBLEM - Apache HTTP on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:32] PROBLEM - DPKG on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:37:32] PROBLEM - SSH on mw1132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:37] Logged the message, Mr. Obvious [12:37:41] RECOVERY - SSH on mw1140 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:37:41] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:37:51] RECOVERY - twemproxy process on mw1140 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:37:51] RECOVERY - Apache HTTP on mw1148 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 9.066 second response time [12:37:51] PROBLEM - Disk space on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:37:51] PROBLEM - Disk space on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:38:01] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:38:13] PROBLEM - twemproxy process on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:38:13] PROBLEM - RAID on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:38:21] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:21] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:38:22] PROBLEM - Apache HTTP on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:22] PROBLEM - SSH on mw1135 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:22] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:38:22] RECOVERY - Apache HTTP on mw1117 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.777 second response time [12:38:22] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:38:23] RECOVERY - twemproxy process on mw1117 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:38:23] RECOVERY - Disk space on mw1137 is OK: DISK OK [12:38:31] RECOVERY - Disk space on mw1130 is OK: DISK OK [12:38:31] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:38:31] PROBLEM - DPKG on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:38:31] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:31] RECOVERY - Disk space on mw1117 is OK: DISK OK [12:38:41] RECOVERY - SSH on mw1117 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:38:44] !log Restarted pybal on lvs1003 [12:38:51] RECOVERY - Disk space on mw1135 is OK: DISK OK [12:38:51] PROBLEM - RAID on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:38:51] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:51] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:54] Logged the message, Mr. Obvious [12:39:01] RECOVERY - DPKG on mw1131 is OK: All packages OK [12:39:01] RECOVERY - DPKG on mw1117 is OK: All packages OK [12:39:11] RECOVERY - DPKG on mw1120 is OK: All packages OK [12:39:11] RECOVERY - RAID on mw1117 is OK: OK: no RAID installed [12:39:11] RECOVERY - Disk space on mw1118 is OK: DISK OK [12:39:11] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:39:21] RECOVERY - twemproxy process on mw1131 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:39:31] RECOVERY - Disk space on mw1126 is OK: DISK OK [12:39:31] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:39:31] PROBLEM - twemproxy process on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:39:31] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:39:41] RECOVERY - Apache HTTP on mw1118 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.909 second response time [12:39:41] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:39:51] RECOVERY - DPKG on mw1121 is OK: All packages OK [12:39:51] PROBLEM - twemproxy process on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:39:51] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:40:01] RECOVERY - DPKG on mw1118 is OK: All packages OK [12:40:11] PROBLEM - DPKG on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:40:11] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:40:11] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:40:11] PROBLEM - RAID on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:40:11] PROBLEM - twemproxy process on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:40:21] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:40:21] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:22] RECOVERY - Apache HTTP on mw1121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.062 second response time [12:40:22] RECOVERY - twemproxy process on mw1126 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:40:22] RECOVERY - DPKG on mw1126 is OK: All packages OK [12:40:22] RECOVERY - Disk space on mw1144 is OK: DISK OK [12:40:22] RECOVERY - Apache HTTP on mw1120 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.382 second response time [12:40:23] RECOVERY - RAID on mw1118 is OK: OK: no RAID installed [12:40:31] RECOVERY - Disk space on mw1141 is OK: DISK OK [12:40:31] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:40:31] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:40:31] RECOVERY - Disk space on mw1133 is OK: DISK OK [12:40:31] RECOVERY - Disk space on mw1143 is OK: DISK OK [12:40:32] RECOVERY - SSH on mw1133 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:40:41] PROBLEM - SSH on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:41] PROBLEM - twemproxy process on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:40:41] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:40:41] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:40:51] RECOVERY - twemproxy process on mw1147 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:40:51] RECOVERY - Apache HTTP on mw1131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 8.204 second response time [12:40:51] PROBLEM - SSH on mw1140 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:51] PROBLEM - Disk space on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:41:01] RECOVERY - DPKG on mw1144 is OK: All packages OK [12:41:01] RECOVERY - Apache HTTP on mw1126 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.056 second response time [12:41:01] RECOVERY - Apache HTTP on mw1123 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.997 second response time [12:41:21] RECOVERY - DPKG on mw1128 is OK: All packages OK [12:41:21] RECOVERY - Apache HTTP on mw1138 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.040 second response time [12:41:21] RECOVERY - DPKG on mw1138 is OK: All packages OK [12:41:22] RECOVERY - twemproxy process on mw1138 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:41:22] RECOVERY - RAID on mw1144 is OK: OK: no RAID installed [12:41:31] RECOVERY - Apache HTTP on mw1134 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 8.619 second response time [12:41:31] RECOVERY - twemproxy process on mw1123 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:41:31] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:41:31] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:41:41] RECOVERY - Apache HTTP on mw1119 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time [12:41:41] RECOVERY - Disk space on mw1128 is OK: DISK OK [12:41:41] RECOVERY - SSH on mw1140 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:41:51] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:41:51] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:51] PROBLEM - Disk space on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:41:51] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:42:01] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.061 second response time [12:42:01] RECOVERY - twemproxy process on mw1136 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:42:01] RECOVERY - RAID on mw1138 is OK: OK: no RAID installed [12:42:11] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:42:11] RECOVERY - DPKG on mw1123 is OK: All packages OK [12:42:11] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.559 second response time [12:42:21] RECOVERY - SSH on mw1136 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:42:22] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:42:22] RECOVERY - RAID on mw1131 is OK: OK: no RAID installed [12:42:22] RECOVERY - RAID on mw1123 is OK: OK: no RAID installed [12:42:31] RECOVERY - RAID on mw1141 is OK: OK: no RAID installed [12:42:31] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:42:31] RECOVERY - RAID on mw1120 is OK: OK: no RAID installed [12:42:41] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:43:01] RECOVERY - RAID on mw1121 is OK: OK: no RAID installed [12:43:01] RECOVERY - RAID on mw1116 is OK: OK: no RAID installed [12:43:11] RECOVERY - SSH on mw1116 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:43:11] RECOVERY - DPKG on mw1141 is OK: All packages OK [12:43:11] RECOVERY - Disk space on mw1134 is OK: DISK OK [12:43:21] RECOVERY - SSH on mw1147 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:43:21] RECOVERY - twemproxy process on mw1116 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:43:22] RECOVERY - Disk space on mw1137 is OK: DISK OK [12:43:22] RECOVERY - Disk space on mw1116 is OK: DISK OK [12:43:31] RECOVERY - DPKG on mw1143 is OK: All packages OK [12:43:31] PROBLEM - Disk space on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:43:31] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:43:31] RECOVERY - SSH on mw1134 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:43:31] PROBLEM - SSH on mw1133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:43:42] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:43:42] RECOVERY - DPKG on mw1116 is OK: All packages OK [12:43:51] RECOVERY - twemproxy process on mw1129 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:43:51] PROBLEM - twemproxy process on mw1147 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:44:01] RECOVERY - DPKG on mw1134 is OK: All packages OK [12:44:01] RECOVERY - RAID on mw1134 is OK: OK: no RAID installed [12:44:01] RECOVERY - twemproxy process on mw1132 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:44:01] RECOVERY - RAID on mw1119 is OK: OK: no RAID installed [12:44:11] RECOVERY - twemproxy process on mw1134 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:44:11] RECOVERY - SSH on mw1135 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:44:11] RECOVERY - DPKG on mw1119 is OK: All packages OK [12:44:11] RECOVERY - Disk space on mw1125 is OK: DISK OK [12:44:21] RECOVERY - DPKG on mw1125 is OK: All packages OK [12:44:22] RECOVERY - RAID on mw1126 is OK: OK: no RAID installed [12:44:22] RECOVERY - Disk space on mw1130 is OK: DISK OK [12:44:31] RECOVERY - twemproxy process on mw1135 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:44:31] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:44:41] RECOVERY - Disk space on mw1136 is OK: DISK OK [12:44:51] PROBLEM - SSH on mw1140 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:51] PROBLEM - twemproxy process on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:44:51] PROBLEM - Disk space on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:44:51] PROBLEM - Host mw1117 is DOWN: PING CRITICAL - Packet loss = 100% [12:45:11] PROBLEM - twemproxy process on mw1136 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:45:11] PROBLEM - DPKG on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:45:21] RECOVERY - Host mw1117 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms [12:45:31] PROBLEM - RAID on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:45:51] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:45:51] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:01] PROBLEM - Host mw1132 is DOWN: PING CRITICAL - Packet loss = 100% [12:46:21] PROBLEM - DPKG on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:21] PROBLEM - DPKG on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:22] RECOVERY - Disk space on mw1147 is OK: DISK OK [12:46:22] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:22] RECOVERY - SSH on mw1132 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:46:31] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:46:31] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:31] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:31] PROBLEM - RAID on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:31] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:31] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:31] PROBLEM - RAID on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:32] PROBLEM - DPKG on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:32] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:33] RECOVERY - Host mw1132 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [12:46:33] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:34] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:41] PROBLEM - twemproxy process on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:41] PROBLEM - RAID on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:46:41] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:46:41] RECOVERY - Disk space on mw1132 is OK: DISK OK [12:46:41] RECOVERY - DPKG on mw1132 is OK: All packages OK [12:46:41] RECOVERY - Apache HTTP on mw1143 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.179 second response time [12:46:41] RECOVERY - Disk space on mw1135 is OK: DISK OK [12:46:42] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:46:51] PROBLEM - twemproxy process on mw1129 is CRITICAL: Timeout while attempting connection [12:46:51] RECOVERY - twemproxy process on mw1130 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:47:11] PROBLEM - RAID on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:47:11] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:47:11] RECOVERY - RAID on mw1132 is OK: OK: no RAID installed [12:47:11] PROBLEM - Apache HTTP on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:21] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:21] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:47:21] PROBLEM - DPKG on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:47:21] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.465 second response time [12:47:21] RECOVERY - DPKG on mw1135 is OK: All packages OK [12:47:22] RECOVERY - Disk space on mw1144 is OK: DISK OK [12:47:22] RECOVERY - DPKG on mw1143 is OK: All packages OK [12:47:23] RECOVERY - Disk space on mw1133 is OK: DISK OK [12:47:23] RECOVERY - SSH on mw1133 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:47:24] RECOVERY - SSH on mw1129 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:47:31] PROBLEM - twemproxy process on mw1121 is CRITICAL: Timeout while attempting connection [12:47:31] PROBLEM - Apache HTTP on mw1121 is CRITICAL: Connection timed out [12:47:31] RECOVERY - Disk space on mw1123 is OK: DISK OK [12:47:31] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:47:31] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:47:32] RECOVERY - RAID on mw1129 is OK: OK: no RAID installed [12:47:32] PROBLEM - SSH on mw1136 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:33] PROBLEM - DPKG on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:47:33] RECOVERY - twemproxy process on mw1123 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:47:41] PROBLEM - RAID on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:47:41] RECOVERY - twemproxy process on mw1129 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:47:41] RECOVERY - Disk space on mw1128 is OK: DISK OK [12:47:51] PROBLEM - DPKG on mw1121 is CRITICAL: Timeout while attempting connection [12:47:51] PROBLEM - Disk space on mw1121 is CRITICAL: Timeout while attempting connection [12:47:51] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:47:51] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:51] PROBLEM - Disk space on mw1136 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:48:01] RECOVERY - Apache HTTP on mw1130 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [12:48:01] RECOVERY - RAID on mw1143 is OK: OK: no RAID installed [12:48:01] RECOVERY - Apache HTTP on mw1123 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [12:48:11] PROBLEM - DPKG on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:48:11] RECOVERY - DPKG on mw1144 is OK: All packages OK [12:48:11] RECOVERY - Disk space on mw1129 is OK: DISK OK [12:48:11] RECOVERY - DPKG on mw1130 is OK: All packages OK [12:48:11] PROBLEM - RAID on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:48:11] PROBLEM - twemproxy process on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:48:21] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:48:21] PROBLEM - DPKG on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:48:21] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:48:21] RECOVERY - DPKG on mw1129 is OK: All packages OK [12:48:21] RECOVERY - RAID on mw1123 is OK: OK: no RAID installed [12:48:21] RECOVERY - RAID on mw1131 is OK: OK: no RAID installed [12:48:21] RECOVERY - RAID on mw1130 is OK: OK: no RAID installed [12:48:22] RECOVERY - Disk space on mw1140 is OK: DISK OK [12:48:22] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:48:31] RECOVERY - Disk space on mw1122 is OK: DISK OK [12:48:31] PROBLEM - Apache HTTP on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:41] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:49:11] RECOVERY - DPKG on mw1123 is OK: All packages OK [12:49:11] RECOVERY - DPKG on mw1141 is OK: All packages OK [12:49:11] RECOVERY - Disk space on mw1120 is OK: DISK OK [12:49:21] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:49:21] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:49:22] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:49:22] PROBLEM - Host mw1121 is DOWN: PING CRITICAL - Packet loss = 100% [12:49:22] RECOVERY - RAID on mw1144 is OK: OK: no RAID installed [12:49:22] RECOVERY - Apache HTTP on mw1134 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.107 second response time [12:49:31] RECOVERY - RAID on mw1120 is OK: OK: no RAID installed [12:49:31] RECOVERY - RAID on mw1141 is OK: OK: no RAID installed [12:49:31] PROBLEM - SSH on mw1147 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:31] PROBLEM - Disk space on mw1147 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:49:31] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:49:31] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:32] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:49:32] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:49:41] RECOVERY - Disk space on mw1121 is OK: DISK OK [12:49:41] RECOVERY - DPKG on mw1121 is OK: All packages OK [12:49:51] RECOVERY - Host mw1121 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [12:49:51] PROBLEM - Apache HTTP on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:49:51] PROBLEM - Disk space on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:49:51] PROBLEM - DPKG on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:01] RECOVERY - Apache HTTP on mw1129 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.136 second response time [12:50:01] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:01] RECOVERY - RAID on mw1121 is OK: OK: no RAID installed [12:50:01] RECOVERY - DPKG on mw1134 is OK: All packages OK [12:50:01] PROBLEM - RAID on mw1127 is CRITICAL: Timeout while attempting connection [12:50:11] PROBLEM - Disk space on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:11] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:50:11] RECOVERY - DPKG on mw1120 is OK: All packages OK [12:50:11] RECOVERY - Disk space on mw1134 is OK: DISK OK [12:50:11] RECOVERY - twemproxy process on mw1134 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:50:11] PROBLEM - RAID on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:11] PROBLEM - SSH on mw1127 is CRITICAL: Connection timed out [12:50:21] PROBLEM - DPKG on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:21] PROBLEM - SSH on mw1135 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:21] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:50:21] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:50:31] PROBLEM - Disk space on mw1127 is CRITICAL: Timeout while attempting connection [12:50:31] PROBLEM - twemproxy process on mw1127 is CRITICAL: Timeout while attempting connection [12:50:31] RECOVERY - Disk space on mw1126 is OK: DISK OK [12:50:31] PROBLEM - SSH on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:31] PROBLEM - DPKG on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:32] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:32] PROBLEM - DPKG on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:33] PROBLEM - RAID on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:33] PROBLEM - Disk space on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:50:34] PROBLEM - SSH on mw1133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:41] RECOVERY - SSH on mw1140 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:50:51] RECOVERY - twemproxy process on mw1140 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:51:01] RECOVERY - RAID on mw1134 is OK: OK: no RAID installed [12:51:11] RECOVERY - Disk space on mw1139 is OK: DISK OK [12:51:21] RECOVERY - DPKG on mw1126 is OK: All packages OK [12:51:22] RECOVERY - RAID on mw1126 is OK: OK: no RAID installed [12:51:22] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:31] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:31] PROBLEM - Disk space on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:51:51] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:52:01] RECOVERY - Disk space on mw1119 is OK: DISK OK [12:52:11] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:52:11] RECOVERY - Disk space on mw1125 is OK: DISK OK [12:52:11] RECOVERY - Apache HTTP on mw1125 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.837 second response time [12:52:11] PROBLEM - RAID on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:52:11] PROBLEM - DPKG on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:52:21] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:21] RECOVERY - Disk space on mw1127 is OK: DISK OK [12:52:21] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:52:31] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:52:31] PROBLEM - RAID on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:52:31] PROBLEM - DPKG on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:52:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:31] PROBLEM - twemproxy process on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:52:32] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:52:41] RECOVERY - DPKG on mw1127 is OK: All packages OK [12:52:51] PROBLEM - Disk space on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:53:01] RECOVERY - RAID on mw1127 is OK: OK: no RAID installed [12:53:01] RECOVERY - RAID on mw1138 is OK: OK: no RAID installed [12:53:11] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:53:11] RECOVERY - DPKG on mw1133 is OK: All packages OK [12:53:11] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.052 second response time [12:53:21] PROBLEM - DPKG on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:53:21] RECOVERY - DPKG on mw1138 is OK: All packages OK [12:53:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [12:53:21] RECOVERY - SSH on mw1119 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:53:21] RECOVERY - Disk space on mw1133 is OK: DISK OK [12:53:22] RECOVERY - SSH on mw1133 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:53:31] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:53:31] RECOVERY - Disk space on mw1144 is OK: DISK OK [12:53:41] PROBLEM - RAID on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:53:41] RECOVERY - Apache HTTP on mw1119 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.777 second response time [12:53:41] RECOVERY - Disk space on mw1128 is OK: DISK OK [12:53:41] RECOVERY - RAID on mw1125 is OK: OK: no RAID installed [12:53:51] PROBLEM - twemproxy process on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:53:51] PROBLEM - Host mw1123 is DOWN: PING CRITICAL - Packet loss = 100% [12:54:01] RECOVERY - RAID on mw1119 is OK: OK: no RAID installed [12:54:08] !log Restarted Apache on mw1118, had to stop by hand using killall -9 (/etc/init.d/apache stop didn't work) [12:54:11] RECOVERY - DPKG on mw1144 is OK: All packages OK [12:54:11] RECOVERY - DPKG on mw1119 is OK: All packages OK [12:54:11] RECOVERY - DPKG on mw1128 is OK: All packages OK [12:54:11] RECOVERY - DPKG on mw1130 is OK: All packages OK [12:54:17] Logged the message, Mr. Obvious [12:54:21] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:54:21] RECOVERY - Host mw1123 is UP: PING OK - Packet loss = 0%, RTA = 1.24 ms [12:54:21] RECOVERY - RAID on mw1144 is OK: OK: no RAID installed [12:54:21] RECOVERY - DPKG on mw1125 is OK: All packages OK [12:54:31] RECOVERY - RAID on mw1141 is OK: OK: no RAID installed [12:54:31] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:54:41] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:54:51] PROBLEM - SSH on mw1140 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:01] RECOVERY - Disk space on mw1142 is OK: DISK OK [12:55:41] RECOVERY - twemproxy process on mw1139 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:55:51] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:55:51] PROBLEM - Apache HTTP on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:11] PROBLEM - DPKG on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:56:11] PROBLEM - RAID on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:56:11] RECOVERY - Disk space on mw1139 is OK: DISK OK [12:56:11] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.417 second response time [12:56:21] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:21] RECOVERY - DPKG on mw1140 is OK: All packages OK [12:56:21] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:56:21] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:56:22] RECOVERY - Disk space on mw1140 is OK: DISK OK [12:56:31] RECOVERY - DPKG on mw1139 is OK: All packages OK [12:56:31] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:56:31] PROBLEM - DPKG on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:56:31] PROBLEM - RAID on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:56:31] PROBLEM - RAID on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:56:32] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:56:41] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:56:51] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:01] RECOVERY - DPKG on mw1142 is OK: All packages OK [12:57:11] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:11] RECOVERY - RAID on mw1142 is OK: OK: no RAID installed [12:57:11] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:57:11] PROBLEM - Host mw1133 is DOWN: PING CRITICAL - Packet loss = 100% [12:57:11] PROBLEM - Apache HTTP on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:12] PROBLEM - Apache HTTP on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:12] PROBLEM - RAID on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:13] PROBLEM - DPKG on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:21] PROBLEM - SSH on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:21] PROBLEM - twemproxy process on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:21] PROBLEM - DPKG on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:22] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:22] PROBLEM - Disk space on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:22] PROBLEM - DPKG on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:22] PROBLEM - DPKG on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:23] RECOVERY - DPKG on mw1143 is OK: All packages OK [12:57:23] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:31] PROBLEM - Apache HTTP on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:31] PROBLEM - twemproxy process on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:31] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:31] PROBLEM - DPKG on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:31] PROBLEM - twemproxy process on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:32] PROBLEM - SSH on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:32] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:33] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:33] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:34] PROBLEM - RAID on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:34] PROBLEM - RAID on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:35] PROBLEM - RAID on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:35] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:36] PROBLEM - Apache HTTP on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:36] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:37] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:37] PROBLEM - DPKG on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:38] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:38] PROBLEM - Disk space on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:41] PROBLEM - RAID on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:41] PROBLEM - RAID on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:41] PROBLEM - RAID on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:41] RECOVERY - twemproxy process on mw1133 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:57:41] RECOVERY - RAID on mw1133 is OK: OK: no RAID installed [12:57:51] RECOVERY - Host mw1133 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [12:57:51] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:51] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:51] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:57:51] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:57:51] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:01] RECOVERY - RAID on mw1143 is OK: OK: no RAID installed [12:58:01] PROBLEM - Disk space on mw1145 is CRITICAL: Timeout while attempting connection [12:58:11] RECOVERY - Apache HTTP on mw1123 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.360 second response time [12:58:11] PROBLEM - DPKG on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:11] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:58:11] PROBLEM - RAID on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:11] PROBLEM - RAID on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:11] PROBLEM - RAID on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:11] PROBLEM - twemproxy process on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:21] PROBLEM - DPKG on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:21] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:21] PROBLEM - RAID on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:21] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:22] RECOVERY - RAID on mw1131 is OK: OK: no RAID installed [12:58:30] Issues? [12:58:31] PROBLEM - Apache HTTP on mw1115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:31] PROBLEM - RAID on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:31] PROBLEM - Apache HTTP on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:31] PROBLEM - DPKG on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:32] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:41] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:41] PROBLEM - SSH on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:41] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:41] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:58:41] RECOVERY - Apache HTTP on mw1131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time [12:58:42] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:58:51] PROBLEM - DPKG on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:51] PROBLEM - RAID on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:51] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:51] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:51] PROBLEM - twemproxy process on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:52] PROBLEM - twemproxy process on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:58:57] Notpeter, do I sense issues? [12:59:01] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:59:01] RECOVERY - DPKG on mw1131 is OK: All packages OK [12:59:11] PROBLEM - DPKG on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:59:11] PROBLEM - Disk space on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:59:11] RECOVERY - DPKG on mw1144 is OK: All packages OK [12:59:11] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time [12:59:21] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:59:21] PROBLEM - DPKG on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:59:21] RECOVERY - Disk space on mw1144 is OK: DISK OK [12:59:22] RECOVERY - RAID on mw1144 is OK: OK: no RAID installed [12:59:22] RECOVERY - Apache HTTP on mw1115 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.078 second response time [12:59:22] RECOVERY - Disk space on mw1130 is OK: DISK OK [12:59:22] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:59:31] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:59:31] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:59:31] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:59:41] PROBLEM - DPKG on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [12:59:41] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:59:41] RECOVERY - twemproxy process on mw1130 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [12:59:51] RECOVERY - Disk space on mw1135 is OK: DISK OK [12:59:51] PROBLEM - Apache HTTP on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:51] PROBLEM - Apache HTTP on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:51] PROBLEM - Host mw1145 is DOWN: PING CRITICAL - Packet loss = 100% [13:00:11] PROBLEM - Apache HTTP on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:11] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:00:11] RECOVERY - SSH on mw1135 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:00:11] RECOVERY - DPKG on mw1130 is OK: All packages OK [13:00:11] RECOVERY - RAID on mw1138 is OK: OK: no RAID installed [13:00:11] PROBLEM - DPKG on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:00:12] RECOVERY - Disk space on mw1120 is OK: DISK OK [13:00:21] PROBLEM - RAID on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:00:21] PROBLEM - Apache HTTP on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:21] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:21] RECOVERY - RAID on mw1130 is OK: OK: no RAID installed [13:00:21] RECOVERY - twemproxy process on mw1145 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:00:22] RECOVERY - RAID on mw1145 is OK: OK: no RAID installed [13:00:22] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:00:22] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:00:23] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:00:24] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:00:24] RECOVERY - DPKG on mw1145 is OK: All packages OK [13:00:31] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:31] RECOVERY - Host mw1145 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [13:00:41] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:00:41] RECOVERY - Apache HTTP on mw1148 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.198 second response time [13:00:51] RECOVERY - DPKG on mw1115 is OK: All packages OK [13:00:51] PROBLEM - SSH on mw1117 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:51] PROBLEM - Apache HTTP on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:01] RECOVERY - Disk space on mw1145 is OK: DISK OK [13:01:11] PROBLEM - RAID on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:01:11] RECOVERY - DPKG on mw1120 is OK: All packages OK [13:01:21] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:21] RECOVERY - Apache HTTP on mw1120 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.556 second response time [13:01:21] RECOVERY - Disk space on mw1137 is OK: DISK OK [13:01:31] RECOVERY - RAID on mw1120 is OK: OK: no RAID installed [13:01:31] PROBLEM - DPKG on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:01:32] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:01:32] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:01:41] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:01:41] RECOVERY - SSH on mw1117 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:01:51] RECOVERY - twemproxy process on mw1137 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:01:51] RECOVERY - twemproxy process on mw1139 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:02:01] RECOVERY - DPKG on mw1118 is OK: All packages OK [13:02:01] RECOVERY - twemproxy process on mw1122 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:02:11] RECOVERY - RAID on mw1115 is OK: OK: no RAID installed [13:02:31] RECOVERY - twemproxy process on mw1135 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:02:31] PROBLEM - twemproxy process on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:02:31] PROBLEM - RAID on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:02:31] PROBLEM - Disk space on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:02:31] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:02:32] PROBLEM - Host mw72 is DOWN: PING CRITICAL - Packet loss = 100% [13:02:41] PROBLEM - Apache HTTP on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:51] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:51] PROBLEM - Apache HTTP on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:51] PROBLEM - RAID on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:02:51] PROBLEM - DPKG on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:02:51] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:02:51] PROBLEM - Apache HTTP on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:52] PROBLEM - Disk space on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:11] PROBLEM - DPKG on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:11] PROBLEM - RAID on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:11] PROBLEM - Apache HTTP on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:11] RECOVERY - Disk space on mw1148 is OK: DISK OK [13:03:11] PROBLEM - Apache HTTP on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:11] PROBLEM - DPKG on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:12] PROBLEM - DPKG on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:21] PROBLEM - SSH on mw1135 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:21] PROBLEM - DPKG on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:21] PROBLEM - RAID on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:21] PROBLEM - RAID on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:22] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:22] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:03:31] RECOVERY - Disk space on mw1122 is OK: DISK OK [13:03:31] PROBLEM - Apache HTTP on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:31] PROBLEM - Apache HTTP on mw1115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:31] PROBLEM - Apache HTTP on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:31] PROBLEM - RAID on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:31] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:31] PROBLEM - DPKG on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:32] PROBLEM - twemproxy process on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:32] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:33] RECOVERY - Host mw72 is UP: PING OK - Packet loss = 0%, RTA = 26.54 ms [13:03:41] PROBLEM - Apache HTTP on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:41] RECOVERY - Disk space on mw1135 is OK: DISK OK [13:03:41] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:03:51] PROBLEM - DPKG on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:51] PROBLEM - DPKG on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:51] PROBLEM - twemproxy process on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:01] RECOVERY - DPKG on mw1148 is OK: All packages OK [13:04:01] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:04:11] PROBLEM - RAID on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:11] PROBLEM - Apache HTTP on mw1129 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:11] RECOVERY - SSH on mw1135 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:04:11] PROBLEM - RAID on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:21] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:21] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:21] PROBLEM - DPKG on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:21] PROBLEM - DPKG on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:22] RECOVERY - Apache HTTP on mw1115 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.045 second response time [13:04:22] RECOVERY - Disk space on mw1140 is OK: DISK OK [13:04:31] RECOVERY - RAID on mw1148 is OK: OK: no RAID installed [13:04:31] RECOVERY - SSH on mw1134 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:04:31] PROBLEM - Apache HTTP on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:31] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:31] PROBLEM - DPKG on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:32] PROBLEM - twemproxy process on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:32] PROBLEM - RAID on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:33] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:33] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:34] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:34] PROBLEM - Apache HTTP on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:42] PROBLEM - RAID on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:42] PROBLEM - RAID on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:42] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:42] RECOVERY - Apache HTTP on mw1118 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.640 second response time [13:04:42] RECOVERY - DPKG on mw1132 is OK: All packages OK [13:04:42] RECOVERY - twemproxy process on mw1129 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:04:52] PROBLEM - Apache HTTP on mw1124 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:52] RECOVERY - DPKG on mw1115 is OK: All packages OK [13:04:52] RECOVERY - RAID on mw1129 is OK: OK: no RAID installed [13:04:52] PROBLEM - DPKG on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:52] PROBLEM - twemproxy process on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:52] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:52] PROBLEM - twemproxy process on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:04:53] PROBLEM - twemproxy process on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:05:02] RECOVERY - RAID on mw1134 is OK: OK: no RAID installed [13:05:12] PROBLEM - RAID on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:05:12] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:05:12] RECOVERY - Disk space on mw1139 is OK: DISK OK [13:05:12] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:05:22] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:22] PROBLEM - DPKG on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:05:22] RECOVERY - SSH on mw1119 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:05:23] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:05:29] !log catrope synchronized wmf-config/CommonSettings.php 'Set $wgParsoidSkipRatio to 1 to let the API cluster breathe' [13:05:38] PROBLEM - RAID on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:05:38] RECOVERY - Disk space on mw1131 is OK: DISK OK [13:05:38] PROBLEM - DPKG on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:05:38] PROBLEM - twemproxy process on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:05:40] Logged the message, Master [13:05:42] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:05:42] RECOVERY - twemproxy process on mw1139 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:06:02] RECOVERY - Disk space on mw1119 is OK: DISK OK [13:06:02] RECOVERY - RAID on mw1116 is OK: OK: no RAID installed [13:06:02] RECOVERY - DPKG on mw1131 is OK: All packages OK [13:06:02] RECOVERY - DPKG on mw1117 is OK: All packages OK [13:06:12] PROBLEM - twemproxy process on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:06:12] PROBLEM - Disk space on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:06:12] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:06:12] RECOVERY - Apache HTTP on mw1116 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time [13:06:12] RECOVERY - RAID on mw1117 is OK: OK: no RAID installed [13:06:12] RECOVERY - DPKG on mw1133 is OK: All packages OK [13:06:22] RECOVERY - Disk space on mw1120 is OK: DISK OK [13:06:22] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:06:23] RECOVERY - twemproxy process on mw1131 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:06:23] RECOVERY - RAID on mw1131 is OK: OK: no RAID installed [13:06:23] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:06:32] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:06:32] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:32] PROBLEM - RAID on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:06:32] PROBLEM - RAID on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:06:32] PROBLEM - twemproxy process on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:06:32] PROBLEM - DPKG on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:06:42] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:06:42] RECOVERY - Disk space on mw1146 is OK: DISK OK [13:06:42] RECOVERY - Apache HTTP on mw1131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.395 second response time [13:06:52] RECOVERY - DPKG on mw1127 is OK: All packages OK [13:06:52] PROBLEM - Apache HTTP on mw1148 is CRITICAL: Connection timed out [13:06:52] PROBLEM - Disk space on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:06:52] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:07:02] RECOVERY - Disk space on mw1142 is OK: DISK OK [13:07:02] RECOVERY - RAID on mw1146 is OK: OK: no RAID installed [13:07:02] RECOVERY - DPKG on mw1142 is OK: All packages OK [13:07:12] PROBLEM - twemproxy process on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:07:12] RECOVERY - Disk space on mw1145 is OK: DISK OK [13:07:12] RECOVERY - twemproxy process on mw1146 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:07:12] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.060 second response time [13:07:12] RECOVERY - RAID on mw1142 is OK: OK: no RAID installed [13:07:22] PROBLEM - SSH on mw1135 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:22] PROBLEM - DPKG on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:07:32] PROBLEM - Apache HTTP on mw1191 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:32] PROBLEM - twemproxy process on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:07:32] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:33] PROBLEM - Disk space on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:07:34] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:07:34] PROBLEM - SSH on mw1148 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:42] PROBLEM - RAID on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:07:42] PROBLEM - SSH on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:42] PROBLEM - Apache HTTP on mw1199 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:07:42] RECOVERY - Apache HTTP on mw1124 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.877 second response time [13:07:42] RECOVERY - twemproxy process on mw1137 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:07:52] PROBLEM - DPKG on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:07:52] PROBLEM - twemproxy process on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:07:52] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:08:02] RECOVERY - DPKG on mw1134 is OK: All packages OK [13:08:02] RECOVERY - DPKG on mw1118 is OK: All packages OK [13:08:12] PROBLEM - DPKG on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:08:12] PROBLEM - RAID on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:08:22] RECOVERY - Apache HTTP on mw1191 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.051 second response time [13:08:22] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:23] PROBLEM - Disk space on mw1148 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:08:23] PROBLEM - RAID on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:08:23] RECOVERY - RAID on mw1132 is OK: OK: no RAID installed [13:08:23] RECOVERY - DPKG on mw1128 is OK: All packages OK [13:08:23] RECOVERY - Apache HTTP on mw1146 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.178 second response time [13:08:23] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 0.071 second response time [13:08:24] RECOVERY - RAID on mw1118 is OK: OK: no RAID installed [13:08:24] RECOVERY - Apache HTTP on mw1121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.595 second response time [13:08:32] RECOVERY - twemproxy process on mw1135 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:08:32] PROBLEM - Disk space on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:08:32] RECOVERY - Apache HTTP on mw1199 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.046 second response time [13:08:33] PROBLEM - Disk space on mw1138 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:08:42] RECOVERY - Disk space on mw1135 is OK: DISK OK [13:08:52] PROBLEM - DPKG on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:09:02] PROBLEM - RAID on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:09:02] RECOVERY - RAID on mw1135 is OK: OK: no RAID installed [13:09:12] PROBLEM - Disk space on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:09:12] RECOVERY - SSH on mw1135 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:09:22] PROBLEM - DPKG on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:09:22] PROBLEM - SSH on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:23] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:09:23] RECOVERY - DPKG on mw1135 is OK: All packages OK [13:09:23] RECOVERY - Apache HTTP on mw1139 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.210 second response time [13:09:23] RECOVERY - Disk space on mw1137 is OK: DISK OK [13:09:32] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:09:32] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:09:32] PROBLEM - SSH on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:32] PROBLEM - twemproxy process on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:09:32] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:09:32] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:09:52] RECOVERY - Disk space on mw1121 is OK: DISK OK [13:09:52] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:52] PROBLEM - DPKG on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:10:02] RECOVERY - Disk space on mw1119 is OK: DISK OK [13:10:02] PROBLEM - Host mw1148 is DOWN: PING CRITICAL - Packet loss = 100% [13:10:12] RECOVERY - twemproxy process on mw1134 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:10:12] RECOVERY - Disk space on mw1134 is OK: DISK OK [13:10:22] RECOVERY - Disk space on mw1144 is OK: DISK OK [13:10:23] RECOVERY - RAID on mw1139 is OK: OK: no RAID installed [13:10:23] RECOVERY - SSH on mw1148 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:10:32] RECOVERY - twemproxy process on mw1145 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:10:32] RECOVERY - RAID on mw1148 is OK: OK: no RAID installed [13:10:32] PROBLEM - Apache HTTP on mw1133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:32] RECOVERY - DPKG on mw1139 is OK: All packages OK [13:10:32] RECOVERY - Host mw1148 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [13:10:42] RECOVERY - twemproxy process on mw1148 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:10:42] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:10:42] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:10:52] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:11:10] RECOVERY - DPKG on mw1148 is OK: All packages OK [13:11:12] RECOVERY - Disk space on mw1148 is OK: DISK OK [13:11:12] RECOVERY - Apache HTTP on mw1125 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [13:11:13] RECOVERY - Disk space on mw1125 is OK: DISK OK [13:11:13] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.055 second response time [13:11:13] RECOVERY - DPKG on mw1144 is OK: All packages OK [13:11:22] PROBLEM - Disk space on mw1120 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:11:22] PROBLEM - RAID on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:11:22] PROBLEM - DPKG on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:11:23] RECOVERY - RAID on mw1144 is OK: OK: no RAID installed [13:11:32] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:11:32] PROBLEM - Apache HTTP on mw1115 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:32] PROBLEM - Apache HTTP on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:32] PROBLEM - twemproxy process on mw1128 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:11:52] PROBLEM - Apache HTTP on mw1124 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:52] PROBLEM - Disk space on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:11:52] PROBLEM - twemproxy process on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:11:52] PROBLEM - Disk space on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:11:52] PROBLEM - Disk space on mw1115 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:12] PROBLEM - SSH on mw1129 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:12] PROBLEM - DPKG on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:12] PROBLEM - RAID on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:12] PROBLEM - twemproxy process on mw1124 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:12] RECOVERY - SSH on mw1126 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:12:22] RECOVERY - DPKG on mw1125 is OK: All packages OK [13:12:23] RECOVERY - Disk space on mw1126 is OK: DISK OK [13:12:25] Hehe [13:12:32] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:32] PROBLEM - RAID on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:32] PROBLEM - DPKG on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:32] PROBLEM - SSH on mw1120 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:32] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:33] PROBLEM - twemproxy process on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:33] PROBLEM - twemproxy process on mw1135 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:34] PROBLEM - Disk space on mw1123 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:42] RECOVERY - Apache HTTP on mw1148 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.106 second response time [13:12:42] RECOVERY - RAID on mw1125 is OK: OK: no RAID installed [13:12:52] PROBLEM - DPKG on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:12:52] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:12] PROBLEM - DPKG on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:12] PROBLEM - DPKG on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:12] PROBLEM - RAID on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:12] PROBLEM - DPKG on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:12] PROBLEM - RAID on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:12] PROBLEM - twemproxy process on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:13] PROBLEM - Apache HTTP on mw1193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:22] PROBLEM - Apache HTTP on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:22] PROBLEM - RAID on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:22] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:22] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:23] PROBLEM - RAID on mw1117 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:32] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:33] PROBLEM - twemproxy process on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:34] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:34] PROBLEM - twemproxy process on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:34] PROBLEM - RAID on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:13:37] addweekend: ^^ [13:13:42] RECOVERY - Disk space on mw1124 is OK: DISK OK [13:13:42] PROBLEM - Apache HTTP on mw1196 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:42] PROBLEM - Apache HTTP on mw1199 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:42] PROBLEM - Apache HTTP on mw1195 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:42] PROBLEM - Apache HTTP on mw1204 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:43] PROBLEM - Apache HTTP on mw1118 is CRITICAL: Connection timed out [13:13:52] PROBLEM - Apache HTTP on mw1132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:52] PROBLEM - Apache HTTP on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:52] PROBLEM - twemproxy process on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:02] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:02] RECOVERY - Apache HTTP on mw1130 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.108 second response time [13:14:02] RECOVERY - Apache HTTP on mw1193 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.938 second response time [13:14:12] PROBLEM - Disk space on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:12] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:12] PROBLEM - Disk space on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:12] RECOVERY - DPKG on mw1128 is OK: All packages OK [13:14:13] RECOVERY - DPKG on mw1117 is OK: All packages OK [13:14:13] PROBLEM - DPKG on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:22] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:22] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:22] RECOVERY - twemproxy process on mw1115 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:14:23] RECOVERY - twemproxy process on mw1128 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:14:32] RECOVERY - DPKG on mw1124 is OK: All packages OK [13:14:32] RECOVERY - Apache HTTP on mw1196 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.190 second response time [13:14:32] PROBLEM - Apache HTTP on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:32] PROBLEM - RAID on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:32] PROBLEM - twemproxy process on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:33] PROBLEM - Disk space on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:33] RECOVERY - Apache HTTP on mw1195 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.086 second response time [13:14:34] RECOVERY - Apache HTTP on mw1199 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.173 second response time [13:14:34] RECOVERY - Apache HTTP on mw1204 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.098 second response time [13:14:42] PROBLEM - DPKG on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:42] RECOVERY - Disk space on mw1135 is OK: DISK OK [13:14:52] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:52] PROBLEM - twemproxy process on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:52] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:14:52] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:14:52] PROBLEM - twemproxy process on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:15:02] RECOVERY - Disk space on mw1145 is OK: DISK OK [13:15:02] RECOVERY - Apache HTTP on mw1126 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time [13:15:12] RECOVERY - RAID on mw1115 is OK: OK: no RAID installed [13:15:12] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.315 second response time [13:15:12] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:15:12] PROBLEM - DPKG on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:15:22] RECOVERY - Disk space on mw1139 is OK: DISK OK [13:15:22] PROBLEM - Apache HTTP on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:23] PROBLEM - SSH on mw1135 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:23] PROBLEM - Disk space on mw1129 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:15:23] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:15:23] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 0.071 second response time [13:15:24] RECOVERY - RAID on mw1126 is OK: OK: no RAID installed [13:15:24] RECOVERY - twemproxy process on mw1126 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:15:24] RECOVERY - DPKG on mw1126 is OK: All packages OK [13:15:25] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:25] RECOVERY - Disk space on mw1137 is OK: DISK OK [13:15:26] RECOVERY - Apache HTTP on mw1115 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.005 second response time [13:15:26] RECOVERY - Disk space on mw1140 is OK: DISK OK [13:15:32] RECOVERY - Disk space on mw1127 is OK: DISK OK [13:15:32] PROBLEM - RAID on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:15:32] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:15:32] PROBLEM - DPKG on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:15:33] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:52] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:15:52] PROBLEM - RAID on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:16:02] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:16:02] RECOVERY - twemproxy process on mw1124 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:16:12] RECOVERY - Disk space on mw1125 is OK: DISK OK [13:16:12] RECOVERY - Disk space on mw1134 is OK: DISK OK [13:16:22] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:16:32] RECOVERY - twemproxy process on mw1116 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:16:32] PROBLEM - Apache HTTP on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:32] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:16:42] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:16:42] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:16:42] RECOVERY - twemproxy process on mw1133 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:16:42] RECOVERY - twemproxy process on mw1139 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:16:42] RECOVERY - Apache HTTP on mw1132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.186 second response time [13:16:42] RECOVERY - Apache HTTP on mw1124 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.052 second response time [13:17:02] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:17:12] PROBLEM - Apache HTTP on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:13] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:17:13] RECOVERY - RAID on mw1117 is OK: OK: no RAID installed [13:17:13] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:17:17] !log Stealing 10 app servers, for the API pool [13:17:22] RECOVERY - Disk space on mw1133 is OK: DISK OK [13:17:23] PROBLEM - Host mw1135 is DOWN: PING CRITICAL - Packet loss = 100% [13:17:23] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:17:23] RECOVERY - Apache HTTP on mw1146 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 5.812 second response time [13:17:28] Logged the message, Master [13:17:32] RECOVERY - Disk space on mw1147 is OK: DISK OK [13:17:32] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:17:42] PROBLEM - Host mw1115 is DOWN: PING CRITICAL - Packet loss = 100% [13:18:02] RECOVERY - RAID on mw1135 is OK: OK: no RAID installed [13:18:02] RECOVERY - Disk space on mw1119 is OK: DISK OK [13:18:12] PROBLEM - Disk space on mw1145 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:18:12] PROBLEM - NTP on mw72 is CRITICAL: NTP CRITICAL: Offset unknown [13:18:12] RECOVERY - SSH on mw1135 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:18:12] RECOVERY - Host mw1135 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms [13:18:22] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:22] RECOVERY - RAID on mw1124 is OK: OK: no RAID installed [13:18:22] RECOVERY - Host mw1115 is UP: PING OK - Packet loss = 0%, RTA = 0.73 ms [13:18:23] RECOVERY - twemproxy process on mw1135 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:18:23] RECOVERY - DPKG on mw1135 is OK: All packages OK [13:18:23] RECOVERY - SSH on mw1119 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:18:23] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:18:32] RECOVERY - Disk space on mw1141 is OK: DISK OK [13:18:32] PROBLEM - Disk space on mw1140 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:18:32] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:18:42] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:18:42] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:18:42] RECOVERY - DPKG on mw1115 is OK: All packages OK [13:18:42] RECOVERY - Disk space on mw1115 is OK: DISK OK [13:18:42] RECOVERY - Disk space on mw1136 is OK: DISK OK [13:18:52] RECOVERY - SSH on mw1122 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:19:02] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.648 second response time [13:19:02] RECOVERY - Disk space on mw1143 is OK: DISK OK [13:19:12] RECOVERY - DPKG on mw1137 is OK: All packages OK [13:19:12] RECOVERY - DPKG on mw1141 is OK: All packages OK [13:19:21] !log Hand-syncing CommonSettings.php with dsh to set $wgParsoidSkipRatio = 0 [13:19:22] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:19:22] RECOVERY - SSH on mw1147 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:19:31] Logged the message, Mr. Obvious [13:19:32] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:32] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:19:52] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:52] PROBLEM - twemproxy process on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:19:52] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:19:55] !log Upping PyBal weight to 15 for stolen appservers [13:20:02] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:20:02] RECOVERY - Disk space on mw1142 is OK: DISK OK [13:20:07] Logged the message, Master [13:20:12] RECOVERY - Disk space on mw1134 is OK: DISK OK [13:20:12] RECOVERY - twemproxy process on mw1134 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:20:22] RECOVERY - RAID on mw1132 is OK: OK: no RAID installed [13:20:22] PROBLEM - SSH on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:22] PROBLEM - SSH on mw1138 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:22] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:20:23] RECOVERY - Apache HTTP on mw1134 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.060 second response time [13:20:23] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:20:23] PROBLEM - Host mw1128 is DOWN: PING CRITICAL - Packet loss = 100% [13:20:32] RECOVERY - twemproxy process on mw1121 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:20:32] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:20:33] RECOVERY - SSH on mw1134 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:20:33] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:33] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:20:33] PROBLEM - Disk space on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:20:33] RECOVERY - RAID on mw1141 is OK: OK: no RAID installed [13:20:42] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:20:42] RECOVERY - Apache HTTP on mw1119 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.611 second response time [13:20:52] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:20:52] PROBLEM - twemproxy process on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:20:52] PROBLEM - Disk space on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:20:52] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:21:02] RECOVERY - DPKG on mw1134 is OK: All packages OK [13:21:02] RECOVERY - RAID on mw1134 is OK: OK: no RAID installed [13:21:02] RECOVERY - DPKG on mw1142 is OK: All packages OK [13:21:12] RECOVERY - twemproxy process on mw1122 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:21:12] RECOVERY - RAID on mw1142 is OK: OK: no RAID installed [13:21:12] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time [13:21:12] RECOVERY - Disk space on mw1125 is OK: DISK OK [13:21:22] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:22] RECOVERY - Disk space on mw1133 is OK: DISK OK [13:21:22] RECOVERY - Host mw1128 is UP: PING OK - Packet loss = 0%, RTA = 1.15 ms [13:21:23] RECOVERY - Apache HTTP on mw1133 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.625 second response time [13:21:23] RECOVERY - Disk space on mw1122 is OK: DISK OK [13:21:23] RECOVERY - DPKG on mw1122 is OK: All packages OK [13:21:23] RECOVERY - SSH on mw1136 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:21:32] RECOVERY - Disk space on mw1130 is OK: DISK OK [13:21:32] RECOVERY - DPKG on mw1146 is OK: All packages OK [13:21:32] PROBLEM - twemproxy process on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:21:32] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:21:32] PROBLEM - SSH on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:21:42] RECOVERY - RAID on mw1128 is OK: OK: no RAID installed [13:21:42] RECOVERY - twemproxy process on mw1133 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:21:42] RECOVERY - RAID on mw1122 is OK: OK: no RAID installed [13:21:42] RECOVERY - RAID on mw1133 is OK: OK: no RAID installed [13:21:52] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:12] RECOVERY - NTP on mw72 is OK: NTP OK: Offset -0.0008192062378 secs [13:22:12] RECOVERY - DPKG on mw1133 is OK: All packages OK [13:22:13] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:22:13] RECOVERY - DPKG on mw1119 is OK: All packages OK [13:22:13] RECOVERY - SSH on mw1116 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:22:22] PROBLEM - DPKG on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:22:22] RECOVERY - SSH on mw1119 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:22:23] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:22:23] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:22:32] PROBLEM - LVS HTTP IPv4 on api.svc.eqiad.wmnet is CRITICAL: Connection timed out [13:22:34] PROBLEM - SSH on mw1147 is CRITICAL: Connection timed out [13:22:34] PROBLEM - Disk space on mw1147 is CRITICAL: Timeout while attempting connection [13:22:34] PROBLEM - Disk space on mw1131 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:22:34] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:22:34] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:22:42] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:02] RECOVERY - RAID on mw1119 is OK: OK: no RAID installed [13:23:12] RECOVERY - DPKG on mw1140 is OK: All packages OK [13:23:22] PROBLEM - SSH on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:22] PROBLEM - SSH on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:23] RECOVERY - LVS HTTP IPv4 on api.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 2759 bytes in 0.079 second response time [13:23:24] RECOVERY - Disk space on mw1140 is OK: DISK OK [13:23:24] RECOVERY - RAID on mw1140 is OK: OK: no RAID installed [13:23:32] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:23:32] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:42] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:23:42] PROBLEM - RAID on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:23:42] RECOVERY - SSH on mw1140 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:23:42] RECOVERY - twemproxy process on mw1140 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:23:52] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:52] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:23:52] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:23:52] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:24:12] PROBLEM - Apache HTTP on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:12] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:24:12] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:24:22] PROBLEM - DPKG on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:24:22] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:24:22] PROBLEM - RAID on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:24:23] RECOVERY - SSH on mw1147 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:24:23] RECOVERY - DPKG on mw1143 is OK: All packages OK [13:24:23] RECOVERY - Disk space on mw1147 is OK: DISK OK [13:24:23] RECOVERY - DPKG on mw1147 is OK: All packages OK [13:24:24] RECOVERY - Apache HTTP on mw1121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [13:24:32] PROBLEM - SSH on mw1136 is CRITICAL: Connection timed out [13:24:32] RECOVERY - RAID on mw1147 is OK: OK: no RAID installed [13:24:32] PROBLEM - DPKG on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:24:32] PROBLEM - Disk space on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:24:32] PROBLEM - Apache HTTP on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:33] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:24:33] PROBLEM - Disk space on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:24:42] RECOVERY - twemproxy process on mw1147 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:24:42] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:24:42] RECOVERY - Apache HTTP on mw1143 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.878 second response time [13:24:42] RECOVERY - Disk space on mw1121 is OK: DISK OK [13:24:52] PROBLEM - Disk space on mw1136 is CRITICAL: Timeout while attempting connection [13:24:52] PROBLEM - RAID on mw1122 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:24:52] PROBLEM - Apache HTTP on mw1132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:24:52] PROBLEM - RAID on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:25:02] RECOVERY - RAID on mw1121 is OK: OK: no RAID installed [13:25:02] PROBLEM - SSH on mw1122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:02] RECOVERY - SSH on mw1129 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:25:02] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.976 second response time [13:25:02] PROBLEM - twemproxy process on mw1122 is CRITICAL: Timeout while attempting connection [13:25:12] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:12] PROBLEM - twemproxy process on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:25:12] RECOVERY - SSH on mw1121 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:25:12] RECOVERY - Disk space on mw1129 is OK: DISK OK [13:25:12] RECOVERY - DPKG on mw1141 is OK: All packages OK [13:25:22] PROBLEM - SSH on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:22] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:25:22] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:25:23] RECOVERY - Disk space on mw1127 is OK: DISK OK [13:25:23] RECOVERY - Disk space on mw1137 is OK: DISK OK [13:25:23] RECOVERY - Disk space on mw1141 is OK: DISK OK [13:25:23] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:25:23] RECOVERY - Apache HTTP on mw1146 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 3.397 second response time [13:25:32] RECOVERY - RAID on mw1141 is OK: OK: no RAID installed [13:25:32] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:33] PROBLEM - Disk space on mw1116 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:25:42] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:25:42] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:25:42] RECOVERY - DPKG on mw1121 is OK: All packages OK [13:25:42] RECOVERY - twemproxy process on mw1137 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:25:42] RECOVERY - twemproxy process on mw1130 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:25:52] RECOVERY - DPKG on mw1132 is OK: All packages OK [13:25:52] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:26:02] RECOVERY - RAID on mw1143 is OK: OK: no RAID installed [13:26:12] RECOVERY - DPKG on mw1130 is OK: All packages OK [13:26:12] RECOVERY - SSH on mw1116 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:26:12] RECOVERY - Apache HTTP on mw1140 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.124 second response time [13:26:12] RECOVERY - DPKG on mw1137 is OK: All packages OK [13:26:22] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:26:22] RECOVERY - RAID on mw1130 is OK: OK: no RAID installed [13:26:23] RECOVERY - SSH on mw1136 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:26:32] RECOVERY - SSH on mw1120 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:26:32] RECOVERY - twemproxy process on mw1120 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:26:42] RECOVERY - Disk space on mw1136 is OK: DISK OK [13:26:42] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:26:42] PROBLEM - Host mw1122 is DOWN: PING CRITICAL - Packet loss = 100% [13:26:42] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:26:42] RECOVERY - twemproxy process on mw1139 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:26:42] RECOVERY - Disk space on mw1146 is OK: DISK OK [13:26:43] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:26:52] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:27:02] RECOVERY - twemproxy process on mw1136 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:27:02] RECOVERY - Apache HTTP on mw1130 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.388 second response time [13:27:02] RECOVERY - DPKG on mw1136 is OK: All packages OK [13:27:02] RECOVERY - twemproxy process on mw1132 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:27:02] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:27:12] RECOVERY - Apache HTTP on mw1147 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.156 second response time [13:27:13] RECOVERY - Disk space on mw1120 is OK: DISK OK [13:27:13] RECOVERY - DPKG on mw1120 is OK: All packages OK [13:27:22] RECOVERY - RAID on mw1120 is OK: OK: no RAID installed [13:27:23] RECOVERY - Disk space on mw1122 is OK: DISK OK [13:27:23] RECOVERY - RAID on mw1136 is OK: OK: no RAID installed [13:27:23] RECOVERY - DPKG on mw1122 is OK: All packages OK [13:27:32] PROBLEM - Apache HTTP on mw1133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:32] PROBLEM - SSH on mw1123 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:33] RECOVERY - Host mw1122 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [13:27:42] RECOVERY - RAID on mw1122 is OK: OK: no RAID installed [13:27:52] RECOVERY - SSH on mw1122 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:28:02] RECOVERY - twemproxy process on mw1122 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:28:02] PROBLEM - SSH on mw1133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:02] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:28:12] PROBLEM - SSH on mw1129 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:12] RECOVERY - Disk space on mw1118 is OK: DISK OK [13:28:22] PROBLEM - DPKG on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:28:22] RECOVERY - Disk space on mw1144 is OK: DISK OK [13:28:32] RECOVERY - SSH on mw1123 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:28:32] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:28:32] PROBLEM - Disk space on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:28:32] PROBLEM - twemproxy process on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:28:32] PROBLEM - Disk space on mw1127 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:28:32] PROBLEM - SSH on mw1145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:52] PROBLEM - SSH on mw1137 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:52] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:28:52] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:52] PROBLEM - twemproxy process on mw1137 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:28:52] PROBLEM - twemproxy process on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:28:53] PROBLEM - DPKG on mw1132 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:29:02] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:29:02] RECOVERY - Apache HTTP on mw1136 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.148 second response time [13:29:12] PROBLEM - Apache HTTP on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:12] RECOVERY - DPKG on mw1133 is OK: All packages OK [13:29:12] RECOVERY - Apache HTTP on mw1125 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.959 second response time [13:29:12] RECOVERY - Disk space on mw1139 is OK: DISK OK [13:29:12] PROBLEM - DPKG on mw1137 is CRITICAL: Timeout while attempting connection [13:29:22] PROBLEM - SSH on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:22] PROBLEM - SSH on mw1116 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:22] PROBLEM - DPKG on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:29:22] PROBLEM - RAID on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:29:22] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:23] RECOVERY - Apache HTTP on mw1120 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.337 second response time [13:29:23] RECOVERY - Apache HTTP on mw1133 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.279 second response time [13:29:32] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:29:42] RECOVERY - DPKG on mw1139 is OK: All packages OK [13:29:42] PROBLEM - RAID on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:29:42] RECOVERY - twemproxy process on mw1133 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:29:52] RECOVERY - SSH on mw1133 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:29:52] RECOVERY - RAID on mw1133 is OK: OK: no RAID installed [13:29:52] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:30:02] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 8.203 second response time [13:30:12] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:30:12] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:30:12] RECOVERY - DPKG on mw1141 is OK: All packages OK [13:30:12] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time [13:30:22] RECOVERY - Disk space on mw1125 is OK: DISK OK [13:30:22] RECOVERY - RAID on mw1144 is OK: OK: no RAID installed [13:30:22] RECOVERY - Apache HTTP on mw1139 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.314 second response time [13:30:23] RECOVERY - Disk space on mw1141 is OK: DISK OK [13:30:32] RECOVERY - Apache HTTP on mw1122 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.097 second response time [13:30:32] PROBLEM - Host mw1123 is DOWN: PING CRITICAL - Packet loss = 100% [13:30:42] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:30:42] PROBLEM - Host mw1137 is DOWN: PING CRITICAL - Packet loss = 100% [13:30:42] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:31:02] RECOVERY - SSH on mw1129 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:31:02] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:31:02] RECOVERY - DPKG on mw1144 is OK: All packages OK [13:31:12] RECOVERY - DPKG on mw1123 is OK: All packages OK [13:31:12] PROBLEM - DPKG on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:31:22] RECOVERY - Host mw1123 is UP: PING OK - Packet loss = 0%, RTA = 1.01 ms [13:31:23] RECOVERY - RAID on mw1123 is OK: OK: no RAID installed [13:31:23] RECOVERY - DPKG on mw1125 is OK: All packages OK [13:31:23] RECOVERY - Disk space on mw1137 is OK: DISK OK [13:31:23] RECOVERY - RAID on mw1137 is OK: OK: no RAID installed [13:31:23] RECOVERY - Disk space on mw1123 is OK: DISK OK [13:31:32] RECOVERY - RAID on mw1141 is OK: OK: no RAID installed [13:31:32] PROBLEM - Apache HTTP on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:32] PROBLEM - SSH on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:32] RECOVERY - Host mw1137 is UP: PING OK - Packet loss = 0%, RTA = 0.23 ms [13:31:42] RECOVERY - SSH on mw1137 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:31:42] RECOVERY - RAID on mw1125 is OK: OK: no RAID installed [13:31:42] RECOVERY - twemproxy process on mw1137 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:32:02] RECOVERY - twemproxy process on mw1123 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:32:02] RECOVERY - RAID on mw1146 is OK: OK: no RAID installed [13:32:12] RECOVERY - DPKG on mw1137 is OK: All packages OK [13:32:12] RECOVERY - SSH on mw1116 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:32:22] RECOVERY - RAID on mw1139 is OK: OK: no RAID installed [13:32:32] RECOVERY - SSH on mw1145 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:32:42] RECOVERY - DPKG on mw1132 is OK: All packages OK [13:33:03] PROBLEM - NTP on mw1115 is CRITICAL: NTP CRITICAL: Offset unknown [13:33:03] RECOVERY - Disk space on mw1142 is OK: DISK OK [13:33:12] RECOVERY - SSH on mw1127 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:33:12] RECOVERY - RAID on mw1132 is OK: OK: no RAID installed [13:33:12] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.798 second response time [13:33:22] PROBLEM - SSH on mw1131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:22] RECOVERY - Apache HTTP on mw1146 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [13:33:22] RECOVERY - Disk space on mw1127 is OK: DISK OK [13:33:32] RECOVERY - SSH on mw1142 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:33:32] RECOVERY - twemproxy process on mw1127 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:33:32] RECOVERY - Apache HTTP on mw1127 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 6.736 second response time [13:33:42] RECOVERY - Apache HTTP on mw1132 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [13:33:42] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:33:52] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:33:52] PROBLEM - Apache HTTP on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:02] RECOVERY - DPKG on mw1142 is OK: All packages OK [13:34:12] PROBLEM - Disk space on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:34:12] PROBLEM - RAID on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:34:12] RECOVERY - Apache HTTP on mw1137 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.130 second response time [13:34:12] RECOVERY - RAID on mw1142 is OK: OK: no RAID installed [13:34:12] PROBLEM - Apache HTTP on mw1126 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:22] PROBLEM - twemproxy process on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:34:22] PROBLEM - SSH on mw1143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:32] PROBLEM - RAID on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:34:33] PROBLEM - DPKG on mw1126 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:34:33] PROBLEM - DPKG on mw1143 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:34:42] PROBLEM - DPKG on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:34:52] PROBLEM - SSH on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:34:52] PROBLEM - twemproxy process on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:34:52] PROBLEM - Disk space on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:02] RECOVERY - Disk space on mw1145 is OK: DISK OK [13:35:02] RECOVERY - Disk space on mw1143 is OK: DISK OK [13:35:12] PROBLEM - Apache HTTP on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:12] RECOVERY - SSH on mw1143 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:35:12] RECOVERY - twemproxy process on mw1143 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:35:12] RECOVERY - Apache HTTP on mw1126 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 9.260 second response time [13:35:12] PROBLEM - RAID on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:12] PROBLEM - RAID on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:12] PROBLEM - RAID on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:13] PROBLEM - DPKG on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:13] PROBLEM - twemproxy process on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:22] PROBLEM - Disk space on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:22] PROBLEM - Disk space on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:22] PROBLEM - DPKG on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:22] PROBLEM - DPKG on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:22] PROBLEM - Apache HTTP on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:22] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:22] PROBLEM - Disk space on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:23] RECOVERY - twemproxy process on mw1145 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:35:23] RECOVERY - DPKG on mw1126 is OK: All packages OK [13:35:24] RECOVERY - RAID on mw1126 is OK: OK: no RAID installed [13:35:24] RECOVERY - RAID on mw1145 is OK: OK: no RAID installed [13:35:25] RECOVERY - DPKG on mw1146 is OK: All packages OK [13:35:25] RECOVERY - DPKG on mw1129 is OK: All packages OK [13:35:26] RECOVERY - DPKG on mw1145 is OK: All packages OK [13:35:26] PROBLEM - Apache HTTP on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:32] RECOVERY - DPKG on mw1143 is OK: All packages OK [13:35:32] RECOVERY - Disk space on mw1138 is OK: DISK OK [13:35:32] PROBLEM - RAID on mw1139 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:32] PROBLEM - Apache HTTP on mw1139 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:32] PROBLEM - Disk space on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:32] PROBLEM - DPKG on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:33] PROBLEM - Disk space on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:33] PROBLEM - twemproxy process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:33] PROBLEM - SSH on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:34] PROBLEM - RAID on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:34] PROBLEM - SSH on mw1125 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:42] PROBLEM - SSH on mw1134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:42] PROBLEM - RAID on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:42] RECOVERY - twemproxy process on mw1129 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:35:42] RECOVERY - Apache HTTP on mw1143 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.060 second response time [13:35:42] RECOVERY - Disk space on mw1146 is OK: DISK OK [13:35:52] RECOVERY - RAID on mw1129 is OK: OK: no RAID installed [13:35:52] PROBLEM - SSH on mw1141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:52] PROBLEM - SSH on mw1144 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:52] PROBLEM - Apache HTTP on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:52] PROBLEM - twemproxy process on mw1144 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:53] PROBLEM - twemproxy process on mw1141 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:35:53] PROBLEM - RAID on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:36:02] PROBLEM - twemproxy process on mw1125 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:36:02] RECOVERY - RAID on mw1143 is OK: OK: no RAID installed [13:36:02] RECOVERY - RAID on mw1138 is OK: OK: no RAID installed [13:36:12] PROBLEM - DPKG on mw1134 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:36:12] PROBLEM - Apache HTTP on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:12] PROBLEM - Disk space on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:36:12] RECOVERY - Disk space on mw1125 is OK: DISK OK [13:36:12] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:36:13] RECOVERY - Disk space on mw1134 is OK: DISK OK [13:36:22] RECOVERY - Disk space on mw1144 is OK: DISK OK [13:36:23] RECOVERY - SSH on mw1125 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:36:32] RECOVERY - SSH on mw1134 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:36:32] PROBLEM - Apache HTTP on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:42] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:36:42] PROBLEM - Apache HTTP on mw1127 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:42] RECOVERY - SSH on mw1144 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:36:42] RECOVERY - twemproxy process on mw1144 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:36:52] PROBLEM - RAID on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:37:02] RECOVERY - NTP on mw1115 is OK: NTP OK: Offset -0.0001119375229 secs [13:37:02] RECOVERY - DPKG on mw1134 is OK: All packages OK [13:37:02] RECOVERY - RAID on mw1134 is OK: OK: no RAID installed [13:37:02] RECOVERY - twemproxy process on mw1134 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:37:12] PROBLEM - Disk space on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:37:12] RECOVERY - SSH on mw1131 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:37:12] RECOVERY - Disk space on mw1118 is OK: DISK OK [13:37:12] PROBLEM - DPKG on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:37:22] PROBLEM - DPKG on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:37:22] PROBLEM - RAID on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:37:22] PROBLEM - Apache HTTP on mw1142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:23] RECOVERY - Disk space on mw1141 is OK: DISK OK [13:37:32] RECOVERY - RAID on mw1141 is OK: OK: no RAID installed [13:37:32] PROBLEM - RAID on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:37:32] PROBLEM - Apache HTTP on mw1121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:32] PROBLEM - Disk space on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:37:42] PROBLEM - SSH on mw1130 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:42] RECOVERY - SSH on mw1141 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:37:42] RECOVERY - twemproxy process on mw1141 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:37:42] RECOVERY - SSH on mw1139 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:37:52] RECOVERY - twemproxy process on mw1139 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:37:52] PROBLEM - DPKG on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:37:52] PROBLEM - twemproxy process on mw1130 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:38:02] PROBLEM - twemproxy process on mw1142 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:38:02] RECOVERY - Apache HTTP on mw1141 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [13:38:02] RECOVERY - Apache HTTP on mw1130 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.561 second response time [13:38:02] RECOVERY - Disk space on mw1142 is OK: DISK OK [13:38:02] RECOVERY - DPKG on mw1144 is OK: All packages OK [13:38:12] PROBLEM - RAID on mw1121 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:38:12] RECOVERY - DPKG on mw1141 is OK: All packages OK [13:38:12] RECOVERY - Apache HTTP on mw1142 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.058 second response time [13:38:12] RECOVERY - Disk space on mw1139 is OK: DISK OK [13:38:12] RECOVERY - DPKG on mw1142 is OK: All packages OK [13:38:12] RECOVERY - Apache HTTP on mw1144 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.375 second response time [13:38:22] RECOVERY - Apache HTTP on mw1139 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 1.397 second response time [13:38:34] RECOVERY - DPKG on mw1125 is OK: All packages OK [13:38:34] RECOVERY - RAID on mw1144 is OK: OK: no RAID installed [13:38:34] RECOVERY - SSH on mw1130 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:38:34] PROBLEM - Disk space on mw1138 is CRITICAL: Timeout while attempting connection [13:38:34] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:38:42] RECOVERY - RAID on mw1133 is OK: OK: no RAID installed [13:38:42] RECOVERY - twemproxy process on mw1130 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:38:52] RECOVERY - twemproxy process on mw1142 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:38:52] RECOVERY - twemproxy process on mw1125 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:39:12] RECOVERY - RAID on mw1142 is OK: OK: no RAID installed [13:39:12] PROBLEM - RAID on mw1138 is CRITICAL: Timeout while attempting connection [13:39:12] PROBLEM - SSH on mw1138 is CRITICAL: Connection timed out [13:39:22] RECOVERY - Apache HTTP on mw1146 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.137 second response time [13:39:22] RECOVERY - RAID on mw1139 is OK: OK: no RAID installed [13:39:23] RECOVERY - DPKG on mw1146 is OK: All packages OK [13:39:23] RECOVERY - Disk space on mw1130 is OK: DISK OK [13:39:32] RECOVERY - Apache HTTP on mw1127 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.165 second response time [13:39:32] RECOVERY - DPKG on mw1139 is OK: All packages OK [13:39:42] PROBLEM - SSH on mw1118 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:39:42] RECOVERY - RAID on mw1125 is OK: OK: no RAID installed [13:39:52] PROBLEM - twemproxy process on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:39:52] PROBLEM - Host mw1131 is DOWN: PING CRITICAL - Packet loss = 100% [13:40:12] RECOVERY - DPKG on mw1130 is OK: All packages OK [13:40:12] RECOVERY - SSH on mw1138 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:40:12] RECOVERY - Apache HTTP on mw1125 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.367 second response time [13:40:22] PROBLEM - Disk space on mw1118 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:40:22] RECOVERY - Host mw1131 is UP: PING OK - Packet loss = 0%, RTA = 1.90 ms [13:40:23] RECOVERY - RAID on mw1130 is OK: OK: no RAID installed [13:40:23] RECOVERY - twemproxy process on mw1138 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:40:23] RECOVERY - DPKG on mw1138 is OK: All packages OK [13:40:23] RECOVERY - Disk space on mw1131 is OK: DISK OK [13:40:23] RECOVERY - twemproxy process on mw1131 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:40:23] RECOVERY - RAID on mw1131 is OK: OK: no RAID installed [13:40:24] RECOVERY - Disk space on mw1138 is OK: DISK OK [13:40:32] RECOVERY - Apache HTTP on mw1121 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 7.980 second response time [13:40:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:52] PROBLEM - Host mw1139 is DOWN: PING CRITICAL - Packet loss = 100% [13:41:02] RECOVERY - DPKG on mw1131 is OK: All packages OK [13:41:02] PROBLEM - Host mw1116 is DOWN: PING CRITICAL - Packet loss = 100% [13:41:02] RECOVERY - RAID on mw1138 is OK: OK: no RAID installed [13:41:22] RECOVERY - Disk space on mw1118 is OK: DISK OK [13:41:22] RECOVERY - Host mw1116 is UP: PING OK - Packet loss = 0%, RTA = 0.83 ms [13:41:23] RECOVERY - twemproxy process on mw1116 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:41:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.148 second response time [13:41:23] RECOVERY - Disk space on mw1116 is OK: DISK OK [13:41:42] RECOVERY - DPKG on mw1116 is OK: All packages OK [13:42:02] RECOVERY - RAID on mw1116 is OK: OK: no RAID installed [13:42:02] RECOVERY - DPKG on mw1118 is OK: All packages OK [13:42:02] RECOVERY - RAID on mw1121 is OK: OK: no RAID installed [13:42:22] PROBLEM - twemproxy process on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:42:22] RECOVERY - Apache HTTP on mw1138 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.162 second response time [13:42:23] RECOVERY - Host mw1139 is UP: PING OK - Packet loss = 0%, RTA = 1.63 ms [13:42:32] RECOVERY - SSH on mw1118 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:42:32] RECOVERY - twemproxy process on mw1119 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:42:32] RECOVERY - RAID on mw1118 is OK: OK: no RAID installed [13:42:33] PROBLEM - DPKG on mw1146 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:42:33] PROBLEM - Apache HTTP on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:42:42] RECOVERY - Apache HTTP on mw1131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.112 second response time [13:42:42] RECOVERY - DPKG on mw1121 is OK: All packages OK [13:42:42] RECOVERY - twemproxy process on mw1118 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:42:42] RECOVERY - Apache HTTP on mw1118 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.359 second response time [13:43:12] PROBLEM - SSH on mw1146 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:52] PROBLEM - RAID on mw1133 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:44:03] RECOVERY - RAID on mw1127 is OK: OK: no RAID installed [13:44:03] RECOVERY - SSH on mw1146 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:44:12] RECOVERY - Apache HTTP on mw1116 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.104 second response time [13:44:12] RECOVERY - twemproxy process on mw1146 is OK: PROCS OK: 1 process with UID = 65534 (nobody), command name nutcracker [13:44:12] RECOVERY - RAID on mw1146 is OK: OK: no RAID installed [13:44:22] RECOVERY - DPKG on mw1146 is OK: All packages OK [13:44:42] RECOVERY - DPKG on mw1127 is OK: All packages OK [13:45:12] PROBLEM - Host mw1118 is DOWN: PING CRITICAL - Packet loss = 100% [13:45:22] RECOVERY - Apache HTTP on mw1146 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.083 second response time [13:45:52] RECOVERY - Host mw1118 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [13:46:12] PROBLEM - NTP on mw1123 is CRITICAL: NTP CRITICAL: Offset unknown [13:47:02] RECOVERY - Disk space on mw1119 is OK: DISK OK [13:47:02] RECOVERY - RAID on mw1119 is OK: OK: no RAID installed [13:47:12] RECOVERY - DPKG on mw1119 is OK: All packages OK [13:47:22] RECOVERY - SSH on mw1119 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:49:48] !log Depooled stolen apaches from the api cluster [13:49:59] Logged the message, Master [13:50:12] RECOVERY - NTP on mw1123 is OK: NTP OK: Offset -0.0007739067078 secs [13:51:12] PROBLEM - Host mw1133 is DOWN: PING CRITICAL - Packet loss = 100% [13:51:16] mark: did you ping me? [13:51:21] i didn't [13:51:42] PROBLEM - Host mw1130 is DOWN: PING CRITICAL - Packet loss = 100% [13:51:55] Ironholds: seems to be back under control [13:52:03] mark: danke :). [13:52:20] hopefully it'll stay that way [13:52:22] RECOVERY - Host mw1133 is UP: PING OK - Packet loss = 0%, RTA = 0.62 ms [13:52:23] RECOVERY - Host mw1130 is UP: PING OK - Packet loss = 0%, RTA = 2.05 ms [13:52:33] well, the gadget thing seems screwy, but I guess that's (query) a different issue? [13:52:42] RECOVERY - RAID on mw1133 is OK: OK: no RAID installed [13:53:27] i don't know if and how those relate to the api cluster [13:53:33] but the api cluster is what we're working on [13:54:54] mark: I think it's memcahed [13:54:57] how [13:55:01] memcached is on separate boxes [13:55:05] Hmm [13:55:19] What about the twemproxy processes on those boxes? Are those for local use only? [13:55:32] yes [13:55:36] It basically looks as if the memcached entry for the gadgets list isn't there [13:55:37] Hmm [13:55:42] PROBLEM - NTP on mw1116 is CRITICAL: NTP CRITICAL: Offset unknown [13:55:47] * RoanKattouw tries making a dummy edit to gadgets-definition [13:56:52] PROBLEM - Host mw1126 is DOWN: PING CRITICAL - Packet loss = 100% [13:57:22] RECOVERY - Host mw1126 is UP: PING OK - Packet loss = 0%, RTA = 2.04 ms [13:58:22] holy shit [13:58:28] ? [13:58:28] bits in eqiad is 10 Gbps at peak [13:58:38] means a single box isn't enough to serve bits anymore ;) [13:58:56] wait it's not a cluster? [13:58:59] it is [13:59:00] !log catrope synchronized php-1.22wmf9/resources/startup.js 'touch' [13:59:03] oh [13:59:04] *whew* [13:59:10] Logged the message, Master [13:59:53] well these sure all look fine.. now :-/ [14:00:23] yeah the bits cluster is fine [14:00:25] very smooth [14:00:42] RECOVERY - NTP on mw1116 is OK: NTP OK: Offset -0.0001537799835 secs [14:01:30] !log Repooled stolen apaches back in the appserver cluster [14:01:41] Logged the message, Master [14:04:41] !log catrope synchronized wmf-config/CommonSettings.php 'Resync for boxes that came back up' [14:04:52] Logged the message, Master [14:08:00] stolen apaches? [14:08:48] cmjohnson1: I panic-pinged you before [14:10:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [14:12:08] uh huh, stolen from regular appserver pol [14:12:10] pool too [14:13:08] AzaToth: don't worry, it's nothing like stolen planets [14:19:02] * apergos thinks stealing a planet would be a pretty great prank [14:22:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [14:32:50] apergos: the Judoon might come after us if we stole a planet [14:33:12] "might" [14:33:24] see hw much flexibility there is in that one little word? [14:33:40] anyways I am now officially afk for rest of the day or until the next emergency... [14:33:41] happy trails [14:50:09] PROBLEM - Puppet freshness on ms-be5 is CRITICAL: No successful Puppet run in the last 10 hours [14:52:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:29] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.131 second response time [15:02:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [15:31:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [15:40:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:41:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [15:53:51] PROBLEM - Puppet freshness on mw1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:05:16] PROBLEM - Puppet freshness on db78 is CRITICAL: No successful Puppet run in the last 10 hours [16:31:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [16:32:59] nice to see icinga-wm is only pestering with the usual puppet master on stafford thingi [16:34:11] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:40:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:11] PROBLEM - Puppet freshness on rubidium is CRITICAL: No successful Puppet run in the last 10 hours [16:41:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.122 second response time [16:42:11] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [16:42:11] PROBLEM - Puppet freshness on mw1007 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:11] PROBLEM - Puppet freshness on manganese is CRITICAL: No successful Puppet run in the last 10 hours [16:42:11] PROBLEM - Puppet freshness on mw1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:11] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:11] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:11] PROBLEM - Puppet freshness on mw1171 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:12] PROBLEM - Puppet freshness on mw1197 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:12] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:13] PROBLEM - Puppet freshness on mw1210 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:13] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:14] PROBLEM - Puppet freshness on mw121 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:14] PROBLEM - Puppet freshness on search1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:15] PROBLEM - Puppet freshness on solr1003 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:15] PROBLEM - Puppet freshness on solr3 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:16] PROBLEM - Puppet freshness on srv292 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:16] PROBLEM - Puppet freshness on search18 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:17] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [16:42:17] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:18] PROBLEM - Puppet freshness on sq76 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:11] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:11] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:11] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:11] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:11] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:12] PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:12] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:13] PROBLEM - Puppet freshness on db1044 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:13] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:14] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [16:43:14] PROBLEM - Puppet freshness on mc1007 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:15] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:15] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:16] PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:16] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:17] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:17] PROBLEM - Puppet freshness on pc1 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:18] PROBLEM - Puppet freshness on potassium is CRITICAL: No successful Puppet run in the last 10 hours [16:43:18] PROBLEM - Puppet freshness on praseodymium is CRITICAL: No successful Puppet run in the last 10 hours [16:43:19] PROBLEM - Puppet freshness on rdb1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:19] PROBLEM - Puppet freshness on sq54 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:20] PROBLEM - Puppet freshness on sq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:20] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:21] PROBLEM - Puppet freshness on srv273 is CRITICAL: No successful Puppet run in the last 10 hours [16:43:21] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:11] PROBLEM - Puppet freshness on db1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:11] PROBLEM - Puppet freshness on cp1010 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:11] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:11] PROBLEM - Puppet freshness on mw1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:11] PROBLEM - Puppet freshness on mw1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:11] PROBLEM - Puppet freshness on mw1003 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:12] PROBLEM - Puppet freshness on mw1069 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:12] PROBLEM - Puppet freshness on mw1046 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:13] PROBLEM - Puppet freshness on mw1201 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:13] PROBLEM - Puppet freshness on mw106 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:14] PROBLEM - Puppet freshness on mw1189 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:14] PROBLEM - Puppet freshness on mw1205 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:15] PROBLEM - Puppet freshness on mw2 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:15] PROBLEM - Puppet freshness on mw79 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:16] PROBLEM - Puppet freshness on mw1150 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:17] PROBLEM - Puppet freshness on mw98 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:17] PROBLEM - Puppet freshness on mw35 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:18] PROBLEM - Puppet freshness on rdb1003 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:18] PROBLEM - Puppet freshness on search33 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:19] PROBLEM - Puppet freshness on wtp1007 is CRITICAL: No successful Puppet run in the last 10 hours [16:44:19] PROBLEM - Puppet freshness on srv193 is CRITICAL: No successful Puppet run in the last 10 hours [16:45:11] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [16:45:11] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [16:45:11] PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 10 hours [16:45:11] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: No successful Puppet run in the last 10 hours [16:45:11] PROBLEM - Puppet freshness on db1033 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:11] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours [16:47:11] PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:11] PROBLEM - Puppet freshness on cp3011 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:11] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:11] PROBLEM - Puppet freshness on db1014 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:11] PROBLEM - Puppet freshness on labstore1 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:11] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:12] PROBLEM - Puppet freshness on labstore3 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:12] PROBLEM - Puppet freshness on db1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:13] PROBLEM - Puppet freshness on mc1012 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:13] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:14] PROBLEM - Puppet freshness on ms-be12 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:14] PROBLEM - Puppet freshness on mw1104 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:15] PROBLEM - Puppet freshness on mw1206 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:15] PROBLEM - Puppet freshness on mw1208 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:16] PROBLEM - Puppet freshness on mw1027 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:16] PROBLEM - Puppet freshness on mw1211 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:17] PROBLEM - Puppet freshness on mw42 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:17] PROBLEM - Puppet freshness on mw75 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:18] PROBLEM - Puppet freshness on search25 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:18] PROBLEM - Puppet freshness on solr1 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:19] PROBLEM - Puppet freshness on srv242 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:19] PROBLEM - Puppet freshness on ssl1 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:20] PROBLEM - Puppet freshness on virt11 is CRITICAL: No successful Puppet run in the last 10 hours [16:47:20] PROBLEM - Puppet freshness on wtp1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:48:11] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [16:48:11] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [16:48:11] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:48:11] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:48:11] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:50:11] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [16:50:11] PROBLEM - Puppet freshness on cp1038 is CRITICAL: No successful Puppet run in the last 10 hours [16:50:11] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [16:50:11] PROBLEM - Puppet freshness on db77 is CRITICAL: No successful Puppet run in the last 10 hours [16:50:11] PROBLEM - Puppet freshness on db65 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:11] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:11] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:11] PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 10 hours [16:51:11] PROBLEM - Puppet freshness on cp1065 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:11] PROBLEM - Puppet freshness on dataset2 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:11] PROBLEM - Puppet freshness on db1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:12] PROBLEM - Puppet freshness on db32 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:12] PROBLEM - Puppet freshness on db51 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:13] PROBLEM - Puppet freshness on db57 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:13] PROBLEM - Puppet freshness on es5 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:14] PROBLEM - Puppet freshness on mc1008 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:14] PROBLEM - Puppet freshness on mw1138 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:15] PROBLEM - Puppet freshness on mw26 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:15] PROBLEM - Puppet freshness on mw33 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:16] PROBLEM - Puppet freshness on mw36 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:16] PROBLEM - Puppet freshness on mw64 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:17] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:17] PROBLEM - Puppet freshness on sq50 is CRITICAL: No successful Puppet run in the last 10 hours [16:51:18] PROBLEM - Puppet freshness on tarin is CRITICAL: No successful Puppet run in the last 10 hours [16:51:18] PROBLEM - Puppet freshness on wtp1013 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:11] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [16:52:11] PROBLEM - Puppet freshness on db1010 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:11] PROBLEM - Puppet freshness on db46 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:11] PROBLEM - Puppet freshness on db55 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:11] PROBLEM - Puppet freshness on es1009 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:11] PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:11] PROBLEM - Puppet freshness on mw1022 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:12] PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:13] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:13] PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:14] PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:14] PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:14] PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:15] PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:15] PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:16] PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:17] PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:17] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:17] PROBLEM - Puppet freshness on sq55 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:18] PROBLEM - Puppet freshness on sq64 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:19] PROBLEM - Puppet freshness on sq81 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:19] PROBLEM - Puppet freshness on wtp1021 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:19] PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 10 hours [16:52:50] !log reedy synchronized php-1.22wmf9/extensions/Gadgets/ [16:53:02] Logged the message, Master [16:53:11] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:11] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [16:53:11] PROBLEM - Puppet freshness on cp3006 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:11] PROBLEM - Puppet freshness on db31 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:11] PROBLEM - Puppet freshness on lvs6 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:11] PROBLEM - Puppet freshness on mc1011 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:12] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:12] PROBLEM - Puppet freshness on mw1072 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:13] PROBLEM - Puppet freshness on mw1134 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:13] PROBLEM - Puppet freshness on mw117 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:14] PROBLEM - Puppet freshness on mw1178 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:14] PROBLEM - Puppet freshness on mw1219 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:15] PROBLEM - Puppet freshness on mw87 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:15] PROBLEM - Puppet freshness on professor is CRITICAL: No successful Puppet run in the last 10 hours [16:53:16] PROBLEM - Puppet freshness on search1006 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:16] PROBLEM - Puppet freshness on search1011 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:17] PROBLEM - Puppet freshness on srv285 is CRITICAL: No successful Puppet run in the last 10 hours [16:53:17] PROBLEM - Puppet freshness on virt2 is CRITICAL: No successful Puppet run in the last 10 hours [16:54:11] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:54:11] PROBLEM - Puppet freshness on amssq43 is CRITICAL: No successful Puppet run in the last 10 hours [16:54:11] PROBLEM - Puppet freshness on cp1015 is CRITICAL: No successful Puppet run in the last 10 hours [16:54:11] PROBLEM - Puppet freshness on db1011 is CRITICAL: No successful Puppet run in the last 10 hours [16:54:11] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [16:54:25] !log reedy synchronized php-1.22wmf10/extensions/Gadgets/ [16:54:37] Logged the message, Master [16:56:11] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [16:56:11] PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:56:11] PROBLEM - Puppet freshness on cp1039 is CRITICAL: No successful Puppet run in the last 10 hours [16:56:11] PROBLEM - Puppet freshness on cp1054 is CRITICAL: No successful Puppet run in the last 10 hours [16:56:11] PROBLEM - Puppet freshness on cp3019 is CRITICAL: No successful Puppet run in the last 10 hours [16:57:11] PROBLEM - Puppet freshness on amssq36 is CRITICAL: No successful Puppet run in the last 10 hours [16:57:11] PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 10 hours [16:57:11] PROBLEM - Puppet freshness on cp1009 is CRITICAL: No successful Puppet run in the last 10 hours [16:57:11] PROBLEM - Puppet freshness on cp1064 is CRITICAL: No successful Puppet run in the last 10 hours [16:57:11] PROBLEM - Puppet freshness on cp1068 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:11] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:11] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:11] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:11] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:11] PROBLEM - Puppet freshness on cp1057 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:11] PROBLEM - Puppet freshness on db1028 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:11] PROBLEM - Puppet freshness on db38 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:12] PROBLEM - Puppet freshness on db68 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:13] PROBLEM - Puppet freshness on ersch is CRITICAL: No successful Puppet run in the last 10 hours [16:58:13] PROBLEM - Puppet freshness on lvs3 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:14] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:14] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:15] PROBLEM - Puppet freshness on mw1039 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:15] PROBLEM - Puppet freshness on mw1177 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:16] PROBLEM - Puppet freshness on mw1184 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:16] PROBLEM - Puppet freshness on mw120 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:17] PROBLEM - Puppet freshness on mw96 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:17] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: No successful Puppet run in the last 10 hours [16:58:18] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [16:59:11] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [16:59:11] PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 10 hours [16:59:11] PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 10 hours [16:59:11] PROBLEM - Puppet freshness on db1041 is CRITICAL: No successful Puppet run in the last 10 hours [16:59:11] PROBLEM - Puppet freshness on db1043 is CRITICAL: No successful Puppet run in the last 10 hours [17:01:12] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [17:01:12] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [17:01:12] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [17:01:12] PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 10 hours [17:01:12] PROBLEM - Puppet freshness on cp1048 is CRITICAL: No successful Puppet run in the last 10 hours [17:03:11] PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours [17:03:11] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [17:03:11] PROBLEM - Puppet freshness on cp1052 is CRITICAL: No successful Puppet run in the last 10 hours [17:03:11] PROBLEM - Puppet freshness on cp1059 is CRITICAL: No successful Puppet run in the last 10 hours [17:03:11] PROBLEM - Puppet freshness on db1006 is CRITICAL: No successful Puppet run in the last 10 hours [17:04:11] PROBLEM - Puppet freshness on amssq33 is CRITICAL: No successful Puppet run in the last 10 hours [17:04:11] PROBLEM - Puppet freshness on amssq42 is CRITICAL: No successful Puppet run in the last 10 hours [17:04:11] PROBLEM - Puppet freshness on amssq45 is CRITICAL: No successful Puppet run in the last 10 hours [17:04:11] PROBLEM - Puppet freshness on amssq49 is CRITICAL: No successful Puppet run in the last 10 hours [17:04:11] PROBLEM - Puppet freshness on amssq54 is CRITICAL: No successful Puppet run in the last 10 hours [17:05:11] PROBLEM - Puppet freshness on amssq39 is CRITICAL: No successful Puppet run in the last 10 hours [17:05:11] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [17:05:11] PROBLEM - Puppet freshness on analytics1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:05:11] PROBLEM - Puppet freshness on analytics1025 is CRITICAL: No successful Puppet run in the last 10 hours [17:05:11] PROBLEM - Puppet freshness on cp1045 is CRITICAL: No successful Puppet run in the last 10 hours [17:09:00] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [17:09:00] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: No successful Puppet run in the last 10 hours [17:09:00] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [17:09:00] PROBLEM - Puppet freshness on cp1007 is CRITICAL: No successful Puppet run in the last 10 hours [17:09:00] PROBLEM - Puppet freshness on cp1056 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:00] PROBLEM - Puppet freshness on cp1013 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:00] PROBLEM - Puppet freshness on cp3010 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:00] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:00] PROBLEM - Puppet freshness on cp1050 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:00] PROBLEM - Puppet freshness on db1048 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:00] PROBLEM - Puppet freshness on gadolinium is CRITICAL: No successful Puppet run in the last 10 hours [17:11:00] PROBLEM - Puppet freshness on lvs5 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:01] PROBLEM - Puppet freshness on ms-be4 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:01] PROBLEM - Puppet freshness on mw1044 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:02] PROBLEM - Puppet freshness on ms5 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:02] PROBLEM - Puppet freshness on mw1101 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:03] PROBLEM - Puppet freshness on mw119 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:03] PROBLEM - Puppet freshness on mw1196 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:04] PROBLEM - Puppet freshness on mw5 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:04] PROBLEM - Puppet freshness on mw1162 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:05] PROBLEM - Puppet freshness on mw83 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:05] PROBLEM - Puppet freshness on mw90 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:06] PROBLEM - Puppet freshness on search1014 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:06] PROBLEM - Puppet freshness on search21 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:07] PROBLEM - Puppet freshness on sq79 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:07] PROBLEM - Puppet freshness on srv275 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:08] PROBLEM - Puppet freshness on srv290 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:08] PROBLEM - Puppet freshness on wtp1024 is CRITICAL: No successful Puppet run in the last 10 hours [17:11:09] PROBLEM - Puppet freshness on zirconium is CRITICAL: No successful Puppet run in the last 10 hours [17:12:00] PROBLEM - Puppet freshness on db1020 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:00] PROBLEM - Puppet freshness on lvs1 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:00] PROBLEM - Puppet freshness on mc1009 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:00] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:00] PROBLEM - Puppet freshness on mw1111 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:00] PROBLEM - Puppet freshness on mw1124 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:00] PROBLEM - Puppet freshness on mw1125 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:01] PROBLEM - Puppet freshness on mw1147 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:01] PROBLEM - Puppet freshness on mw1161 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:02] PROBLEM - Puppet freshness on mw1130 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:02] PROBLEM - Puppet freshness on mw1190 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:03] PROBLEM - Puppet freshness on mw1214 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:03] PROBLEM - Puppet freshness on mw38 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:04] PROBLEM - Puppet freshness on mw8 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:04] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:05] PROBLEM - Puppet freshness on search36 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:05] PROBLEM - Puppet freshness on solr2 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:06] PROBLEM - Puppet freshness on sq51 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:06] PROBLEM - Puppet freshness on srv301 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:07] PROBLEM - Puppet freshness on wtp1011 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:07] PROBLEM - Puppet freshness on wtp1018 is CRITICAL: No successful Puppet run in the last 10 hours [17:13:00] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [17:13:00] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [17:13:00] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [17:13:00] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 10 hours [17:13:00] PROBLEM - Puppet freshness on db43 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:00] PROBLEM - Puppet freshness on amssq34 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:00] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:00] PROBLEM - Puppet freshness on cp1012 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:00] PROBLEM - Puppet freshness on db34 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:00] PROBLEM - Puppet freshness on es1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:00] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:00] PROBLEM - Puppet freshness on db1023 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:01] PROBLEM - Puppet freshness on pc1003 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:01] PROBLEM - Puppet freshness on mw1139 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:02] PROBLEM - Puppet freshness on search1018 is CRITICAL: No successful Puppet run in the last 10 hours [17:23:02] PROBLEM - Puppet freshness on srv258 is CRITICAL: No successful Puppet run in the last 10 hours [17:24:00] PROBLEM - Puppet freshness on cp1046 is CRITICAL: No successful Puppet run in the last 10 hours [17:24:00] PROBLEM - Puppet freshness on db40 is CRITICAL: No successful Puppet run in the last 10 hours [17:24:00] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [17:24:00] PROBLEM - Puppet freshness on ms-be1 is CRITICAL: No successful Puppet run in the last 10 hours [17:24:00] PROBLEM - Puppet freshness on mw20 is CRITICAL: No successful Puppet run in the last 10 hours [17:24:01] PROBLEM - Puppet freshness on search14 is CRITICAL: No successful Puppet run in the last 10 hours [17:31:47] cmjohnson1: still problem with puppet freshness I see [17:32:00] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [17:32:00] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [17:32:00] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:32:00] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [17:32:00] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [17:32:01] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [17:32:01] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [17:42:40] PROBLEM - Host mw1089 is DOWN: PING CRITICAL - Packet loss = 100% [17:45:15] New patchset: GWicke; "Throttle Parsoid template updates" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73622 [17:48:09] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73622 [17:49:44] !log gwicke synchronized wmf-config/CommonSettings.php 'Re-enable Parsoid updates after throttling template update rate' [17:49:54] Logged the message, Master [17:53:25] Azatoth: i noticed that yesterday. I want to check L-carr tomorrow about that. [18:40:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.069 second response time [18:42:11] New patchset: GWicke; "Further decrease titles per job to 6" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73631 [18:42:39] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73631 [18:43:44] !log gwicke synchronized wmf-config/CommonSettings.php 'Throttle titles per template update job further from 10 to 6' [18:43:55] Logged the message, Master [19:02:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:03:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time [19:10:58] gwicke, since I'm not really here (sunday evening), it would be pretty great if you could check on ganglia an specifically memory usage from time to time on the api boxes [19:11:02] *and [19:17:58] * paravoid hands gwicke http://isitreadonlyfriday.com/ :-) [19:19:47] apergos: yup [19:19:57] awesome [19:20:02] * apergos will now sleep well :-D [19:20:26] apergos: have a good night [19:20:32] and sorry for keeping you busy! [19:20:44] thank you! (not going to sleep just this instant, but when I do it will be good) [19:22:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time [19:25:26] some API machines currently seem to have much higher load than others [19:26:42] the newer machines can handle more so they are weight balanced (as of today) that way [19:31:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.151 second response time [19:36:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:37:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [19:51:57] paravoid: :-) re isitreadonlyfriday [20:52:00] PROBLEM - Puppet freshness on grosley is CRITICAL: No successful Puppet run in the last 10 hours [21:00:00] PROBLEM - Puppet freshness on mw56 is CRITICAL: No successful Puppet run in the last 10 hours [21:13:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [22:10:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [22:22:18] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [22:36:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:37:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [22:57:08] New patchset: Odder; "(bug 51327) Configure $wgImportSources for tnwiki, xhwiki, zuwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73716 [23:57:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds