[00:03:48] PROBLEM Total Processes is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:11:17] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [00:16:17] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [00:41:19] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [00:46:19] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [01:11:21] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [01:14:51] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 7.64, 7.25, 5.78 [01:16:21] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [01:43:29] PROBLEM Total Processes is now: WARNING on psm-precise i-000002f2 output: PROCS WARNING: 166 processes [01:44:39] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [01:49:49] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [02:04:19] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [02:09:09] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [02:15:34] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [02:20:44] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [02:43:41] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 24% free memory [02:47:45] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [02:49:15] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 1.24, 4.88, 4.00 [02:49:32] 06/29/2012 - 02:49:32 - User laner may have been modified in LDAP or locally, updating key in project(s): deployment-prep [02:55:47] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [02:55:57] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 7.31, 6.39, 4.90 [02:59:53] PROBLEM Free ram is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:00:14] PROBLEM Free ram is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [03:01:48] PROBLEM Current Load is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:06:48] PROBLEM Current Users is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:06:48] PROBLEM Current Load is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:06:48] PROBLEM dpkg-check is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:06:48] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:09:14] PROBLEM Disk Space is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:09:15] PROBLEM Current Users is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:11:05] PROBLEM Disk Space is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:11:05] PROBLEM Free ram is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [03:11:05] PROBLEM Total Processes is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:11:16] PROBLEM Free ram is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:12:19] PROBLEM dpkg-check is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:12:19] PROBLEM Total Processes is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:12:24] PROBLEM Total Processes is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:13:33] RECOVERY Disk Space is now: OK on gluster-4 i-000002e4 output: DISK OK [03:13:33] RECOVERY Current Users is now: OK on gluster-4 i-000002e4 output: USERS OK - 0 users currently logged in [03:13:33] PROBLEM Free ram is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:15:47] PROBLEM Free ram is now: UNKNOWN on gluster-4 i-000002e4 output: NRPE: Unable to read output [03:15:52] PROBLEM Free ram is now: UNKNOWN on configtest-main i-000002dd output: NRPE: Unable to read output [03:15:52] PROBLEM Current Load is now: WARNING on mobile-testing i-00000271 output: WARNING - load average: 5.41, 9.65, 8.26 [03:16:01] PROBLEM Total Processes is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:06] PROBLEM Current Users is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:07] PROBLEM dpkg-check is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:07] PROBLEM Free ram is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:07] PROBLEM Current Users is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:07] PROBLEM Disk Space is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:07] PROBLEM dpkg-check is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:07] PROBLEM Total Processes is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:08] PROBLEM Current Load is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:11] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:12] PROBLEM Disk Space is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:12] PROBLEM Current Users is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:12] PROBLEM Total Processes is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:17] PROBLEM Total Processes is now: CRITICAL on redis1 i-000002b6 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:22] PROBLEM Current Load is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:22] PROBLEM Disk Space is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:22] PROBLEM Current Load is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:22] PROBLEM Free ram is now: CRITICAL on redis1 i-000002b6 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:16:22] PROBLEM Current Load is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [03:18:01] RECOVERY Current Load is now: OK on pybal-precise i-00000289 output: OK - load average: 1.18, 2.54, 3.40 [03:18:01] RECOVERY dpkg-check is now: OK on gluster-4 i-000002e4 output: All packages OK [03:18:01] RECOVERY Total Processes is now: OK on gluster-4 i-000002e4 output: PROCS OK: 84 processes [03:18:06] RECOVERY Disk Space is now: OK on precise-test i-00000231 output: DISK OK [03:18:06] RECOVERY Free ram is now: OK on mwreview i-000002ae output: OK: 68% free memory [03:18:06] RECOVERY Current Users is now: OK on precise-test i-00000231 output: USERS OK - 0 users currently logged in [03:18:06] RECOVERY Current Load is now: OK on gluster-4 i-000002e4 output: OK - load average: 0.27, 3.43, 4.96 [03:18:06] RECOVERY dpkg-check is now: OK on precise-test i-00000231 output: All packages OK [03:18:07] RECOVERY Current Load is now: OK on precise-test i-00000231 output: OK - load average: 1.37, 3.56, 3.91 [03:18:07] RECOVERY Total Processes is now: OK on worker1 i-00000208 output: PROCS OK: 83 processes [03:18:11] RECOVERY Free ram is now: OK on precise-test i-00000231 output: OK: 80% free memory [03:18:11] RECOVERY Total Processes is now: OK on precise-test i-00000231 output: PROCS OK: 90 processes [03:18:22] PROBLEM Current Load is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:18:22] PROBLEM Current Load is now: CRITICAL on redis1 i-000002b6 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:18:22] PROBLEM Disk Space is now: CRITICAL on redis1 i-000002b6 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:18:22] PROBLEM dpkg-check is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [03:18:28] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [03:19:06] RECOVERY Free ram is now: OK on deployment-apache31 i-000002d4 output: OK: 88% free memory [03:20:28] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [03:20:28] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 6.95, 11.12, 9.40 [03:20:28] RECOVERY Disk Space is now: OK on deployment-apache31 i-000002d4 output: DISK OK [03:20:28] PROBLEM Current Load is now: WARNING on deployment-apache31 i-000002d4 output: WARNING - load average: 0.70, 4.28, 5.50 [03:20:28] PROBLEM Free ram is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [03:20:59] RECOVERY Total Processes is now: OK on deployment-apache31 i-000002d4 output: PROCS OK: 125 processes [03:21:04] RECOVERY Current Users is now: OK on deployment-apache31 i-000002d4 output: USERS OK - 0 users currently logged in [03:21:05] RECOVERY Free ram is now: OK on pybal-precise i-00000289 output: OK: 78% free memory [03:21:05] RECOVERY Current Users is now: OK on pybal-precise i-00000289 output: USERS OK - 0 users currently logged in [03:21:05] RECOVERY dpkg-check is now: OK on deployment-apache31 i-000002d4 output: All packages OK [03:21:05] RECOVERY Disk Space is now: OK on pybal-precise i-00000289 output: DISK OK [03:21:05] RECOVERY Current Load is now: OK on mwreview i-000002ae output: OK - load average: 0.34, 2.24, 3.35 [03:21:05] RECOVERY Disk Space is now: OK on configtest-main i-000002dd output: DISK OK [03:21:06] RECOVERY Current Users is now: OK on configtest-main i-000002dd output: USERS OK - 0 users currently logged in [03:21:06] RECOVERY Total Processes is now: OK on redis1 i-000002b6 output: PROCS OK: 88 processes [03:21:10] RECOVERY Free ram is now: OK on redis1 i-000002b6 output: OK: 85% free memory [03:21:10] RECOVERY Current Load is now: OK on configtest-main i-000002dd output: OK - load average: 0.45, 3.18, 3.72 [03:21:10] RECOVERY Total Processes is now: OK on pybal-precise i-00000289 output: PROCS OK: 89 processes [03:21:15] RECOVERY dpkg-check is now: OK on pybal-precise i-00000289 output: All packages OK [03:21:21] RECOVERY Current Load is now: OK on worker1 i-00000208 output: OK - load average: 3.15, 4.71, 4.86 [03:21:21] RECOVERY Total Processes is now: OK on migration1 i-00000261 output: PROCS OK: 85 processes [03:22:05] RECOVERY Current Load is now: OK on maps-tilemill1 i-00000294 output: OK - load average: 0.16, 3.28, 4.27 [03:22:05] RECOVERY Current Load is now: OK on redis1 i-000002b6 output: OK - load average: 0.97, 4.08, 4.17 [03:22:05] RECOVERY Disk Space is now: OK on redis1 i-000002b6 output: DISK OK [03:22:05] RECOVERY dpkg-check is now: OK on bots-sql2 i-000000af output: All packages OK [03:24:35] PROBLEM Free ram is now: UNKNOWN on integration-apache1 i-000002eb output: NRPE: Unable to read output [03:24:35] RECOVERY Current Load is now: OK on deployment-apache31 i-000002d4 output: OK - load average: 0.27, 1.82, 4.10 [03:26:55] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [03:29:50] RECOVERY Free ram is now: OK on bots-sql2 i-000000af output: OK: 20% free memory [03:30:39] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 1.45, 1.93, 4.12 [03:43:14] PROBLEM Free ram is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:47:37] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:49:06] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [03:49:42] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [03:49:42] PROBLEM Total Processes is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:49:47] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:53:54] PROBLEM Free ram is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:53:54] PROBLEM dpkg-check is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:56:48] PROBLEM Disk Space is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:56:49] PROBLEM Current Users is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:56:59] PROBLEM SSH is now: CRITICAL on bots-sql2 i-000000af output: CRITICAL - Socket timeout after 10 seconds [03:57:25] RECOVERY Disk Space is now: OK on worker1 i-00000208 output: DISK OK [03:57:25] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [03:58:19] PROBLEM Total Processes is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:58:44] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 12% free memory [03:58:55] PROBLEM Free ram is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [03:58:55] PROBLEM Current Load is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [03:58:55] PROBLEM dpkg-check is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [04:00:45] PROBLEM Free ram is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [04:00:45] PROBLEM Total Processes is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [04:00:50] PROBLEM Current Load is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [04:02:45] RECOVERY SSH is now: OK on bots-sql2 i-000000af output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [04:02:45] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 68 MB (5% inode=57%): [04:03:00] PROBLEM Current Load is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:03:05] PROBLEM Current Load is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:03:05] PROBLEM Current Users is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:03:05] PROBLEM Disk Space is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:03:05] PROBLEM Total Processes is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:03:13] PROBLEM Free ram is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:05:17] RECOVERY Free ram is now: OK on migration1 i-00000261 output: OK: 89% free memory [04:05:23] RECOVERY dpkg-check is now: OK on migration1 i-00000261 output: All packages OK [04:07:33] PROBLEM Current Load is now: WARNING on integration-apache1 i-000002eb output: WARNING - load average: 14.19, 13.47, 9.13 [04:08:10] PROBLEM Current Users is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [04:08:10] PROBLEM Current Users is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [04:08:10] PROBLEM Free ram is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [04:08:10] PROBLEM Disk Space is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [04:08:10] PROBLEM Total Processes is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [04:08:15] PROBLEM Disk Space is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [04:09:36] RECOVERY Disk Space is now: OK on migration1 i-00000261 output: DISK OK [04:09:36] RECOVERY Current Users is now: OK on migration1 i-00000261 output: USERS OK - 0 users currently logged in [04:09:36] RECOVERY Current Load is now: OK on migration1 i-00000261 output: OK - load average: 0.25, 1.97, 3.61 [04:12:00] RECOVERY Disk Space is now: OK on reportcard2 i-000001ea output: DISK OK [04:12:01] RECOVERY Current Users is now: OK on reportcard2 i-000001ea output: USERS OK - 0 users currently logged in [04:12:01] RECOVERY Free ram is now: OK on reportcard2 i-000001ea output: OK: 88% free memory [04:12:01] RECOVERY Total Processes is now: OK on reportcard2 i-000001ea output: PROCS OK: 86 processes [04:12:27] PROBLEM Free ram is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:12:28] PROBLEM Disk Space is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:12:28] PROBLEM Current Users is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:12:28] PROBLEM Total Processes is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:12:33] PROBLEM dpkg-check is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:13:09] RECOVERY Current Load is now: OK on upload-wizard i-0000021c output: OK - load average: 4.93, 5.13, 4.81 [04:13:09] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 14% free memory [04:13:09] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory [04:13:30] PROBLEM Current Load is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:13:30] RECOVERY Free ram is now: OK on upload-wizard i-0000021c output: OK: 93% free memory [04:13:30] RECOVERY Total Processes is now: OK on upload-wizard i-0000021c output: PROCS OK: 91 processes [04:14:26] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 11% free memory [04:15:00] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:15:09] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 13% free memory [04:15:24] PROBLEM Total Processes is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:15:54] RECOVERY Disk Space is now: OK on mwreview i-000002ae output: DISK OK [04:15:54] RECOVERY Current Users is now: OK on mwreview i-000002ae output: USERS OK - 0 users currently logged in [04:15:54] RECOVERY Total Processes is now: OK on mwreview i-000002ae output: PROCS OK: 110 processes [04:15:59] RECOVERY dpkg-check is now: OK on mwreview i-000002ae output: All packages OK [04:16:09] PROBLEM Current Load is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [04:16:39] PROBLEM Free ram is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:17:02] PROBLEM Total Processes is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:17:26] PROBLEM Current Users is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:20:48] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:20:49] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [04:20:49] PROBLEM Free ram is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [04:20:54] PROBLEM Current Users is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:20:54] PROBLEM Current Load is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:20:54] PROBLEM Total Processes is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:20:59] PROBLEM dpkg-check is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:20:59] RECOVERY Disk Space is now: OK on rds i-00000207 output: DISK OK [04:20:59] RECOVERY Total Processes is now: OK on rds i-00000207 output: PROCS OK: 96 processes [04:21:25] RECOVERY Free ram is now: OK on rds i-00000207 output: OK: 94% free memory [04:21:46] PROBLEM Free ram is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:21:46] PROBLEM Disk Space is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:21:46] PROBLEM dpkg-check is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:21:46] PROBLEM Disk Space is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:21:46] PROBLEM Total Processes is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:21:51] RECOVERY Current Load is now: OK on rds i-00000207 output: OK - load average: 0.71, 2.49, 4.21 [04:21:51] RECOVERY Current Users is now: OK on rds i-00000207 output: USERS OK - 0 users currently logged in [04:21:57] PROBLEM dpkg-check is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:22:49] RECOVERY Current Users is now: OK on upload-wizard i-0000021c output: USERS OK - 0 users currently logged in [04:22:49] RECOVERY Disk Space is now: OK on upload-wizard i-0000021c output: DISK OK [04:22:50] PROBLEM Current Load is now: WARNING on integration-apache1 i-000002eb output: WARNING - load average: 1.19, 3.20, 5.90 [04:23:15] RECOVERY Total Processes is now: OK on mobile-testing i-00000271 output: PROCS OK: 136 processes [04:24:17] PROBLEM Free ram is now: UNKNOWN on psm-precise i-000002f2 output: NRPE: Unable to read output [04:24:32] PROBLEM Free ram is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:24:38] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:25:15] RECOVERY Current Users is now: OK on maps-tilemill1 i-00000294 output: USERS OK - 0 users currently logged in [04:25:20] RECOVERY Disk Space is now: OK on maps-tilemill1 i-00000294 output: DISK OK [04:25:20] RECOVERY Free ram is now: OK on maps-tilemill1 i-00000294 output: OK: 83% free memory [04:25:54] RECOVERY dpkg-check is now: OK on maps-tilemill1 i-00000294 output: All packages OK [04:25:54] RECOVERY Total Processes is now: OK on maps-tilemill1 i-00000294 output: PROCS OK: 104 processes [04:27:52] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [04:27:52] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [04:27:57] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:28:27] RECOVERY Free ram is now: OK on worker1 i-00000208 output: OK: 92% free memory [04:29:24] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [04:30:20] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:32:39] PROBLEM Current Users is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [04:32:39] PROBLEM Disk Space is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [04:33:36] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:35:13] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:37:02] PROBLEM dpkg-check is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:37:02] PROBLEM Total Processes is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:37:07] PROBLEM Disk Space is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:37:08] PROBLEM Current Load is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:37:08] PROBLEM Current Users is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:37:08] PROBLEM Free ram is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:40:29] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [04:41:49] PROBLEM Total Processes is now: WARNING on incubator-bot2 i-00000252 output: PROCS WARNING: 156 processes [04:41:54] RECOVERY Disk Space is now: OK on incubator-bot2 i-00000252 output: DISK OK [04:41:54] RECOVERY dpkg-check is now: OK on incubator-bot2 i-00000252 output: All packages OK [04:41:54] RECOVERY Current Load is now: OK on incubator-bot2 i-00000252 output: OK - load average: 0.54, 1.64, 1.98 [04:41:54] RECOVERY Current Users is now: OK on incubator-bot2 i-00000252 output: USERS OK - 0 users currently logged in [04:41:54] RECOVERY Free ram is now: OK on incubator-bot2 i-00000252 output: OK: 35% free memory [04:41:58] PROBLEM Disk Space is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:41:58] PROBLEM Total Processes is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:42:03] PROBLEM Current Users is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:42:03] PROBLEM dpkg-check is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [04:43:40] PROBLEM Puppet freshness is now: CRITICAL on wikistats-01 i-00000042 output: Puppet has not run in last 20 hours [04:45:11] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:46:46] RECOVERY Total Processes is now: OK on incubator-bot2 i-00000252 output: PROCS OK: 148 processes [04:51:42] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [04:59:20] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [05:03:16] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.70, 0.90, 3.25 [05:08:09] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.53, 0.86, 2.61 [05:21:42] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [05:30:32] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [05:41:13] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 18% free memory [05:51:48] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [05:54:09] PROBLEM Free ram is now: UNKNOWN on gluster-4 i-000002e4 output: NRPE: Unable to read output [06:00:29] PROBLEM Free ram is now: UNKNOWN on configtest-main i-000002dd output: NRPE: Unable to read output [06:01:29] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [06:14:21] PROBLEM Free ram is now: UNKNOWN on integration-apache1 i-000002eb output: NRPE: Unable to read output [06:21:56] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [06:31:15] PROBLEM Free ram is now: CRITICAL on gluster-1 i-000002df output: CHECK_NRPE: Socket timeout after 10 seconds. [06:31:16] PROBLEM Total Processes is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:31:26] PROBLEM Free ram is now: CRITICAL on etherpad-lite i-000002de output: CHECK_NRPE: Socket timeout after 10 seconds. [06:33:37] PROBLEM Free ram is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [06:35:36] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [06:35:46] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 46.51, 22.26, 10.59 [06:35:46] PROBLEM Total Processes is now: WARNING on incubator-bot2 i-00000252 output: PROCS WARNING: 153 processes [06:35:51] PROBLEM Free ram is now: UNKNOWN on gluster-1 i-000002df output: NRPE: Unable to read output [06:35:51] PROBLEM Total Processes is now: WARNING on psm-precise i-000002f2 output: PROCS WARNING: 166 processes [06:35:57] PROBLEM Free ram is now: UNKNOWN on etherpad-lite i-000002de output: NRPE: Unable to read output [06:35:57] PROBLEM Free ram is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:40:06] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:40:06] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:43:07] RECOVERY Disk Space is now: OK on ipv6test1 i-00000282 output: DISK OK [06:43:39] PROBLEM Free ram is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:44:15] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:44:15] PROBLEM Disk Space is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:44:15] PROBLEM Current Users is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:44:15] PROBLEM Total Processes is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:45:57] PROBLEM Free ram is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:26] PROBLEM dpkg-check is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:26] PROBLEM Total Processes is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:57] PROBLEM Free ram is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:58:46] PROBLEM dpkg-check is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [06:58:46] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:59:21] PROBLEM Total Processes is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:06:11] uh? what just happened. my instance diead? [07:06:14] http://ganglia.wmflabs.org/latest/?r=2hr&cs=&ce=&c=integration&h=integration-apache1&tab=m&vn=&mc=2&z=small&metric_group=ALLGROUPS [07:06:27] since 6:20 started building up something [07:06:29] I wasn't even using it [07:14:18] PROBLEM host: nginx-dev2 is DOWN address: i-000002f0 CRITICAL - Host Unreachable (i-000002f0) [07:14:18] PROBLEM host: incubator-apache is DOWN address: i-00000211 CRITICAL - Host Unreachable (i-00000211) [07:14:18] PROBLEM Total Processes is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:15:40] PROBLEM Free ram is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:18:16] PROBLEM Free ram is now: UNKNOWN on gluster-4 i-000002e4 output: NRPE: Unable to read output [07:18:16] PROBLEM Free ram is now: UNKNOWN on integration-apache1 i-000002eb output: NRPE: Unable to read output [07:22:36] PROBLEM Free ram is now: UNKNOWN on psm-precise i-000002f2 output: NRPE: Unable to read output [07:24:42] PROBLEM Disk Space is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:24:42] PROBLEM Total Processes is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:24:52] PROBLEM Disk Space is now: CRITICAL on grail i-000002c6 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:24:53] PROBLEM Total Processes is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [07:25:32] PROBLEM Total Processes is now: WARNING on incubator-bot2 i-00000252 output: PROCS WARNING: 156 processes [07:25:37] PROBLEM Current Load is now: WARNING on precise-test i-00000231 output: WARNING - load average: 6.79, 7.66, 7.84 [07:25:37] RECOVERY Current Users is now: OK on precise-test i-00000231 output: USERS OK - 0 users currently logged in [07:25:37] RECOVERY Disk Space is now: OK on precise-test i-00000231 output: DISK OK [07:25:42] RECOVERY Total Processes is now: OK on precise-test i-00000231 output: PROCS OK: 115 processes [07:25:47] RECOVERY Free ram is now: OK on precise-test i-00000231 output: OK: 72% free memory [07:25:47] RECOVERY dpkg-check is now: OK on precise-test i-00000231 output: All packages OK [07:25:47] PROBLEM Current Load is now: WARNING on worker1 i-00000208 output: WARNING - load average: 5.95, 6.84, 7.36 [07:25:52] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [07:25:52] RECOVERY Disk Space is now: OK on worker1 i-00000208 output: DISK OK [07:25:52] RECOVERY Free ram is now: OK on worker1 i-00000208 output: OK: 92% free memory [07:25:52] RECOVERY Total Processes is now: OK on worker1 i-00000208 output: PROCS OK: 90 processes [07:28:15] PROBLEM Current Load is now: WARNING on gluster-4 i-000002e4 output: WARNING - load average: 9.11, 8.22, 7.22 [07:28:15] PROBLEM Current Load is now: WARNING on labs-nfs1 i-0000005d output: WARNING - load average: 8.34, 8.82, 9.00 [07:28:15] PROBLEM Current Load is now: WARNING on mobile-testing i-00000271 output: WARNING - load average: 9.69, 12.19, 13.61 [07:28:25] PROBLEM Free ram is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [07:28:30] PROBLEM Current Load is now: CRITICAL on grail i-000002c6 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:29:36] PROBLEM Current Load is now: CRITICAL on integration-apache1 i-000002eb output: CRITICAL - load average: 10.12, 23.84, 29.67 [07:29:36] PROBLEM Current Load is now: WARNING on upload-wizard i-0000021c output: WARNING - load average: 5.04, 7.01, 8.03 [07:30:04] RECOVERY Total Processes is now: OK on integration-apache1 i-000002eb output: PROCS OK: 104 processes [07:32:34] PROBLEM Current Load is now: WARNING on pybal-precise i-00000289 output: WARNING - load average: 5.32, 5.95, 5.99 [07:32:34] PROBLEM Current Users is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:34] PROBLEM SSH is now: CRITICAL on bots-sql2 i-000000af output: CRITICAL - Socket timeout after 10 seconds [07:32:34] PROBLEM dpkg-check is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:34] PROBLEM Total Processes is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:39] PROBLEM Current Load is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:39] PROBLEM Current Users is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:39] PROBLEM Disk Space is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:39] PROBLEM Free ram is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:39] PROBLEM dpkg-check is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:39] PROBLEM Current Users is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:39] PROBLEM Disk Space is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:40] PROBLEM Free ram is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:40] PROBLEM Total Processes is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:44] PROBLEM dpkg-check is now: CRITICAL on pybal-precise i-00000289 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:44] PROBLEM Current Load is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:32:50] PROBLEM Free ram is now: CRITICAL on psm-precise i-000002f2 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:33:34] PROBLEM Current Load is now: WARNING on ee-prototype i-0000013d output: WARNING - load average: 4.22, 5.26, 5.83 [07:33:34] PROBLEM Current Load is now: WARNING on bots-2 i-0000009c output: WARNING - load average: 4.12, 4.80, 5.81 [07:33:34] PROBLEM Current Load is now: CRITICAL on bots-cb i-0000009e output: CRITICAL - load average: 5.71, 7.80, 21.17 [07:33:34] PROBLEM Current Load is now: WARNING on hugglewiki i-000000aa output: WARNING - load average: 7.43, 6.73, 7.21 [07:34:00] PROBLEM Current Load is now: WARNING on grail i-000002c6 output: WARNING - load average: 1.16, 4.66, 6.10 [07:34:00] PROBLEM Current Load is now: WARNING on wep i-000000c2 output: WARNING - load average: 5.05, 5.94, 5.92 [07:34:00] PROBLEM Current Load is now: WARNING on kripke i-00000268 output: WARNING - load average: 7.43, 6.90, 6.69 [07:34:00] PROBLEM Current Load is now: WARNING on ganglia-test2 i-00000250 output: WARNING - load average: 6.00, 6.77, 6.24 [07:34:00] PROBLEM Current Load is now: WARNING on swift-fe1 i-000001d2 output: WARNING - load average: 4.36, 5.20, 5.62 [07:34:00] PROBLEM Current Load is now: WARNING on bots-apache1 i-000000b0 output: WARNING - load average: 7.05, 5.86, 7.60 [07:34:36] PROBLEM Current Load is now: WARNING on redis1 i-000002b6 output: WARNING - load average: 5.49, 5.71, 6.20 [07:34:36] PROBLEM Disk Space is now: WARNING on ipv6test1 i-00000282 output: DISK WARNING - free space: / 69 MB (5% inode=57%): [07:34:36] PROBLEM Current Load is now: WARNING on ipv6test1 i-00000282 output: WARNING - load average: 0.44, 4.35, 5.85 [07:35:07] PROBLEM Free ram is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:08] PROBLEM Free ram is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:18] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:23] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:28] PROBLEM Current Load is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:28] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:35:28] PROBLEM Current Load is now: CRITICAL on configtest-main i-000002dd output: CHECK_NRPE: Socket timeout after 10 seconds. [07:37:56] RECOVERY Current Load is now: OK on pybal-precise i-00000289 output: OK - load average: 1.18, 3.24, 4.74 [07:37:56] RECOVERY Current Users is now: OK on pybal-precise i-00000289 output: USERS OK - 0 users currently logged in [07:37:56] RECOVERY Disk Space is now: OK on pybal-precise i-00000289 output: DISK OK [07:37:56] RECOVERY Free ram is now: OK on pybal-precise i-00000289 output: OK: 78% free memory [07:37:56] RECOVERY Total Processes is now: OK on pybal-precise i-00000289 output: PROCS OK: 83 processes [07:38:01] RECOVERY dpkg-check is now: OK on pybal-precise i-00000289 output: All packages OK [07:38:01] RECOVERY Total Processes is now: OK on gluster-4 i-000002e4 output: PROCS OK: 97 processes [07:38:06] RECOVERY Disk Space is now: OK on grail i-000002c6 output: DISK OK [07:38:07] PROBLEM Current Load is now: WARNING on wikisource-web i-000000fe output: WARNING - load average: 1.32, 3.32, 5.24 [07:38:11] RECOVERY Current Users is now: OK on bots-sql2 i-000000af output: USERS OK - 0 users currently logged in [07:38:12] PROBLEM Current Load is now: WARNING on incubator-bot1 i-00000251 output: WARNING - load average: 2.80, 4.67, 6.66 [07:38:12] PROBLEM Current Load is now: WARNING on psm-precise i-000002f2 output: WARNING - load average: 8.26, 10.87, 10.22 [07:38:12] PROBLEM Current Users is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:12] PROBLEM Total Processes is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:17] PROBLEM Disk Space is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:21] PROBLEM Current Load is now: CRITICAL on mwreview i-000002ae output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:21] PROBLEM Current Load is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:38:41] PROBLEM Current Load is now: WARNING on migration1 i-00000261 output: WARNING - load average: 10.54, 9.87, 8.12 [07:38:55] PROBLEM Total Processes is now: CRITICAL on ganglia-test2 i-00000250 output: PROCS CRITICAL: 205 processes [07:39:00] RECOVERY Current Load is now: OK on ee-prototype i-0000013d output: OK - load average: 0.42, 2.27, 4.37 [07:39:00] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 15.95, 10.67, 18.28 [07:39:05] PROBLEM Free ram is now: UNKNOWN on configtest-main i-000002dd output: NRPE: Unable to read output [07:39:05] RECOVERY Current Load is now: OK on grail i-000002c6 output: OK - load average: 0.24, 1.83, 4.46 [07:39:05] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af output: Warning: 19% free memory [07:39:05] RECOVERY dpkg-check is now: OK on bots-sql2 i-000000af output: All packages OK [07:39:05] RECOVERY Current Load is now: OK on ipv6test1 i-00000282 output: OK - load average: 3.88, 3.14, 4.82 [07:39:05] PROBLEM Current Load is now: WARNING on configtest-main i-000002dd output: WARNING - load average: 8.35, 9.38, 8.07 [07:39:05] PROBLEM Total Processes is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:10] PROBLEM dpkg-check is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:10] PROBLEM Current Load is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:10] PROBLEM Current Users is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:10] PROBLEM Total Processes is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:16] PROBLEM dpkg-check is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:16] PROBLEM Current Load is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:16] PROBLEM Free ram is now: CRITICAL on maps-tilemill1 i-00000294 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:16] PROBLEM Free ram is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:16] PROBLEM Disk Space is now: CRITICAL on deployment-apache31 i-000002d4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:39:47] PROBLEM Current Load is now: CRITICAL on redis1 i-000002b6 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:40:25] pfff [07:40:51] PROBLEM Free ram is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:01] PROBLEM Current Users is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:01] PROBLEM Disk Space is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:01] PROBLEM dpkg-check is now: CRITICAL on gluster-4 i-000002e4 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:01] PROBLEM Total Processes is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:07] PROBLEM Current Load is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:07] PROBLEM Current Users is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:07] PROBLEM Disk Space is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:07] PROBLEM dpkg-check is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:41:07] PROBLEM Total Processes is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:42:01] PROBLEM Total Processes is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [07:42:06] PROBLEM Current Users is now: CRITICAL on integration-apache1 i-000002eb output: CHECK_NRPE: Socket timeout after 10 seconds. [07:42:06]