[00:06:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 5.31, 4.29, 4.95 [00:14:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 7.13, 5.98, 5.48 [00:19:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:20:32] PROBLEM host: testing-echo.pmtpa.wmflabs is DOWN address: 10.4.1.93 CRITICAL - Host Unreachable (10.4.1.93) [00:22:32] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [00:22:42] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [00:27:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:27:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [00:28:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:29:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 7.78, 6.97, 5.69 [00:30:12] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 5.65, 5.82, 5.17 [00:32:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.59, 7.09, 5.61 [00:35:13] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 2.50, 4.44, 4.83 [00:46:42] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 22% free memory [00:48:53] PROBLEM Current Load is now: CRITICAL on testing-resize.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [00:49:32] PROBLEM Disk Space is now: CRITICAL on testing-resize.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [00:49:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [00:50:13] PROBLEM Free ram is now: CRITICAL on testing-resize.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [00:51:43] PROBLEM Total processes is now: CRITICAL on testing-resize.pmtpa.wmflabs 10.4.0.42 output: Connection refused or timed out [00:52:30] PROBLEM dpkg-check is now: CRITICAL on testing-resize.pmtpa.wmflabs 10.4.0.42 output: Connection refused or timed out [00:52:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [00:52:43] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [00:53:13] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 6.37, 6.44, 5.64 [00:57:59] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [00:58:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [00:59:02] PROBLEM host: testing-resize.pmtpa.wmflabs is DOWN address: 10.4.0.42 CRITICAL - Host Unreachable (10.4.0.42) [00:59:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:02:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.68, 2.40, 4.21 [01:14:42] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [01:19:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:22:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [01:23:02] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [01:24:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 8.31, 7.74, 5.99 [01:25:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.81, 7.24, 5.67 [01:27:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:28:23] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [01:29:03] PROBLEM host: testing-resize.pmtpa.wmflabs is DOWN address: 10.4.0.42 CRITICAL - Host Unreachable (10.4.0.42) [01:29:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:42:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 9.69, 8.85, 7.63 [01:46:42] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 503 processes [01:47:52] PROBLEM Free ram is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: Warning: 7% free memory [01:50:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [01:51:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 108 processes [01:52:55] RECOVERY Free ram is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: OK: 88% free memory [01:53:13] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 3.47, 4.05, 4.96 [01:55:32] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [01:55:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [01:58:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [01:59:32] PROBLEM host: testing-resize.pmtpa.wmflabs is DOWN address: 10.4.0.42 CRITICAL - Host Unreachable (10.4.0.42) [01:59:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [01:59:52] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 0.57, 1.68, 4.58 [02:00:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:21:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:25:32] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [02:25:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [02:28:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [02:29:32] PROBLEM host: testing-resize.pmtpa.wmflabs is DOWN address: 10.4.0.42 CRITICAL - Host Unreachable (10.4.0.42) [02:29:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [02:30:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:43:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 6.80, 6.90, 6.44 [02:52:38] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [02:54:54] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 0.33, 3.00, 4.99 [02:55:32] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [02:55:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [02:58:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [02:59:32] PROBLEM host: testing-resize.pmtpa.wmflabs is DOWN address: 10.4.0.42 CRITICAL - Host Unreachable (10.4.0.42) [02:59:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:00:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:23:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:25:32] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [03:25:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [03:28:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [03:28:52] PROBLEM Current Load is now: CRITICAL on testing-resize2.pmtpa.wmflabs 10.4.1.34 output: Connection refused by host [03:29:32] PROBLEM Disk Space is now: CRITICAL on testing-resize2.pmtpa.wmflabs 10.4.1.34 output: Connection refused by host [03:29:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [03:30:12] PROBLEM Free ram is now: CRITICAL on testing-resize2.pmtpa.wmflabs 10.4.1.34 output: Connection refused by host [03:31:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:31:42] PROBLEM Total processes is now: CRITICAL on testing-resize2.pmtpa.wmflabs 10.4.1.34 output: Connection refused by host [03:32:32] PROBLEM dpkg-check is now: CRITICAL on testing-resize2.pmtpa.wmflabs 10.4.1.34 output: CHECK_NRPE: Socket timeout after 10 seconds. [03:39:03] PROBLEM host: testing-resize2.pmtpa.wmflabs is DOWN address: 10.4.1.34 CRITICAL - Host Unreachable (10.4.1.34) [03:51:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 3.39, 6.01, 5.39 [03:53:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [03:54:54] PROBLEM host: testing-resize3.pmtpa.wmflabs is DOWN address: 10.4.0.42 CRITICAL - Host Unreachable (10.4.0.42) [03:55:34] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [03:55:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [03:58:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [03:59:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:01:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:11:23] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 10.88, 9.51, 6.59 [04:15:12] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 11.22, 9.64, 7.10 [04:15:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 12.00, 11.03, 7.59 [04:17:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 11.71, 10.76, 7.56 [04:18:52] PROBLEM Current Load is now: CRITICAL on testing-resize4.pmtpa.wmflabs 10.4.1.34 output: Connection refused by host [04:19:32] PROBLEM Disk Space is now: CRITICAL on testing-resize4.pmtpa.wmflabs 10.4.1.34 output: Connection refused by host [04:20:13] PROBLEM Free ram is now: CRITICAL on testing-resize4.pmtpa.wmflabs 10.4.1.34 output: Connection refused by host [04:21:43] PROBLEM Total processes is now: CRITICAL on testing-resize4.pmtpa.wmflabs 10.4.1.34 output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:22:33] PROBLEM dpkg-check is now: CRITICAL on testing-resize4.pmtpa.wmflabs 10.4.1.34 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:24:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:25:52] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [04:26:22] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [04:29:02] PROBLEM host: testing-resize4.pmtpa.wmflabs is DOWN address: 10.4.1.34 CRITICAL - Host Unreachable (10.4.1.34) [04:29:22] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [04:29:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [04:31:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:39:07] RECOVERY host: testing-resize4.pmtpa.wmflabs is UP address: 10.4.1.34 PING OK - Packet loss = 0%, RTA = 0.71 ms [04:39:47] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 23% free memory [04:54:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [04:55:56] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [04:56:26] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [05:00:12] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [05:01:02] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:02:42] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [05:03:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:17:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [05:24:36] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:27:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [05:27:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [05:30:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [05:31:04] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [05:34:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:42:42] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 21% free memory [05:52:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [05:54:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [05:57:15] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [05:57:35] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [06:02:13] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [06:02:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:04:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:20:43] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [06:27:16] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [06:27:36] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [06:28:56] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [06:32:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [06:32:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [06:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:57:45] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [06:57:45] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [06:59:25] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:02:24] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:02:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [07:04:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:27:45] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [07:27:45] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [07:29:25] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:32:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [07:33:23] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [07:35:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:45:25] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 276 processes [07:57:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [07:58:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [07:59:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:02:42] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 211 processes [08:03:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [08:04:42] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:05:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:15:44] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 21% free memory [08:27:56] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [08:29:36] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:29:46] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [08:33:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [08:33:44] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [08:34:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [08:36:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:38:44] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 23% free memory [08:57:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [08:59:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:59:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [09:00:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [09:04:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:05:55] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [09:06:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:22:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 187 processes [09:29:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:30:34] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [09:30:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [09:31:44] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [09:36:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [09:37:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [09:38:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:59:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:00:52] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [10:01:22] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [10:07:26] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:07:46] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [10:08:26] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:29:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:31:01] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [10:31:23] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [10:32:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [10:37:26] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [10:37:46] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [10:37:46] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [10:38:26] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:46:43] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 21% free memory [10:59:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:00:53] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [11:01:23] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [11:07:27] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:07:47] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [11:08:27] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:19:46] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [11:30:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [11:31:24] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [11:33:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:37:27] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [11:37:47] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [11:38:27] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:00:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [12:01:24] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [12:03:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:07:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:08:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [12:09:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:25:24] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 229 processes [12:27:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [12:30:24] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 151 processes [12:30:54] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [12:31:24] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [12:33:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:35:24] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 99 processes [12:37:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [12:38:24] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [12:39:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:39:44] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 23% free memory [12:52:42] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 181 processes [13:02:13] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [13:02:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [13:04:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:07:45] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [13:07:55] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:08:25] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [13:09:05] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:32:13] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [13:32:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [13:32:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 211 processes [13:34:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:37:55] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [13:38:25] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [13:39:05] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:42:44] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 21% free memory [14:02:13] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [14:02:33] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [14:04:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:08:34] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [14:09:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:09:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:25:44] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [14:32:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [14:32:34] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [14:34:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:38:37] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [14:39:47] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [14:39:57] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:57:24] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 209 processes [15:02:22] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 184 processes [15:02:42] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [15:02:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [15:04:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:07:27] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 218 processes [15:08:37] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [15:09:47] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:09:57] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:32:42] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [15:32:52] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [15:34:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:40:13] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [15:40:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:41:13] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [15:51:26] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 1.54, 1.79, 4.33 [16:02:43] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [16:02:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 0.17, 1.08, 4.91 [16:04:13] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [16:04:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:05:45] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 1.18, 1.43, 4.31 [16:05:55] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: OK - load average: 0.67, 0.91, 4.53 [16:06:45] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.31, 1.54, 4.07 [16:10:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:10:54] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [16:11:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [16:15:45] RECOVERY Disk Space is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK OK [16:15:45] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 21% free memory [16:32:44] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [16:34:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:35:25] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [16:41:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:42:13] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [16:42:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:02:53] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [17:05:58] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [17:08:48] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [17:08:58] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:12:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:12:42] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [17:12:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:33:02] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [17:33:58] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 13.17, 12.16, 7.77 [17:34:22] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 4.79, 7.96, 5.68 [17:34:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 9.82, 8.82, 5.85 [17:35:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 11.92, 10.78, 6.85 [17:36:14] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [17:38:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.92, 10.32, 7.22 [17:39:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:42:43] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [17:42:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [17:43:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:05:33] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [18:06:23] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [18:06:24] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 194 processes [18:09:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:12:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [18:12:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:14:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:16:24] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 103 processes [18:35:35] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [18:36:25] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [18:39:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:42:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [18:42:54] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [18:44:04] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:50:42] PROBLEM Free ram is now: WARNING on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Warning: 19% free memory [19:05:35] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [19:06:25] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [19:09:45] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:13:23] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [19:13:43] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 21% free memory [19:14:43] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:14:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:16:24] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 162 processes [19:35:36] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [19:36:26] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [19:39:46] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:41:42] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [19:43:32] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [19:44:52] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [19:45:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:05:55] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [20:06:55] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [20:08:45] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 2.10, 2.25, 4.53 [20:10:35] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:10:55] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 2.18, 2.34, 4.62 [20:13:41] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [20:15:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:16:14] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:22:43] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 202 processes [20:35:56] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [20:37:36] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [20:41:25] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:41:45] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 23% free memory [20:43:35] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [20:44:25] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 0.08, 1.26, 5.00 [20:46:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:47:23] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [20:53:54] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: OK - load average: 0.29, 2.56, 4.17 [20:54:54] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.15, 2.97, 4.49 [21:07:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [21:07:44] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [21:09:44] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [21:12:22] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:15:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [21:16:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:17:24] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:37:14] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [21:37:45] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [21:43:24] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:44:44] RECOVERY Free ram is now: OK on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: OK: 21% free memory [21:45:53] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [21:47:53] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [21:48:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:07:16] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [22:07:46] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [22:13:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:15:55] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [22:17:55] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [22:19:05] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:22:44] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 217 processes [22:23:54] PROBLEM Current Load is now: CRITICAL on testing-yp.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [22:24:34] PROBLEM Disk Space is now: CRITICAL on testing-yp.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [22:25:17] PROBLEM Free ram is now: CRITICAL on testing-yp.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [22:26:44] PROBLEM Total processes is now: CRITICAL on testing-yp.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [22:27:24] PROBLEM dpkg-check is now: CRITICAL on testing-yp.pmtpa.wmflabs 10.4.0.42 output: Connection refused by host [22:27:44] PROBLEM Free ram is now: WARNING on deployment-sql.pmtpa.wmflabs 10.4.0.53 output: Warning: 19% free memory [22:37:45] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [22:40:53] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [22:42:43] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 196 processes [22:44:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:47:44] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [22:49:44] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [22:49:54] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:07:46] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [23:10:53] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [23:14:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [23:17:46] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 232 processes [23:17:46] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [23:19:46] PROBLEM host: vumi-metrics.pmtpa.wmflabs is DOWN address: 10.4.1.13 CRITICAL - Host Unreachable (10.4.1.13) [23:19:56] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:37:46] PROBLEM host: gerrit-stream.pmtpa.wmflabs is DOWN address: 10.4.1.31 CRITICAL - Host Unreachable (10.4.1.31) [23:40:54] PROBLEM host: catsort.pmtpa.wmflabs is DOWN address: 10.4.1.11 CRITICAL - Host Unreachable (10.4.1.11) [23:44:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [23:48:25] PROBLEM host: basictestsugar.pmtpa.wmflabs is DOWN address: 10.4.1.10 CRITICAL - Host Unreachable (10.4.1.10) [23:50:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [23:51:05]