[00:42:53] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [01:12:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 440 processes [01:18:23] PROBLEM Free ram is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: Critical: 5% free memory [01:22:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 97 processes [01:23:23] RECOVERY Free ram is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: OK: 84% free memory [03:42:53] RECOVERY Free ram is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: OK: 27% free memory [03:44:53] PROBLEM Total processes is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: PROCS CRITICAL: 201 processes [04:44:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to fork() failed [04:49:52] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [05:57:43] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 184 processes [06:02:42] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 99 processes [06:37:43] PROBLEM dpkg-check is now: CRITICAL on kubo.pmtpa.wmflabs 10.4.0.19 output: DPKG CRITICAL dpkg reports broken packages [06:38:53] PROBLEM dpkg-check is now: CRITICAL on puppetdoc2.pmtpa.wmflabs 10.4.1.29 output: DPKG CRITICAL dpkg reports broken packages [06:40:52] PROBLEM dpkg-check is now: CRITICAL on staging.pmtpa.wmflabs 10.4.1.24 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:42:22] PROBLEM dpkg-check is now: CRITICAL on build-lucid1.pmtpa.wmflabs 10.4.1.53 output: DPKG CRITICAL dpkg reports broken packages [06:42:42] RECOVERY dpkg-check is now: OK on kubo.pmtpa.wmflabs 10.4.0.19 output: All packages OK [06:43:52] RECOVERY dpkg-check is now: OK on puppetdoc2.pmtpa.wmflabs 10.4.1.29 output: All packages OK [06:46:06] PROBLEM dpkg-check is now: CRITICAL on build-precise1.pmtpa.wmflabs 10.4.0.173 output: DPKG CRITICAL dpkg reports broken packages [06:46:56] PROBLEM dpkg-check is now: CRITICAL on cvn-apache2.pmtpa.wmflabs 10.4.0.204 output: DPKG CRITICAL dpkg reports broken packages [06:47:16] PROBLEM dpkg-check is now: CRITICAL on integration-apache1.pmtpa.wmflabs 10.4.0.102 output: DPKG CRITICAL dpkg reports broken packages [06:47:16] RECOVERY dpkg-check is now: OK on build-lucid1.pmtpa.wmflabs 10.4.1.53 output: All packages OK [06:47:26] PROBLEM dpkg-check is now: CRITICAL on dumps-1.pmtpa.wmflabs 10.4.0.218 output: DPKG CRITICAL dpkg reports broken packages [06:48:06] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: DPKG CRITICAL dpkg reports broken packages [06:48:56] PROBLEM dpkg-check is now: CRITICAL on mediawiki-dev-1.pmtpa.wmflabs 10.4.0.89 output: DPKG CRITICAL dpkg reports broken packages [06:49:06] PROBLEM dpkg-check is now: CRITICAL on wikiversity-sandbox-test.pmtpa.wmflabs 10.4.0.46 output: DPKG CRITICAL dpkg reports broken packages [06:50:43] RECOVERY dpkg-check is now: OK on staging.pmtpa.wmflabs 10.4.1.24 output: All packages OK [06:51:03] PROBLEM dpkg-check is now: CRITICAL on wep2.pmtpa.wmflabs 10.4.0.57 output: DPKG CRITICAL dpkg reports broken packages [06:51:03] RECOVERY dpkg-check is now: OK on build-precise1.pmtpa.wmflabs 10.4.0.173 output: All packages OK [06:51:13] PROBLEM dpkg-check is now: CRITICAL on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: DPKG CRITICAL dpkg reports broken packages [06:51:53] PROBLEM dpkg-check is now: CRITICAL on follow01d.pmtpa.wmflabs 10.4.1.40 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:51:53] RECOVERY dpkg-check is now: OK on cvn-apache2.pmtpa.wmflabs 10.4.0.204 output: All packages OK [06:52:03] RECOVERY dpkg-check is now: OK on integration-apache1.pmtpa.wmflabs 10.4.0.102 output: All packages OK [06:52:23] PROBLEM dpkg-check is now: CRITICAL on deployment-jobrunner08.pmtpa.wmflabs 10.4.1.30 output: DPKG CRITICAL dpkg reports broken packages [06:52:23] RECOVERY dpkg-check is now: OK on dumps-1.pmtpa.wmflabs 10.4.0.218 output: All packages OK [06:52:33] PROBLEM Total processes is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:52:53] PROBLEM dpkg-check is now: CRITICAL on bots-analytics.pmtpa.wmflabs 10.4.1.44 output: DPKG CRITICAL dpkg reports broken packages [06:53:03] PROBLEM dpkg-check is now: CRITICAL on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: DPKG CRITICAL dpkg reports broken packages [06:53:03] RECOVERY dpkg-check is now: OK on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: All packages OK [06:53:53] PROBLEM dpkg-check is now: CRITICAL on cvresearch-bots.pmtpa.wmflabs 10.4.1.17 output: DPKG CRITICAL dpkg reports broken packages [06:54:03] RECOVERY dpkg-check is now: OK on mediawiki-dev-1.pmtpa.wmflabs 10.4.0.89 output: All packages OK [06:54:23] PROBLEM dpkg-check is now: CRITICAL on xn--tekruo-t1a.pmtpa.wmflabs 10.4.1.60 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:33] PROBLEM dpkg-check is now: CRITICAL on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: DPKG CRITICAL dpkg reports broken packages [06:55:53] RECOVERY dpkg-check is now: OK on wep2.pmtpa.wmflabs 10.4.0.57 output: All packages OK [06:55:53] PROBLEM dpkg-check is now: CRITICAL on deployment-cache-mobile01.pmtpa.wmflabs 10.4.1.82 output: DPKG CRITICAL dpkg reports broken packages [06:56:13] RECOVERY dpkg-check is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: All packages OK [06:56:23] PROBLEM dpkg-check is now: CRITICAL on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: DPKG CRITICAL dpkg reports broken packages [06:56:33] PROBLEM dpkg-check is now: CRITICAL on rocsteady-cleanup.pmtpa.wmflabs 10.4.0.206 output: DPKG CRITICAL dpkg reports broken packages [06:56:43] RECOVERY dpkg-check is now: OK on follow01d.pmtpa.wmflabs 10.4.1.40 output: All packages OK [06:57:13] PROBLEM dpkg-check is now: CRITICAL on echo-xmpp.pmtpa.wmflabs 10.4.0.146 output: DPKG CRITICAL dpkg reports broken packages [06:57:23] RECOVERY dpkg-check is now: OK on deployment-jobrunner08.pmtpa.wmflabs 10.4.1.30 output: All packages OK [06:57:33] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 166 processes [06:57:53] RECOVERY dpkg-check is now: OK on bots-analytics.pmtpa.wmflabs 10.4.1.44 output: All packages OK [06:58:03] RECOVERY dpkg-check is now: OK on deployment-bastion.pmtpa.wmflabs 10.4.0.58 output: All packages OK [06:58:53] RECOVERY dpkg-check is now: OK on cvresearch-bots.pmtpa.wmflabs 10.4.1.17 output: All packages OK [06:59:03] RECOVERY dpkg-check is now: OK on wikiversity-sandbox-test.pmtpa.wmflabs 10.4.0.46 output: All packages OK [06:59:13] RECOVERY dpkg-check is now: OK on xn--tekruo-t1a.pmtpa.wmflabs 10.4.1.60 output: All packages OK [07:00:32] RECOVERY dpkg-check is now: OK on mobile-osm2.pmtpa.wmflabs 10.4.1.67 output: All packages OK [07:00:52] RECOVERY dpkg-check is now: OK on deployment-cache-mobile01.pmtpa.wmflabs 10.4.1.82 output: All packages OK [07:01:22] RECOVERY dpkg-check is now: OK on testlabs-abogott-dev.pmtpa.wmflabs 10.4.1.20 output: All packages OK [07:01:32] RECOVERY dpkg-check is now: OK on rocsteady-cleanup.pmtpa.wmflabs 10.4.0.206 output: All packages OK [07:02:12] RECOVERY dpkg-check is now: OK on echo-xmpp.pmtpa.wmflabs 10.4.0.146 output: All packages OK [10:13:43] PROBLEM Total processes is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS CRITICAL: 212 processes [10:18:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 128 processes [10:46:43] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 169 processes [11:35:23] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [11:40:23] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 6% free memory [12:05:24] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [14:20:23] PROBLEM Free ram is now: CRITICAL on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Critical: 5% free memory [14:45:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [14:50:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [14:53:43] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 184 processes [15:20:23] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 16% free memory [15:50:53] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 19% free memory [16:05:53] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [16:25:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 10% free memory [19:46:43] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 183 processes [19:51:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 108 processes [22:00:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [22:09:43] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 187 processes [22:14:43] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 98 processes [22:30:42] PROBLEM Free ram is now: CRITICAL on tools-exec-02.pmtpa.wmflabs 10.4.1.65 output: CHECK_NRPE: Socket timeout after 10 seconds. [22:35:33] PROBLEM Free ram is now: UNKNOWN on tools-exec-02.pmtpa.wmflabs 10.4.1.65 output: NRPE: Unable to read output [22:39:53] PROBLEM host: testing-lucidimage1.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [22:45:32] PROBLEM Total processes is now: CRITICAL on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS CRITICAL: 201 processes [23:14:33] PROBLEM Disk Space is now: CRITICAL on testing-lucidimage2.pmtpa.wmflabs 10.4.0.27 output: Connection refused or timed out [23:15:33] PROBLEM Total processes is now: WARNING on aggregator-test1.pmtpa.wmflabs 10.4.0.192 output: PROCS WARNING: 199 processes [23:19:43] PROBLEM Total processes is now: CRITICAL on testing-lucidimage2.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [23:20:12] PROBLEM Free ram is now: CRITICAL on testing-lucidimage2.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [23:20:22] PROBLEM dpkg-check is now: CRITICAL on testing-lucidimage2.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [23:20:52] PROBLEM Free ram is now: CRITICAL on aggregator2.pmtpa.wmflabs 10.4.0.193 output: NRPE: Call to popen() failed [23:20:52] PROBLEM Current Load is now: CRITICAL on testing-lucidimage2.pmtpa.wmflabs 10.4.0.27 output: Connection refused by host [23:31:33] PROBLEM Disk Space is now: CRITICAL on testing-lucidimage3.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [23:31:53] PROBLEM Current Load is now: CRITICAL on testing-lucidimage3.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [23:32:13] PROBLEM Free ram is now: CRITICAL on testing-lucidimage3.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [23:33:53] PROBLEM Total processes is now: CRITICAL on testing-lucidimage3.pmtpa.wmflabs 10.4.0.119 output: CHECK_NRPE: Socket timeout after 10 seconds. [23:34:23] PROBLEM dpkg-check is now: CRITICAL on testing-lucidimage3.pmtpa.wmflabs 10.4.0.119 output: Connection refused by host [23:35:53] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [23:48:52] PROBLEM Current Load is now: CRITICAL on testing-lucidimage4.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [23:49:32] PROBLEM Disk Space is now: CRITICAL on testing-lucidimage4.pmtpa.wmflabs 10.4.0.142 output: Connection refused by host [23:50:13] PROBLEM Free ram is now: CRITICAL on testing-lucidimage4.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:51:43] PROBLEM Total processes is now: CRITICAL on testing-lucidimage4.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake. [23:52:23] PROBLEM dpkg-check is now: CRITICAL on testing-lucidimage4.pmtpa.wmflabs 10.4.0.142 output: CHECK_NRPE: Error - Could not complete SSL handshake.