[00:25:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [01:12:43] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 241 processes [01:22:43] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 98 processes [01:25:42] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 7.21, 8.00, 6.39 [01:45:43] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 4.82, 4.41, 4.97 [02:05:52] PROBLEM Disk Space is now: CRITICAL on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK CRITICAL - free space: / 30 MB (0% inode=72%): [02:25:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [02:47:33] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 149 processes [02:58:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [06:30:33] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 151 processes [06:30:53] PROBLEM Disk Space is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK WARNING - free space: / 482 MB (5% inode=72%): [06:35:33] RECOVERY Total processes is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS OK: 145 processes [07:30:52] RECOVERY Disk Space is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK OK [07:39:33] PROBLEM Total processes is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: PROCS WARNING: 151 processes [10:23:52] PROBLEM Disk Space is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: DISK WARNING - free space: / 573 MB (5% inode=72%): [10:43:42] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 5.70, 6.03, 5.38 [12:47:53] PROBLEM Total processes is now: WARNING on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS WARNING: 179 processes [12:48:53] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 151 processes [12:49:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 156 processes [13:24:53] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 150 processes [13:37:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 156 processes [13:53:53] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 148 processes [14:01:52] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 154 processes [15:29:53] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 160 processes [16:49:53] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 142 processes [17:48:53] PROBLEM Current Load is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - load average: 63.67, 38.35, 20.41 [17:48:53] PROBLEM Free ram is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: Critical: 1% free memory [17:50:12] PROBLEM Total processes is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CHECK_NRPE: Socket timeout after 10 seconds. [17:50:52] PROBLEM Total processes is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS CRITICAL: 434 processes [17:51:42] PROBLEM dpkg-check is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CHECK_NRPE: Socket timeout after 10 seconds. [17:52:02] PROBLEM SSH is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CRITICAL - Socket timeout after 10 seconds [17:52:52] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 151 processes [17:59:23] PROBLEM Free ram is now: CRITICAL on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: Critical: 5% free memory [18:02:53] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 144 processes [18:02:54] RECOVERY Total processes is now: OK on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS OK: 133 processes [18:03:03] PROBLEM Disk Space is now: CRITICAL on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: CHECK_NRPE: Socket timeout after 10 seconds. [18:09:23] RECOVERY Free ram is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: OK: 84% free memory [18:10:52] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 89 processes [18:11:32] RECOVERY dpkg-check is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: All packages OK [18:11:32] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 198 processes [18:11:52] RECOVERY SSH is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [18:12:53] RECOVERY Disk Space is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: DISK OK [18:13:52] RECOVERY Free ram is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK: 88% free memory [18:17:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 171 processes [18:33:43] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 8.09, 7.75, 17.84 [18:57:53] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 150 processes [19:00:52] PROBLEM Total processes is now: WARNING on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS WARNING: 152 processes [19:41:53] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 153 processes [20:21:53] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 150 processes [20:49:52] PROBLEM Total processes is now: WARNING on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS WARNING: 163 processes [21:03:33] PROBLEM Total processes is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: PROCS WARNING: 180 processes [21:04:23] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 26% free memory [21:19:53] RECOVERY Total processes is now: OK on bots-bnr3.pmtpa.wmflabs 10.4.0.81 output: PROCS OK: 147 processes [21:23:42] RECOVERY Current Load is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: OK - load average: 2.28, 2.39, 4.23 [23:00:51] PROBLEM Total processes is now: CRITICAL on bots-bnr4.pmtpa.wmflabs 10.4.0.59 output: PROCS CRITICAL: 248 processes [23:03:01] PROBLEM Total processes is now: CRITICAL on bots-bnr2.pmtpa.wmflabs 10.4.0.40 output: PROCS CRITICAL: 275 processes [23:06:46] PROBLEM Current Load is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: WARNING - load average: 3.40, 6.68, 6.14 [23:20:52] PROBLEM Total processes is now: WARNING on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS WARNING: 167 processes [23:30:53] RECOVERY Total processes is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: PROCS OK: 127 processes