[00:14:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [00:20:23] PROBLEM Free ram is now: CRITICAL on rt-puppetdev7.pmtpa.wmflabs 10.4.0.195 output: Critical: 4% free memory [00:35:22] PROBLEM Free ram is now: WARNING on rt-puppetdev7.pmtpa.wmflabs 10.4.0.195 output: Warning: 8% free memory [00:38:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [00:39:32] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 24% free memory [00:39:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 26% free memory [00:40:13] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [00:41:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [00:45:22] RECOVERY Free ram is now: OK on rt-puppetdev7.pmtpa.wmflabs 10.4.0.195 output: OK: 74% free memory [00:47:22] PROBLEM Free ram is now: CRITICAL on bots-cb.pmtpa.wmflabs 10.4.0.44 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:48:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [00:49:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [00:52:13] RECOVERY Free ram is now: OK on bots-cb.pmtpa.wmflabs 10.4.0.44 output: OK: 46% free memory [00:56:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [01:02:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 17% free memory [01:14:52] PROBLEM Free ram is now: CRITICAL on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Critical: 3% free memory [01:17:33] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [01:57:23] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [01:59:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [02:10:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [02:27:23] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 150 processes [04:05:33] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 20% free memory [04:13:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [04:28:32] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 20% free memory [04:37:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 26% free memory [04:38:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [04:39:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [04:41:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [04:46:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [04:52:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 14% free memory [04:54:53] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 17% free memory [05:10:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 18% free memory [06:27:34] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [06:29:24] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 154 processes [06:29:54] PROBLEM Total processes is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS WARNING: 151 processes [06:32:32] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 20% free memory [06:34:22] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [06:34:52] RECOVERY Total processes is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS OK: 146 processes [07:05:33] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [08:37:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [08:39:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [08:40:32] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 22% free memory [08:40:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 24% free memory [08:41:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [08:48:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 15% free memory [08:49:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [09:02:42] PROBLEM Free ram is now: CRITICAL on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: Critical: 4% free memory [09:03:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [09:07:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [09:10:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [10:19:02] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: Warning: 13% free memory [10:24:03] RECOVERY Free ram is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK: 49% free memory [11:51:22] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 16% free memory [12:37:32] PROBLEM Free ram is now: WARNING on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: Warning: 7% free memory [12:37:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [12:38:32] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 24% free memory [12:38:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 25% free memory [12:39:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:40:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [12:47:13] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [12:50:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [13:02:32] PROBLEM Free ram is now: CRITICAL on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: Critical: 5% free memory [13:08:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [13:11:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 17% free memory [13:37:32] PROBLEM Free ram is now: WARNING on abogott-request-tracker.pmtpa.wmflabs 10.4.1.48 output: Warning: 8% free memory [13:47:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [15:01:22] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [15:09:22] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [16:37:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [16:37:32] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 24% free memory [16:38:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 27% free memory [16:39:22] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 21% free memory [16:40:53] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [16:41:43] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 25% free memory [16:45:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [16:47:22] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [16:49:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 17% free memory [16:51:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [17:08:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [17:10:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [20:37:22] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [20:38:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 21% free memory [20:39:42] RECOVERY Free ram is now: OK on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: OK: 25% free memory [20:40:32] RECOVERY Free ram is now: OK on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: OK: 24% free memory [20:41:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [20:45:22] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [20:49:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 14% free memory [20:51:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 16% free memory [20:58:32] PROBLEM Free ram is now: WARNING on nova-precise2.pmtpa.wmflabs 10.4.1.57 output: Warning: 19% free memory [21:00:22] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [21:07:42] PROBLEM Free ram is now: WARNING on mediawiki-bugfix-kozuch.pmtpa.wmflabs 10.4.0.26 output: Warning: 17% free memory [21:53:23] PROBLEM Free ram is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: Warning: 19% free memory [22:28:23] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 20% free memory [23:45:42] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 19% free memory