[00:04:15] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:04:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [00:06:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:09:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:14:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:19:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:19:22] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [00:24:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:24:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:26:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 151 processes [00:34:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:37:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:37:52] RECOVERY Free ram is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK: 21% free memory [00:40:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [00:41:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:41:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [00:45:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:49:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:53:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [00:54:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:54:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:59:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [01:04:35] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [01:07:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [01:07:43] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 144 processes [01:09:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 152 processes [01:12:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:15:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:19:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:24:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:24:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:34:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [01:39:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [01:40:44] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 179 processes [01:42:04] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:46:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:47:22] PROBLEM Free ram is now: CRITICAL on sf-book-server.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [01:47:32] PROBLEM Disk Space is now: CRITICAL on sf-book-server.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [01:47:52] PROBLEM Current Load is now: CRITICAL on sf-book-server.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [01:48:42] PROBLEM Total processes is now: CRITICAL on sf-book-server.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [01:49:22] PROBLEM dpkg-check is now: CRITICAL on sf-book-server.pmtpa.wmflabs 10.4.1.87 output: Connection refused by host [01:49:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:54:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:54:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:07:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:09:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:12:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:16:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:19:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:24:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:24:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:34:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [02:38:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:39:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:42:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:44:22] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 4% free memory [02:46:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:49:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:54:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:54:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:08:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:09:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [03:12:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:15:43] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 150 processes [03:16:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:19:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:24:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:25:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:38:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:39:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [03:42:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:48:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:51:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:54:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:55:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:01:43] PROBLEM Free ram is now: UNKNOWN on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Unknown [04:08:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:09:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [04:11:43] PROBLEM Free ram is now: WARNING on aggregator2.pmtpa.wmflabs 10.4.0.193 output: Warning: 9% free memory [04:12:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:19:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:22:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:24:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:26:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:38:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:38:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [04:39:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [04:39:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [04:42:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:46:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [04:49:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:52:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:54:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:54:23] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [04:57:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:04:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [05:08:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:09:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:12:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:19:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:22:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:24:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:28:24] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:38:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:39:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:42:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:49:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:53:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:54:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:58:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:59:22] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 4% free memory [06:08:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:09:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [06:13:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:19:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:23:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [06:24:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:28:13] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [06:28:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:28:43] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 154 processes [06:30:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 155 processes [06:31:12] PROBLEM Total processes is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS WARNING: 151 processes [06:31:52] PROBLEM Total processes is now: WARNING on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS WARNING: 155 processes [06:33:42] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 149 processes [06:35:33] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 147 processes [06:38:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:39:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [06:41:12] RECOVERY Total processes is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: PROCS OK: 146 processes [06:43:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:43:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 20% free memory [06:49:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:53:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [06:54:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:56:53] RECOVERY Total processes is now: OK on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS OK: 150 processes [06:58:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:08:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:09:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:13:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:19:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:23:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:24:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:28:45] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:34:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [07:38:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:39:23] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 5% free memory [07:39:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:43:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:46:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [07:49:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:53:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:55:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:58:52] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:09:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:11:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [08:13:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:19:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:23:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:26:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:29:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:39:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:39:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [08:39:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [08:41:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [08:41:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [08:43:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:49:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:53:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:56:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:59:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:04:23] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 4% free memory [09:07:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [09:09:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [09:09:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [09:12:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:14:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:19:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:23:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [09:27:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [09:30:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:39:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [09:42:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:44:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:49:33] PROBLEM Total processes is now: WARNING on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS WARNING: 182 processes [09:49:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:51:42] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 165 processes [09:54:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [09:57:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [10:00:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:04:33] RECOVERY Total processes is now: OK on bots-salebot.pmtpa.wmflabs 10.4.0.163 output: PROCS OK: 103 processes [10:06:42] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 143 processes [10:09:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [10:12:24] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:14:34] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:20:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:24:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:29:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [10:30:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:39:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [10:42:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:45:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:50:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:54:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:59:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:00:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:09:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:13:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [11:16:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:20:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:24:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:29:17] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:30:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:37:52] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 5% free memory [11:39:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:44:26] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [11:44:46] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 153 processes [11:47:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:47:13] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [11:51:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:54:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:59:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:01:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:01:52] PROBLEM Total processes is now: WARNING on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS WARNING: 151 processes [12:02:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 6% free memory [12:04:42] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 148 processes [12:06:53] RECOVERY Total processes is now: OK on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS OK: 150 processes [12:09:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:14:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:17:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:21:37] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [12:24:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:29:28] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:31:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:37:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 28% free memory [12:37:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [12:39:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 7% free memory [12:39:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:39:52] PROBLEM Free ram is now: CRITICAL on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Critical: 3% free memory [12:42:42] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 154 processes [12:44:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 18% free memory [12:46:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:47:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:50:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [12:54:02] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [12:57:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:59:23] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 4% free memory [12:59:43] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [13:03:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:05:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [13:09:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:16:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:17:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:24:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [13:27:47] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:30:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 159 processes [13:30:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [13:33:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:36:52] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [13:43:05] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:46:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:47:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:47:52] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 5% free memory [13:54:03] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [13:57:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:57:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 7% free memory [14:01:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:03:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:13:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:16:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:17:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:21:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [14:24:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:27:52] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [14:27:52] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 5% free memory [14:30:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [14:31:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:33:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:44:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:46:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 20% free memory [14:46:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:47:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:53:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [14:54:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:54:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [14:58:34] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:01:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:03:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:14:26] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:16:32] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:16:42] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 163 processes [15:17:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:24:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:28:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:31:42] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 143 processes [15:32:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:33:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:39:43] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 161 processes [15:44:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:46:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:47:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:54:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:58:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:02:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:03:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:14:55] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:16:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [16:16:53] PROBLEM Total processes is now: WARNING on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS WARNING: 151 processes [16:17:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [16:19:13] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [16:21:52] RECOVERY Total processes is now: OK on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS OK: 150 processes [16:24:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:28:56] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:34:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:34:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:38:13] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [16:38:23] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [16:39:23] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [16:47:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [16:48:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:48:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [16:54:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:54:23] PROBLEM Free ram is now: CRITICAL on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Critical: 4% free memory [16:59:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:04:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:04:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:06:32] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [17:07:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 6% free memory [17:15:32] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 149 processes [17:17:29] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:18:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:18:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:19:52] PROBLEM Total processes is now: WARNING on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS WARNING: 151 processes [17:24:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:24:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [17:29:17] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:29:37] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 156 processes [17:32:53] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 5% free memory [17:34:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:34:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:47:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:48:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:48:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:54:22] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:59:17] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:04:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:05:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:09:43] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 144 processes [18:17:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:17:43] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 158 processes [18:18:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:18:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:24:32] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 150 processes [18:24:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:27:42] RECOVERY Total processes is now: OK on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS OK: 138 processes [18:29:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:34:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:35:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:45:13] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [18:47:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:48:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:48:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:49:23] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [18:50:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 25% free memory [18:54:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:57:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 7% free memory [18:59:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:02:53] PROBLEM Free ram is now: CRITICAL on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Critical: 5% free memory [19:04:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:06:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:18:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:18:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [19:18:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:19:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [19:24:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:27:53] PROBLEM Total processes is now: WARNING on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS WARNING: 151 processes [19:29:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [19:29:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:32:52] RECOVERY Total processes is now: OK on babbage0.pmtpa.wmflabs 10.4.1.51 output: PROCS OK: 150 processes [19:34:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:36:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:48:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:49:13] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:49:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [19:52:42] PROBLEM Total processes is now: WARNING on bots-bsql01.pmtpa.wmflabs 10.4.0.226 output: PROCS WARNING: 161 processes [19:55:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:59:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [20:02:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [20:04:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [20:08:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:19:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [20:19:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [20:19:22] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 154 processes [20:21:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [20:26:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [20:32:54] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [20:36:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [20:37:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [20:38:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 28% free memory [20:38:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:39:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 6% free memory [20:39:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [20:40:24]