[00:00:12] RECOVERY Free ram is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: OK: 91% free memory [00:01:42] RECOVERY Total processes is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: PROCS OK: 83 processes [00:02:22] RECOVERY dpkg-check is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: All packages OK [00:03:52] RECOVERY Current Load is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.1.54 output: OK - load average: 0.32, 0.66, 0.61 [00:03:53] RECOVERY Disk Space is now: OK on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: DISK OK [00:04:32] RECOVERY Free ram is now: OK on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: OK: 91% free memory [00:05:13] RECOVERY Total processes is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: PROCS OK: 83 processes [00:06:03] RECOVERY Total processes is now: OK on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: PROCS OK: 83 processes [00:06:03] RECOVERY dpkg-check is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: All packages OK [00:06:43] RECOVERY dpkg-check is now: OK on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: All packages OK [00:07:23] RECOVERY Current Load is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: OK - load average: 0.09, 0.70, 0.64 [00:07:53] RECOVERY Current Load is now: OK on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: OK - load average: 0.08, 0.62, 0.66 [00:07:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 2.81, 3.21, 4.88 [00:08:03] RECOVERY Disk Space is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: DISK OK [00:08:53] RECOVERY Free ram is now: OK on tools-webserver-01.pmtpa.wmflabs 10.4.1.85 output: OK: 91% free memory [00:11:32] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.53, 3.10, 4.48 [00:17:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:20:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:20:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:21:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [00:21:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:22:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:28:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [00:37:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [00:39:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [00:40:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [00:40:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 6.30, 6.08, 5.40 [00:47:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [00:50:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [00:50:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [00:51:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [00:51:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [00:52:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [00:52:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [00:53:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 3.04, 3.36, 4.90 [00:53:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 14% free memory [00:55:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 15% free memory [00:58:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:17:46] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:20:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [01:20:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:21:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [01:21:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [01:22:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:28:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [01:34:32] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 151 processes [01:44:32] RECOVERY Total processes is now: OK on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS OK: 149 processes [01:50:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [01:51:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [01:51:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [01:51:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [01:51:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [01:52:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [01:58:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:20:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:21:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:21:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:21:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [02:21:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:22:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:28:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [02:50:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [02:51:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [02:51:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [02:51:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [02:51:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [02:52:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [02:58:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:20:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:21:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:21:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:21:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [03:21:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [03:22:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:28:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [03:39:23] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 8.77, 7.90, 6.19 [03:48:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.74, 9.61, 6.66 [03:50:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [03:51:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [03:51:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [03:51:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [03:51:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 10.46, 9.24, 6.81 [03:51:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [03:52:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [03:58:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:20:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:21:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:21:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:21:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [04:21:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [04:22:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:28:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [04:37:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [04:38:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [04:44:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 356 processes [04:46:33] PROBLEM Current Load is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: WARNING - load average: 14.27, 14.39, 7.37 [04:50:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [04:51:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [04:51:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [04:51:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [04:51:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [04:52:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [04:55:13] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [04:56:33] RECOVERY Current Load is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK - load average: 0.17, 2.28, 4.10 [04:58:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:01:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [05:19:53] PROBLEM Current Load is now: WARNING on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: WARNING - load average: 7.51, 6.59, 5.85 [05:20:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:21:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:21:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:21:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [05:21:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:22:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:25:02] PROBLEM Free ram is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: Warning: 16% free memory [05:28:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:45:02] RECOVERY Free ram is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: OK: 38% free memory [05:50:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [05:51:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [05:51:33] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [05:51:43] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [05:51:43] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: OK - load average: 2.96, 2.95, 4.78 [05:51:53] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [05:52:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [05:54:53] RECOVERY Current Load is now: OK on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: OK - load average: 2.25, 3.77, 4.66 [05:58:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [05:58:52] PROBLEM Current Load is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [05:59:22] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [05:59:32] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 2.36, 2.51, 4.18 [05:59:32] PROBLEM Disk Space is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [06:00:13] PROBLEM Free ram is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [06:01:43] PROBLEM Total processes is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [06:02:23] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: Connection refused by host [06:03:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 2.29, 2.49, 4.42 [06:04:23] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes [06:20:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:21:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:21:33] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:21:43] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [06:21:53] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [06:22:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:28:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [06:29:23] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 191 processes [06:50:52] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [06:51:42] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [06:51:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [06:52:22] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [06:52:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [06:54:02] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [06:55:53] RECOVERY Free ram is now: OK on stackfarm-sql1.pmtpa.wmflabs 10.4.1.8 output: OK: 27% free memory [06:58:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:20:52] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:21:42] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:21:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [07:22:22] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:22:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:24:02] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:29:22] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [07:50:53] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [07:51:43] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [07:51:53] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [07:52:33] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [07:54:03] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [07:54:03] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [07:59:22] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:02:14] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 30% free memory [08:20:12] RECOVERY Free ram is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: OK: 96% free memory [08:21:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:21:42] RECOVERY Total processes is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: PROCS OK: 101 processes [08:22:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:22:22] RECOVERY dpkg-check is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: All packages OK [08:22:32] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [08:22:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:23:52] RECOVERY Current Load is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: OK - load average: 0.03, 0.45, 0.37 [08:24:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:24:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [08:24:32] RECOVERY Disk Space is now: OK on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: DISK OK [08:29:23] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [08:40:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [08:40:52] RECOVERY Free ram is now: OK on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: OK: 20% free memory [08:41:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [08:49:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 14% free memory [08:51:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [08:52:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [08:52:32] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [08:52:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [08:53:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 15% free memory [08:54:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [08:54:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [08:59:23] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [09:03:52] PROBLEM Free ram is now: WARNING on integration-jobbuilder.pmtpa.wmflabs 10.4.0.21 output: Warning: 15% free memory [09:21:22] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:22:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [09:22:32] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [09:22:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:24:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [09:24:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:29:23] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [09:51:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [09:52:33] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [09:52:33] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [09:52:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [09:54:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [09:54:33] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [09:59:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:22:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:22:52] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:23:12] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [10:24:32] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [10:24:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:24:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [10:30:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [10:52:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [10:52:53] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [10:53:53] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [10:54:33] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [10:54:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [10:54:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:01:43] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:22:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:22:53] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:23:53] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:24:33] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [11:24:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [11:24:53] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:32:23] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [11:52:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [11:53:22] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [11:54:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [11:55:42] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [11:55:42] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [11:56:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:03:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:22:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:23:12] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Critical: 3% free memory [12:24:22] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [12:24:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [12:25:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:25:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:26:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:28:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [12:33:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [12:38:12] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [12:39:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [12:52:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [12:54:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [12:55:22] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [12:55:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [12:55:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [12:56:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [12:57:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [13:03:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:03:52] PROBLEM Current Load is now: CRITICAL on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [13:04:32] PROBLEM Disk Space is now: CRITICAL on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [13:05:12] PROBLEM Free ram is now: CRITICAL on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [13:06:42] PROBLEM Total processes is now: CRITICAL on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [13:07:22] PROBLEM dpkg-check is now: CRITICAL on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [13:08:52] RECOVERY Current Load is now: OK on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: OK - load average: 0.73, 0.96, 0.56 [13:09:32] RECOVERY Disk Space is now: OK on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: DISK OK [13:10:12] RECOVERY Free ram is now: OK on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: OK: 89% free memory [13:11:42] RECOVERY Total processes is now: OK on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: PROCS OK: 84 processes [13:12:22] RECOVERY dpkg-check is now: OK on wikidata-test-repo.pmtpa.wmflabs 10.4.0.156 output: All packages OK [13:22:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:24:53] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [13:25:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:25:53] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [13:26:22] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [13:26:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:33:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [13:36:12] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [13:52:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [13:54:53] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [13:55:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [13:55:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [13:56:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [13:57:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:03:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [14:25:32] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:25:52] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:25:52] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:26:02] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [14:26:22] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:27:12] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:33:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [14:55:33] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [14:56:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [14:56:03] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [14:56:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [14:56:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [14:57:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [14:59:23] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 206 processes [15:03:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:04:23] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 188 processes [15:25:52] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:26:32] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [15:26:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:26:42] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:26:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:27:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:33:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [15:55:52] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [15:56:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [15:56:42] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [15:56:42] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [15:56:52] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [15:57:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [15:59:22] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 203 processes [16:02:13] PROBLEM Total processes is now: WARNING on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS WARNING: 155 processes [16:03:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:17:03] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [16:17:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:18:23] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:18:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:18:33] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:19:03] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:19:03] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [16:19:23] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [16:23:32] PROBLEM Current Load is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [16:23:52] PROBLEM Free ram is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [16:24:12] PROBLEM Disk Space is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [16:24:22] PROBLEM Total processes is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [16:25:13] PROBLEM dpkg-check is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: Connection refused by host [16:28:33] RECOVERY Current Load is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: OK - load average: 0.95, 0.92, 0.52 [16:28:53] RECOVERY Free ram is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: OK: 96% free memory [16:29:13] RECOVERY Disk Space is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: DISK OK [16:29:23] RECOVERY Total processes is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: PROCS OK: 99 processes [16:29:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [16:30:53] RECOVERY dpkg-check is now: OK on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: All packages OK [16:39:12] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [16:39:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 23% free memory [16:47:05] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [16:48:05] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [16:48:25] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:48:35] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [16:49:05] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [16:49:05] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [16:49:15] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [16:49:25] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:02:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [17:02:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [17:17:23] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:18:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:18:43] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:18:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:19:13] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:19:13] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:19:33] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [17:19:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:31:23] PROBLEM Total processes is now: WARNING on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: PROCS WARNING: 153 processes [17:37:22] RECOVERY Total processes is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: PROCS OK: 147 processes [17:44:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 202 processes [17:47:42] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [17:49:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:49:12] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [17:49:12] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [17:49:22] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [17:49:42] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [17:49:42] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [17:52:03] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [17:54:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 199 processes [18:06:12] PROBLEM Total processes is now: WARNING on etherpad-lite.pmtpa.wmflabs 10.4.0.87 output: PROCS WARNING: 152 processes [18:17:43] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:19:13] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:19:13] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:19:23] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:19:43] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:19:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:19:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:22:12] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [18:26:12] RECOVERY Total processes is now: OK on etherpad-lite.pmtpa.wmflabs 10.4.0.87 output: PROCS OK: 117 processes [18:48:02] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [18:49:32] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [18:49:32] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [18:49:42] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [18:50:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [18:50:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:51:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [18:52:13] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [19:18:06] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [19:19:56] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:19:56] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:20:43] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:20:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:21:33] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:22:14] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [19:22:53] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:48:55] PROBLEM Current Load is now: CRITICAL on rl2-apache2.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [19:49:25] PROBLEM host: centralauth-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.250 CRITICAL - Host Unreachable (10.4.0.250) [19:49:35] PROBLEM Disk Space is now: CRITICAL on rl2-apache2.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [19:50:22] PROBLEM Free ram is now: CRITICAL on rl2-apache2.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [19:51:02] PROBLEM host: phabricator.pmtpa.wmflabs is DOWN address: 10.4.0.119 CRITICAL - Host Unreachable (10.4.0.119) [19:51:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:51:32] PROBLEM host: metavidwiki.pmtpa.wmflabs is DOWN address: 10.4.0.216 CRITICAL - Host Unreachable (10.4.0.216) [19:51:42] PROBLEM Total processes is now: CRITICAL on rl2-apache2.pmtpa.wmflabs 10.4.1.54 output: Connection refused by host [19:52:12] PROBLEM host: wikiversity-sandbox-frontend.pmtpa.wmflabs is DOWN address: 10.4.0.203 CRITICAL - Host Unreachable (10.4.0.203) [19:52:42] PROBLEM dpkg-check is now: CRITICAL on rl2-apache2.pmtpa.wmflabs 10.4.1.54 output: CHECK_NRPE: Socket timeout after 10 seconds. [19:52:52] PROBLEM host: resourceloader2-apache.pmtpa.wmflabs is DOWN address: 10.4.0.128 CRITICAL - Host Unreachable (10.4.0.128) [19:53:02] PROBLEM host: blamemaps-m1xsmall.pmtpa.wmflabs is DOWN address: 10.4.0.142 CRITICAL - Host Unreachable (10.4.0.142) [19:53:32] PROBLEM host: glam-gwtoolset-apt.pmtpa.wmflabs is DOWN address: 10.4.0.27 CRITICAL - Host Unreachable (10.4.0.27) [19:58:53] PROBLEM Current Load is now: CRITICAL on gadgets-apache3.pmtpa.wmflabs 10.4.1.92 output: Connection refused by host [19:59:33] PROBLEM Disk Space is now: CRITICAL on gadgets-apache3.pmtpa.wmflabs 10.4.1.92 output: Connection refused by host [20:00:12]