[00:10:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:14:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [00:19:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [00:21:53] RECOVERY Free ram is now: OK on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: OK: 21% free memory [00:23:43] PROBLEM SSH is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.0.54 output: Server answer: [00:27:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [00:30:23] RECOVERY Free ram is now: OK on aggregator1.pmtpa.wmflabs 10.4.0.79 output: OK: 86% free memory [00:31:43] RECOVERY Free ram is now: OK on aggregator2.pmtpa.wmflabs 10.4.0.193 output: OK: 92% free memory [00:33:43] RECOVERY SSH is now: OK on bastion1.pmtpa.wmflabs 10.4.0.54 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [00:33:53] PROBLEM Current Load is now: CRITICAL on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:34:33] PROBLEM Disk Space is now: CRITICAL on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:35:12] PROBLEM Free ram is now: CRITICAL on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:36:42] PROBLEM Total processes is now: CRITICAL on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:37:22] PROBLEM dpkg-check is now: CRITICAL on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: Connection refused by host [00:38:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 27% free memory [00:39:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [00:39:33] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [00:40:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [00:41:43] PROBLEM SSH is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.0.54 output: Server answer: [00:43:53] RECOVERY Current Load is now: OK on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: OK - load average: 0.54, 0.58, 0.51 [00:44:33] RECOVERY Disk Space is now: OK on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: DISK OK [00:44:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [00:45:12] RECOVERY Free ram is now: OK on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: OK: 86% free memory [00:45:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [00:46:42] RECOVERY Total processes is now: OK on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: PROCS OK: 85 processes [00:47:22] RECOVERY dpkg-check is now: OK on haproxy-test.pmtpa.wmflabs 10.4.0.115 output: All packages OK [00:49:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:02:23] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [01:06:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [01:07:32] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 12% free memory [01:11:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:15:32] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [01:19:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:34:03] PROBLEM Free ram is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [01:34:23] PROBLEM dpkg-check is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [01:34:23] PROBLEM Total processes is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [01:34:43] PROBLEM Current Load is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [01:36:22] PROBLEM Disk Space is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.1.55 output: Connection refused by host [01:41:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [01:44:53] PROBLEM Free ram is now: WARNING on dumps-bot2.pmtpa.wmflabs 10.4.0.60 output: Warning: 19% free memory [01:46:32] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [01:49:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [01:51:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [02:13:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:16:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [02:19:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [02:24:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [02:43:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [02:46:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [02:49:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:10:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [03:13:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:16:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [03:18:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [03:19:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [03:43:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [03:47:02] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [03:49:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:13:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:17:02] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [04:19:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [04:19:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [04:37:23] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [04:38:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [04:40:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [04:43:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [04:47:03] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [04:49:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:00:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [05:00:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [05:01:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [05:13:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:13:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [05:17:03] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [05:19:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [05:26:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [05:43:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [05:47:52] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [05:49:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:14:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:14:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [06:17:53] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [06:19:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:30:23] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [06:31:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [06:44:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [06:49:12] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [06:50:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [06:51:16] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [06:56:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [07:03:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [07:05:23] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [07:15:02] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:19:13] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [07:22:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:24:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [07:28:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [07:45:50] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [07:49:13] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [07:52:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [07:53:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [07:54:12] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [07:58:34] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 192 processes [08:15:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:19:13] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [08:24:13] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [08:24:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:37:32] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 21% free memory [08:38:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 27% free memory [08:38:22] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [08:38:32] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 201 processes [08:38:52] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 27% free memory [08:39:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [08:45:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [08:49:13] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [08:52:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 15% free memory [08:54:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [08:54:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:05:32] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [09:06:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [09:06:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 14% free memory [09:13:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 194 processes [09:16:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:19:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [09:21:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [09:24:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:24:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [09:26:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [09:36:22] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [09:40:23] PROBLEM Free ram is now: CRITICAL on sube.pmtpa.wmflabs 10.4.0.245 output: Critical: 2% free memory [09:45:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 14% free memory [09:46:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [09:49:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [09:54:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [09:54:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:09:23] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [10:16:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:19:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [10:19:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [10:24:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:24:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:24:23] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [10:27:22] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [10:46:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [10:49:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [10:54:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [10:54:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [10:55:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [10:58:52] PROBLEM Current Load is now: CRITICAL on bots-secure.pmtpa.wmflabs 10.4.0.54 output: Connection refused by host [11:03:53] RECOVERY Current Load is now: OK on bots-secure.pmtpa.wmflabs 10.4.0.54 output: OK - load average: 0.15, 0.57, 0.49 [11:07:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [11:16:32] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:19:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [11:24:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:24:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:43:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 204 processes [11:46:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [11:48:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 192 processes [11:49:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [11:49:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [11:54:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [11:54:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [11:57:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [12:04:22] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [12:07:27] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [12:16:33] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:18:33] RECOVERY Total processes is now: OK on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS OK: 150 processes [12:19:23] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [12:24:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:24:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [12:25:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [12:32:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [12:37:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 27% free memory [12:37:23] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [12:40:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 23% free memory [12:40:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [12:48:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [12:49:42] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [12:54:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [12:54:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:01:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 181 processes [13:13:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [13:18:23] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:19:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [13:20:24] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [13:24:34] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:24:44] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [13:30:23] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [13:31:33] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 217 processes [13:36:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [13:48:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [13:50:23] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [13:54:43] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [13:57:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:13:53] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 19% free memory [14:18:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:20:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [14:21:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [14:24:52] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:26:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [14:28:03] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [14:35:22] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [14:48:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [14:48:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [14:51:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [14:54:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [14:54:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [14:58:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:03:23] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [15:18:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:21:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [15:24:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:24:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [15:26:32] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 200 processes [15:28:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [15:48:42] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [15:51:43] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [15:54:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [15:58:52] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:07:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 19% free memory [16:18:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:22:02] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [16:25:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:26:34] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 207 processes [16:28:54] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [16:37:53] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 21% free memory [16:38:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 22% free memory [16:38:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 27% free memory [16:38:23] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 21% free memory [16:38:53] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 23% free memory [16:40:34] RECOVERY Free ram is now: OK on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: OK: 20% free memory [16:41:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 21% free memory [16:46:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [16:46:52] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 19% free memory [16:49:12] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [16:52:03] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [16:56:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [16:56:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [16:59:02] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:05:52] PROBLEM Free ram is now: CRITICAL on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: Critical: 4% free memory [17:06:22] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [17:08:32] PROBLEM Free ram is now: WARNING on bots-sql2.pmtpa.wmflabs 10.4.0.41 output: Warning: 14% free memory [17:09:52] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [17:11:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [17:11:33] PROBLEM Total processes is now: WARNING on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS WARNING: 165 processes [17:13:13] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 7.55, 7.25, 5.55 [17:14:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 7.53, 7.56, 6.09 [17:15:53] RECOVERY Free ram is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK: 24% free memory [17:16:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 6.75, 6.68, 5.42 [17:19:34] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:20:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 7.08, 6.58, 5.42 [17:21:41] PROBLEM Total processes is now: CRITICAL on bots-bnr1.pmtpa.wmflabs 10.4.1.68 output: PROCS CRITICAL: 327 processes [17:22:05] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [17:26:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:29:03] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:39:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 4.71, 5.35, 5.04 [17:40:53] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: Warning: 13% free memory [17:44:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 3.78, 4.63, 4.85 [17:49:53] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [17:52:53] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [17:53:12] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: OK - load average: 2.61, 3.51, 4.74 [17:54:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 19% free memory [17:56:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 20% free memory [17:56:53] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: OK - load average: 1.66, 2.47, 4.82 [17:56:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [17:59:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [17:59:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 1.91, 2.70, 4.83 [18:00:54] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 1.63, 2.25, 4.20 [18:19:55] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:22:54] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [18:26:53] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:29:23] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [18:29:23] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [18:41:54] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [18:50:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [18:54:13] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [18:57:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [18:59:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:06:53] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 20% free memory [19:19:52] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 19% free memory [19:20:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:21:23] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [19:24:13] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [19:26:52] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [19:27:02] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:29:22] PROBLEM Free ram is now: WARNING on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: Warning: 19% free memory [19:29:42] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [19:34:22] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [19:47:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 18% free memory [19:50:43] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [19:54:13] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [19:54:53] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [19:59:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [19:59:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:20:44] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:24:14] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [20:24:54] RECOVERY Free ram is now: OK on swift-be2.pmtpa.wmflabs 10.4.0.112 output: OK: 20% free memory [20:29:23] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [20:29:43] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [20:30:54] PROBLEM dpkg-check is now: CRITICAL on nagios-dev.pmtpa.wmflabs 10.4.0.201 output: DPKG CRITICAL dpkg reports broken packages [20:32:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core.pmtpa.wmflabs 10.4.0.222 output: WARNING - load average: 9.90, 8.12, 6.32 [20:34:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: WARNING - load average: 11.98, 10.13, 6.63 [20:37:23] RECOVERY Free ram is now: OK on swift-be4.pmtpa.wmflabs 10.4.0.127 output: OK: 20% free memory [20:37:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: WARNING - load average: 10.68, 9.72, 7.01 [20:37:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: WARNING - load average: 11.51, 9.28, 6.26 [20:39:23] RECOVERY Free ram is now: OK on sultestdb.pmtpa.wmflabs 10.4.0.199 output: OK: 28% free memory [20:39:53] RECOVERY Free ram is now: OK on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: OK: 22% free memory [20:39:53] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 21% free memory [20:40:22] RECOVERY Free ram is now: OK on sube.pmtpa.wmflabs 10.4.0.245 output: OK: 22% free memory [20:43:12] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: WARNING - load average: 10.68, 8.01, 6.12 [20:43:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: WARNING - load average: 10.89, 9.87, 6.88 [20:47:53] PROBLEM Free ram is now: WARNING on conventionextension-trial.pmtpa.wmflabs 10.4.0.165 output: Warning: 13% free memory [20:51:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [20:53:22] PROBLEM Free ram is now: WARNING on sube.pmtpa.wmflabs 10.4.0.245 output: Warning: 13% free memory [20:54:22] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [20:59:36] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:00:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:07:52] PROBLEM Free ram is now: WARNING on swift-be3.pmtpa.wmflabs 10.4.0.124 output: Warning: 19% free memory [21:10:22] PROBLEM Free ram is now: WARNING on swift-be4.pmtpa.wmflabs 10.4.0.127 output: Warning: 18% free memory [21:12:22] PROBLEM Free ram is now: WARNING on sultestdb.pmtpa.wmflabs 10.4.0.199 output: Warning: 14% free memory [21:12:52] PROBLEM Free ram is now: WARNING on swift-be2.pmtpa.wmflabs 10.4.0.112 output: Warning: 18% free memory [21:21:35] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:24:23] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [21:29:32] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [21:31:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [21:37:52] RECOVERY Free ram is now: OK on swift-be3.pmtpa.wmflabs 10.4.0.124 output: OK: 20% free memory [21:49:22] RECOVERY Free ram is now: OK on wikistream-1.pmtpa.wmflabs 10.4.0.110 output: OK: 24% free memory [21:53:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [21:54:42] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [21:59:33] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:00:53] RECOVERY dpkg-check is now: OK on nagios-dev.pmtpa.wmflabs 10.4.0.201 output: All packages OK [22:02:33] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:03:53] RECOVERY Current Load is now: OK on ve-roundtrip2.pmtpa.wmflabs 10.4.0.162 output: OK - load average: 0.87, 1.49, 4.90 [22:12:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core.pmtpa.wmflabs 10.4.0.39 output: OK - load average: 2.89, 2.98, 4.69 [22:17:23] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [22:22:22] PROBLEM Free ram is now: WARNING on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Warning: 6% free memory [22:22:52] RECOVERY Current Load is now: OK on parsoid-roundtrip3.pmtpa.wmflabs 10.4.0.62 output: OK - load average: 3.23, 3.99, 4.89 [22:23:22] PROBLEM host: orgcharts-dev.pmtpa.wmflabs is DOWN address: 10.4.0.122 CRITICAL - Host Unreachable (10.4.0.122) [22:24:42] PROBLEM host: proveit.pmtpa.wmflabs is DOWN address: 10.4.0.236 CRITICAL - Host Unreachable (10.4.0.236) [22:27:22] PROBLEM Free ram is now: CRITICAL on dumps-bot1.pmtpa.wmflabs 10.4.0.4 output: Critical: 5% free memory [22:29:42] PROBLEM host: stackfarm-sql2.pmtpa.wmflabs is DOWN address: 10.4.1.23 CRITICAL - Host Unreachable (10.4.1.23) [22:33:34] PROBLEM host: tstarling-puppet.pmtpa.wmflabs is DOWN address: 10.4.1.79 CRITICAL - Host Unreachable (10.4.1.79) [22:39:52] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core.pmtpa.wmflabs 10.4.0.125 output: OK - load average: 2.99, 3.56, 4.82 [22:43:23] PROBLEM dpkg-check is now: CRITICAL on reportcard.pmtpa.wmflabs 10.4.0.229 output: DPKG CRITICAL dpkg reports broken packages [22:43:43] PROBLEM dpkg-check is now: CRITICAL on openid-wiki2.pmtpa.wmflabs 10.4.1.80 output: DPKG CRITICAL dpkg reports broken packages [22:44:33] PROBLEM dpkg-check is now: CRITICAL on tools-login.pmtpa.wmflabs 10.4.0.220 output: DPKG CRITICAL dpkg reports broken packages [22:44:43] PROBLEM dpkg-check is now: CRITICAL on bastion3.pmtpa.wmflabs 10.4.1.85 output: DPKG CRITICAL dpkg reports broken packages [22:44:53] PROBLEM dpkg-check is now: CRITICAL on cvn-app1.pmtpa.wmflabs 10.4.1.90 output: DPKG CRITICAL dpkg reports broken packages [22:45:23] PROBLEM dpkg-check is now: CRITICAL on saltminion.pmtpa.wmflabs 10.4.0.153 output: DPKG CRITICAL dpkg reports broken packages [22:45:23] PROBLEM dpkg-check is now: CRITICAL on tools-db.pmtpa.wmflabs 10.4.0.8 output: DPKG CRITICAL dpkg reports broken packages [22:45:33] PROBLEM dpkg-check is now: CRITICAL on test-hhvm1.pmtpa.wmflabs 10.4.0.52 output: DPKG CRITICAL dpkg reports broken packages [22:45:43] PROBLEM dpkg-check is now: CRITICAL on lucene-test1.pmtpa.wmflabs 10.4.0.205 output: CHECK_NRPE: Socket timeout after 10 seconds. [22:45:43] PROBLEM dpkg-check is now: CRITICAL on dumps-2.pmtpa.wmflabs 10.4.1.36 output: DPKG CRITICAL dpkg reports broken packages [22:45:43] PROBLEM dpkg-check is now: CRITICAL on generator.pmtpa.wmflabs 10.4.0.189 output: DPKG CRITICAL dpkg reports broken packages [22:45:43] PROBLEM dpkg-check is now: CRITICAL on stackfarm-mc.pmtpa.wmflabs 10.4.1.4 output: DPKG CRITICAL dpkg reports broken packages [22:45:43] PROBLEM dpkg-check is now: CRITICAL on maps-osmmapnik.pmtpa.wmflabs 10.4.1.34 output: DPKG CRITICAL dpkg reports broken packages [22:45:53] PROBLEM dpkg-check is now: CRITICAL on pediapress-ocg2.pmtpa.wmflabs 10.4.0.28 output: DPKG CRITICAL dpkg reports broken packages [22:45:53] PROBLEM dpkg-check is now: CRITICAL on tools-exec-01.pmtpa.wmflabs 10.4.0.156 output: DPKG CRITICAL dpkg reports broken packages [22:45:53] PROBLEM dpkg-check is now: CRITICAL on tools-master.pmtpa.wmflabs 10.4.1.54 output: DPKG CRITICAL dpkg reports broken packages [22:45:53] PROBLEM dpkg-check is now: CRITICAL on webtools-login.pmtpa.wmflabs 10.4.0.24 output: DPKG CRITICAL dpkg reports broken packages [22:46:03] PROBLEM dpkg-check is now: CRITICAL on xn--tekruo-t1a.pmtpa.wmflabs 10.4.1.60 output: DPKG CRITICAL dpkg reports broken packages [22:46:03] PROBLEM dpkg-check is now: CRITICAL on glam-gwtoolset.pmtpa.wmflabs 10.4.1.9 output: DPKG CRITICAL dpkg reports broken packages [22:46:03] PROBLEM dpkg-check is now: CRITICAL on stackfarm-sql1.pmtpa.wmflabs 10.4.1.8 output: DPKG CRITICAL dpkg reports broken packages [22:46:03] PROBLEM dpkg-check is now: CRITICAL on saltmaster.pmtpa.wmflabs 10.4.0.150 output: DPKG CRITICAL dpkg reports broken packages [22:46:13] PROBLEM dpkg-check is now: CRITICAL on dumpmuncher.pmtpa.wmflabs 10.4.0.32 output: DPKG CRITICAL dpkg reports broken packages [22:46:23] PROBLEM dpkg-check is now: CRITICAL on damian-keytest.pmtpa.wmflabs 10.4.0.97 output: DPKG CRITICAL dpkg reports broken packages [22:46:23] PROBLEM dpkg-check is now: CRITICAL on ee-isarra.pmtpa.wmflabs 10.4.0.82 output: DPKG CRITICAL dpkg reports broken packages [22:46:33] PROBLEM dpkg-check is now: CRITICAL on mingle.pmtpa.wmflabs 10.4.0.55 output: DPKG CRITICAL dpkg reports broken packages [22:46:33] PROBLEM dpkg-check is now: CRITICAL on parsoid-roundtrip7-8core.pmtpa.wmflabs 10.4.1.26 output: DPKG CRITICAL dpkg reports broken packages [22:46:33] PROBLEM dpkg-check is now: CRITICAL on puppet1.pmtpa.wmflabs 10.4.0.251 output: DPKG CRITICAL dpkg reports broken packages [22:46:43] PROBLEM dpkg-check is now: CRITICAL on parsoid-spof.pmtpa.wmflabs 10.4.0.33 output: DPKG CRITICAL dpkg reports broken packages [22:46:53] PROBLEM dpkg-check is now: CRITICAL on bots-abogott-devel.pmtpa.wmflabs 10.4.1.42 output: DPKG CRITICAL dpkg reports broken packages [22:46:53] PROBLEM dpkg-check is now: CRITICAL on bots-liwa.pmtpa.wmflabs 10.4.1.65 output: DPKG CRITICAL dpkg reports broken packages [22:46:53] PROBLEM dpkg-check is now: CRITICAL on mediawiki-dev-1.pmtpa.wmflabs 10.4.0.89 output: DPKG CRITICAL dpkg reports broken packages [22:46:53] PROBLEM dpkg-check is now: CRITICAL on authorlinks.pmtpa.wmflabs 10.4.1.62 output: DPKG CRITICAL dpkg reports broken packages [22:46:53] PROBLEM dpkg-check is now: CRITICAL on gerrit-dev-zap.pmtpa.wmflabs 10.4.1.21 output: DPKG CRITICAL dpkg reports broken packages [22:46:53] PROBLEM dpkg-check is now: CRITICAL on incubator-mc.pmtpa.wmflabs 10.4.0.20 output: DPKG CRITICAL dpkg reports broken packages [22:46:53] PROBLEM dpkg-check is now: CRITICAL on util-abogott.pmtpa.wmflabs 10.4.0.217 output: DPKG CRITICAL dpkg reports broken packages [22:47:03] PROBLEM dpkg-check is now: CRITICAL on bots-1.pmtpa.wmflabs 10.4.0.48 output: DPKG CRITICAL dpkg reports broken packages [22:47:03] PROBLEM dpkg-check is now: CRITICAL on maps-pgsql3.pmtpa.wmflabs 10.4.1.75 output: DPKG CRITICAL dpkg reports broken packages [22:47:03] PROBLEM dpkg-check is now: CRITICAL on bots-apache01.pmtpa.wmflabs 10.4.0.141 output: DPKG CRITICAL dpkg reports broken packages [22:47:03] PROBLEM dpkg-check is now: CRITICAL on instance-proxy.pmtpa.wmflabs 10.4.1.15 output: DPKG CRITICAL dpkg reports broken packages [22:47:13] PROBLEM dpkg-check is now: CRITICAL on gerrit-build.pmtpa.wmflabs 10.4.0.70 output: DPKG CRITICAL dpkg reports broken packages [22:47:13] PROBLEM dpkg-check is now: CRITICAL on home-migrate-precise.pmtpa.wmflabs 10.4.1.45 output: DPKG CRITICAL dpkg reports broken packages [22:47:14] PROBLEM dpkg-check is now: CRITICAL on tools-webproxy.pmtpa.wmflabs 10.4.1.89 output: DPKG CRITICAL dpkg reports broken packages [22:47:23] PROBLEM dpkg-check is now: CRITICAL on robh-mingle.pmtpa.wmflabs 10.4.0.92 output: DPKG CRITICAL dpkg reports broken packages [22:47:23] PROBLEM dpkg-check is now: CRITICAL on umn-snuggle.pmtpa.wmflabs 10.4.1.5 output: DPKG CRITICAL dpkg reports broken packages [22:47:23] PROBLEM dpkg-check is now: CRITICAL on nginx-devunt.pmtpa.wmflabs 10.4.0.65 output: DPKG CRITICAL dpkg reports broken packages [22:47:23] PROBLEM dpkg-check is now: CRITICAL on tools-puppet-test.pmtpa.wmflabs 10.4.0.240 output: DPKG CRITICAL dpkg reports broken packages [22:47:23] PROBLEM dpkg-check is now: CRITICAL on webtools-odie.pmtpa.wmflabs 10.4.1.39 output: DPKG CRITICAL dpkg reports broken packages [22:47:23] PROBLEM dpkg-check is now: CRITICAL on bastion1.pmtpa.wmflabs 10.4.1.55 output: DPKG CRITICAL dpkg reports broken packages [22:47:23] PROBLEM dpkg-check is now: CRITICAL on wlgroupstester.pmtpa.wmflabs 10.4.1.1 output: DPKG CRITICAL dpkg reports broken packages [22:47:24] PROBLEM dpkg-check is now: CRITICAL on bastion2.pmtpa.wmflabs 10.4.1.84 output: DPKG CRITICAL dpkg reports broken packages [22:47:24] PROBLEM dpkg-check is now: CRITICAL on bots-nr1.pmtpa.wmflabs 10.4.1.2 output: DPKG CRITICAL dpkg reports broken packages [22:47:25] PROBLEM dpkg-check is now: CRITICAL on fundraising-ci.pmtpa.wmflabs 10.4.0.221 output: DPKG CRITICAL dpkg reports broken packages [22:47:25] PROBLEM dpkg-check is now: CRITICAL on orgchart.pmtpa.wmflabs 10.4.0.86 output: DPKG CRITICAL dpkg reports broken packages [22:47:33] PROBLEM dpkg-check is now: CRITICAL on redmine.pmtpa.wmflabs 10.4.1.71 output: DPKG CRITICAL dpkg reports broken packages [22:47:33] PROBLEM dpkg-check is now: CRITICAL on gerrit-stream.pmtpa.wmflabs 10.4.1.31 output: DPKG CRITICAL dpkg reports broken packages [22:47:33] PROBLEM dpkg-check is now: CRITICAL on proposals-1.pmtpa.wmflabs 10.4.0.188 output: DPKG CRITICAL dpkg reports broken packages [22:47:33] PROBLEM dpkg-check is now: CRITICAL on checkwiki-web3.pmtpa.wmflabs 10.4.1.74 output: DPKG CRITICAL dpkg reports broken packages [22:47:33]