[00:11:38] PROBLEM - Lucene on searchidx1001 is CRITICAL: Connection refused [00:28:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 4.171 seconds [00:51:57] !log reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe' [00:52:00] Logged the message, Master [00:54:00] gn8 folks [00:54:16] !log reedy synchronized wmf-config/InitialiseSettings.php 'Add NS to stewardwiki at request of Philippe' [00:54:19] Logged the message, Master [00:54:30] 'night DaB [01:09:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:36] !log LocalisationUpdate completed (1.19) at Fri Mar 16 02:21:35 UTC 2012 [02:21:40] Logged the message, Master [02:36:33] RECOVERY - RAID on cp1035 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [02:36:34] RECOVERY - DPKG on mw1107 is OK: All packages OK [02:36:34] RECOVERY - DPKG on mw1156 is OK: All packages OK [02:36:34] RECOVERY - RAID on mw1146 is OK: OK: no RAID installed [02:36:34] RECOVERY - RAID on mw1011 is OK: OK: no RAID installed [02:36:42] RECOVERY - Disk space on mw1139 is OK: DISK OK [02:36:42] RECOVERY - Disk space on mw1074 is OK: DISK OK [02:36:42] RECOVERY - RAID on cp1012 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [02:36:42] RECOVERY - DPKG on cp1015 is OK: All packages OK [02:36:42] RECOVERY - DPKG on virt3 is OK: All packages OK [02:36:43] RECOVERY - RAID on cp1013 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [02:36:51] RECOVERY - Disk space on mw1127 is OK: DISK OK [02:36:51] RECOVERY - RAID on ms5 is OK: OK: Active: 50, Working: 50, Failed: 0, Spare: 0 [02:36:51] RECOVERY - DPKG on mw1131 is OK: All packages OK [02:36:51] RECOVERY - DPKG on mw1112 is OK: All packages OK [02:36:51] RECOVERY - DPKG on virt4 is OK: All packages OK [02:36:52] RECOVERY - Disk space on mw1015 is OK: DISK OK [02:36:52] RECOVERY - RAID on mw1032 is OK: OK: no RAID installed [02:36:53] RECOVERY - Disk space on mw1106 is OK: DISK OK [02:36:53] RECOVERY - Disk space on mw1112 is OK: DISK OK [02:36:54] RECOVERY - Disk space on mw1100 is OK: DISK OK [02:36:54] RECOVERY - Disk space on cp1035 is OK: DISK OK [02:36:55] RECOVERY - DPKG on mw1067 is OK: All packages OK [02:36:55] RECOVERY - RAID on mw1109 is OK: OK: no RAID installed [02:46:23] yay, nagios-wm gone [02:46:27] ow, it's gone everywhere.. [03:05:21] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:05:39] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.148 seconds [03:07:18] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 0.020 seconds [03:18:18] test [03:22:11] 1234 [03:23:30] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [03:28:30] PROBLEM - Puppet freshness on cp1022 is CRITICAL: Puppet has not run in the last 10 hours [03:34:24] PROBLEM - Puppet freshness on cp1021 is CRITICAL: Puppet has not run in the last 10 hours [03:35:27] RECOVERY - RAID on aluminium is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [03:35:36] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [03:36:39] PROBLEM - Puppet freshness on cp1041 is CRITICAL: Puppet has not run in the last 10 hours [03:36:48] PROBLEM - swift-object-server on copper is CRITICAL: Connection refused by host [03:36:48] PROBLEM - swift-container-auditor on copper is CRITICAL: Connection refused by host [03:36:57] PROBLEM - swift-container-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [03:36:57] PROBLEM - swift-container-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [03:36:57] PROBLEM - swift-object-auditor on magnesium is CRITICAL: Connection refused by host [03:36:57] PROBLEM - swift-account-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [03:36:57] PROBLEM - swift-object-updater on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [03:36:57] PROBLEM - swift-account-replicator on magnesium is CRITICAL: Connection refused by host [03:36:58] PROBLEM - swift-account-server on ms3 is CRITICAL: NRPE: Command check_swift_account_server not defined [03:36:58] PROBLEM - swift-object-replicator on ms3 is CRITICAL: NRPE: Command check_swift_object_replicator not defined [03:37:15] PROBLEM - swift-account-reaper on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [03:37:15] PROBLEM - swift-container-updater on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [03:37:15] PROBLEM - swift-container-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [03:37:15] PROBLEM - swift-object-server on ms3 is CRITICAL: NRPE: Command check_swift_object_server not defined [03:37:15] PROBLEM - swift-account-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [03:37:15] PROBLEM - swift-container-auditor on ms3 is CRITICAL: NRPE: Command check_swift_container_auditor not defined [03:37:15] PROBLEM - swift-container-replicator on copper is CRITICAL: Connection refused by host [03:37:16] PROBLEM - swift-object-updater on copper is CRITICAL: Connection refused by host [03:37:16] PROBLEM - swift-account-server on zinc is CRITICAL: Connection refused by host [04:24:44] PROBLEM - SSH on sq40 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:47] PROBLEM - Backend Squid HTTP on sq40 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:26:05] PROBLEM - Frontend Squid HTTP on sq40 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:17] PROBLEM - Host sq40 is DOWN: PING CRITICAL - Packet loss = 100% [05:01:17] PROBLEM - swift-object-server on copper is CRITICAL: Connection refused by host [05:01:17] PROBLEM - swift-container-auditor on copper is CRITICAL: Connection refused by host [05:01:26] PROBLEM - swift-account-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:01:26] PROBLEM - swift-container-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:01:26] PROBLEM - swift-account-replicator on magnesium is CRITICAL: Connection refused by host [05:01:26] PROBLEM - swift-object-auditor on magnesium is CRITICAL: Connection refused by host [05:01:26] PROBLEM - swift-container-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:01:26] PROBLEM - swift-object-replicator on ms3 is CRITICAL: NRPE: Command check_swift_object_replicator not defined [05:01:27] PROBLEM - swift-object-updater on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:01:27] PROBLEM - swift-account-server on ms3 is CRITICAL: NRPE: Command check_swift_account_server not defined [05:01:35] PROBLEM - swift-container-replicator on copper is CRITICAL: Connection refused by host [05:01:35] PROBLEM - swift-object-updater on copper is CRITICAL: Connection refused by host [05:01:44] PROBLEM - swift-container-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:01:44] PROBLEM - swift-account-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:01:44] PROBLEM - swift-container-updater on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:01:44] PROBLEM - swift-account-reaper on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:01:44] PROBLEM - swift-container-auditor on ms3 is CRITICAL: NRPE: Command check_swift_container_auditor not defined [05:01:45] PROBLEM - swift-object-server on ms3 is CRITICAL: NRPE: Command check_swift_object_server not defined [05:01:53] PROBLEM - swift-account-server on zinc is CRITICAL: Connection refused by host [05:01:53] PROBLEM - swift-object-replicator on zinc is CRITICAL: Connection refused by host [05:02:02] PROBLEM - swift-account-server on magnesium is CRITICAL: Connection refused by host [05:02:02] PROBLEM - swift-account-auditor on copper is CRITICAL: Connection refused by host [05:02:02] PROBLEM - swift-container-server on copper is CRITICAL: Connection refused by host [05:02:02] PROBLEM - swift-object-server on zinc is CRITICAL: Connection refused by host [05:02:02] PROBLEM - swift-container-auditor on zinc is CRITICAL: Connection refused by host [05:02:11] PROBLEM - swift-container-replicator on ms3 is CRITICAL: NRPE: Command check_swift_container_replicator not defined [05:02:11] PROBLEM - swift-object-updater on ms3 is CRITICAL: NRPE: Command check_swift_object_updater not defined [05:02:11] PROBLEM - swift-object-server on magnesium is CRITICAL: Connection refused by host [05:02:11] PROBLEM - swift-account-reaper on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:02:11] PROBLEM - swift-account-replicator on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:02:12] PROBLEM - swift-object-replicator on magnesium is CRITICAL: Connection refused by host [05:02:20] PROBLEM - swift-container-updater on copper is CRITICAL: Connection refused by host [05:02:20] PROBLEM - swift-account-reaper on copper is CRITICAL: Connection refused by host [05:02:20] PROBLEM - swift-container-auditor on magnesium is CRITICAL: Connection refused by host [05:02:20] PROBLEM - swift-container-updater on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:02:20] PROBLEM - swift-object-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:02:29] PROBLEM - swift-object-replicator on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:02:29] PROBLEM - swift-object-updater on magnesium is CRITICAL: Connection refused by host [05:02:29] PROBLEM - swift-container-replicator on magnesium is CRITICAL: Connection refused by host [05:02:29] PROBLEM - swift-account-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:02:29] PROBLEM - swift-account-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:02:29] PROBLEM - swift-object-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:02:29] PROBLEM - swift-account-auditor on ms3 is CRITICAL: NRPE: Command check_swift_account_auditor not defined [05:02:30] PROBLEM - swift-container-server on ms3 is CRITICAL: NRPE: Command check_swift_container_server not defined [05:02:38] PROBLEM - swift-account-auditor on zinc is CRITICAL: Connection refused by host [05:02:38] PROBLEM - swift-container-server on zinc is CRITICAL: Connection refused by host [05:02:38] PROBLEM - swift-object-updater on zinc is CRITICAL: Connection refused by host [05:02:38] PROBLEM - swift-container-replicator on zinc is CRITICAL: Connection refused by host [05:02:47] PROBLEM - swift-container-server on magnesium is CRITICAL: Connection refused by host [05:02:47] PROBLEM - swift-account-auditor on magnesium is CRITICAL: Connection refused by host [05:02:47] PROBLEM - swift-account-replicator on copper is CRITICAL: Connection refused by host [05:02:56] PROBLEM - swift-object-auditor on copper is CRITICAL: Connection refused by host [05:02:56] PROBLEM - swift-container-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:03:05] PROBLEM - swift-account-server on copper is CRITICAL: Connection refused by host [05:03:05] PROBLEM - swift-object-replicator on copper is CRITICAL: Connection refused by host [05:03:05] PROBLEM - swift-account-reaper on zinc is CRITICAL: Connection refused by host [05:03:05] PROBLEM - swift-container-updater on ms3 is CRITICAL: NRPE: Command check_swift_container_updater not defined [05:03:05] PROBLEM - swift-container-updater on zinc is CRITICAL: Connection refused by host [05:03:05] PROBLEM - swift-account-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:03:06] PROBLEM - swift-account-reaper on magnesium is CRITICAL: Connection refused by host [05:03:06] PROBLEM - swift-container-updater on magnesium is CRITICAL: Connection refused by host [05:03:07] PROBLEM - swift-object-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:03:14] PROBLEM - swift-object-auditor on ms3 is CRITICAL: NRPE: Command check_swift_object_auditor not defined [05:03:14] PROBLEM - swift-object-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:03:14] PROBLEM - swift-object-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:03:23] PROBLEM - swift-object-auditor on zinc is CRITICAL: Connection refused by host [05:03:23] PROBLEM - swift-account-replicator on zinc is CRITICAL: Connection refused by host [05:03:23] PROBLEM - swift-account-reaper on ms3 is CRITICAL: NRPE: Command check_swift_account_reaper not defined [05:03:23] PROBLEM - swift-container-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:03:23] PROBLEM - swift-container-replicator on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:03:23] PROBLEM - swift-object-updater on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [05:03:24] PROBLEM - swift-account-replicator on ms3 is CRITICAL: NRPE: Command check_swift_account_replicator not defined [05:05:07] well, that is quite a few problems [05:06:24] Alpha_Quadrant: how many? [05:07:19] jeremyb: no idea, but the bot appears to be listing quite a few [05:07:41] it looks like 19 problems listed above [06:53:19] PROBLEM - swift-container-auditor on copper is CRITICAL: Connection refused by host [06:53:19] PROBLEM - swift-object-server on copper is CRITICAL: Connection refused by host [06:53:28] PROBLEM - swift-object-auditor on magnesium is CRITICAL: Connection refused by host [06:53:28] PROBLEM - swift-account-replicator on magnesium is CRITICAL: Connection refused by host [06:53:28] PROBLEM - swift-container-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:53:28] PROBLEM - swift-object-updater on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:53:28] PROBLEM - swift-account-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:53:29] PROBLEM - swift-container-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:53:29] PROBLEM - swift-object-replicator on ms3 is CRITICAL: NRPE: Command check_swift_object_replicator not defined [06:53:30] PROBLEM - swift-account-server on ms3 is CRITICAL: NRPE: Command check_swift_account_server not defined [06:53:37] PROBLEM - swift-object-updater on copper is CRITICAL: Connection refused by host [06:53:37] PROBLEM - swift-container-replicator on copper is CRITICAL: Connection refused by host [06:53:46] PROBLEM - swift-container-updater on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:53:46] PROBLEM - swift-account-reaper on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:53:46] PROBLEM - swift-container-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:53:46] PROBLEM - swift-account-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:53:46] PROBLEM - swift-object-server on ms3 is CRITICAL: NRPE: Command check_swift_object_server not defined [06:53:47] PROBLEM - swift-container-auditor on ms3 is CRITICAL: NRPE: Command check_swift_container_auditor not defined [06:53:47] PROBLEM - swift-account-server on zinc is CRITICAL: Connection refused by host [06:53:55] PROBLEM - swift-object-replicator on zinc is CRITICAL: Connection refused by host [06:54:04] PROBLEM - swift-account-auditor on copper is CRITICAL: Connection refused by host [06:54:04] PROBLEM - swift-container-server on copper is CRITICAL: Connection refused by host [06:54:04] PROBLEM - swift-container-auditor on zinc is CRITICAL: Connection refused by host [06:54:04] PROBLEM - swift-object-server on zinc is CRITICAL: Connection refused by host [06:54:04] PROBLEM - swift-object-replicator on magnesium is CRITICAL: Connection refused by host [06:54:13] PROBLEM - swift-container-replicator on ms3 is CRITICAL: NRPE: Command check_swift_container_replicator not defined [06:54:13] PROBLEM - swift-object-updater on ms3 is CRITICAL: NRPE: Command check_swift_object_updater not defined [06:54:13] PROBLEM - swift-account-server on magnesium is CRITICAL: Connection refused by host [06:54:13] PROBLEM - swift-object-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:13] PROBLEM - swift-object-server on magnesium is CRITICAL: Connection refused by host [06:54:13] PROBLEM - swift-account-reaper on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:14] PROBLEM - swift-container-auditor on magnesium is CRITICAL: Connection refused by host [06:54:22] PROBLEM - swift-account-reaper on copper is CRITICAL: Connection refused by host [06:54:22] PROBLEM - swift-container-updater on copper is CRITICAL: Connection refused by host [06:54:22] PROBLEM - swift-container-replicator on zinc is CRITICAL: Connection refused by host [06:54:22] PROBLEM - swift-object-updater on zinc is CRITICAL: Connection refused by host [06:54:22] PROBLEM - swift-container-updater on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:22] PROBLEM - swift-account-replicator on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:31] PROBLEM - swift-object-updater on magnesium is CRITICAL: Connection refused by host [06:54:31] PROBLEM - swift-account-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:31] PROBLEM - swift-container-replicator on magnesium is CRITICAL: Connection refused by host [06:54:31] PROBLEM - swift-object-replicator on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:31] PROBLEM - swift-account-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:31] PROBLEM - swift-object-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:31] PROBLEM - swift-account-auditor on ms3 is CRITICAL: NRPE: Command check_swift_account_auditor not defined [06:54:32] PROBLEM - swift-container-server on ms3 is CRITICAL: NRPE: Command check_swift_container_server not defined [06:54:40] PROBLEM - swift-container-server on zinc is CRITICAL: Connection refused by host [06:54:40] PROBLEM - swift-account-auditor on zinc is CRITICAL: Connection refused by host [06:54:49] PROBLEM - swift-object-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:49] PROBLEM - swift-object-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:49] PROBLEM - swift-account-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:54:49] PROBLEM - swift-account-reaper on ms3 is CRITICAL: NRPE: Command check_swift_account_reaper not defined [06:54:49] PROBLEM - swift-account-replicator on copper is CRITICAL: Connection refused by host [06:54:50] PROBLEM - swift-container-updater on ms3 is CRITICAL: NRPE: Command check_swift_container_updater not defined [06:54:58] PROBLEM - swift-object-auditor on copper is CRITICAL: Connection refused by host [06:55:07] PROBLEM - swift-container-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:55:07] PROBLEM - swift-account-auditor on magnesium is CRITICAL: Connection refused by host [06:55:07] PROBLEM - swift-object-replicator on copper is CRITICAL: Connection refused by host [06:55:07] PROBLEM - swift-account-server on copper is CRITICAL: Connection refused by host [06:55:07] PROBLEM - swift-container-updater on zinc is CRITICAL: Connection refused by host [06:55:07] PROBLEM - swift-account-reaper on zinc is CRITICAL: Connection refused by host [06:55:16] PROBLEM - swift-object-updater on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:55:16] PROBLEM - swift-object-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:55:16] PROBLEM - swift-container-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:55:16] PROBLEM - swift-container-replicator on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [06:55:16] PROBLEM - swift-account-replicator on ms3 is CRITICAL: NRPE: Command check_swift_account_replicator not defined [06:55:16] PROBLEM - swift-object-auditor on ms3 is CRITICAL: NRPE: Command check_swift_object_auditor not defined [06:55:16] PROBLEM - swift-account-reaper on magnesium is CRITICAL: Connection refused by host [06:55:25] PROBLEM - swift-container-server on magnesium is CRITICAL: Connection refused by host [06:55:25] PROBLEM - swift-object-auditor on zinc is CRITICAL: Connection refused by host [06:55:25] PROBLEM - swift-account-replicator on zinc is CRITICAL: Connection refused by host [06:55:25] PROBLEM - swift-container-updater on magnesium is CRITICAL: Connection refused by host [07:55:57] mhm [08:26:01] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 316 seconds [08:26:28] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 343 seconds [08:28:07] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 27 seconds [08:30:40] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 0 seconds [08:43:25] RECOVERY - Auth DNS on ns2.wikimedia.org is OK: DNS OK: 0.138 seconds response time. www.wikipedia.org returns 208.80.154.225 [08:44:55] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 182 seconds [08:45:22] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 188 seconds [08:47:01] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 4 seconds [08:47:28] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 0 seconds [08:57:17] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 208 seconds [08:58:56] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 271 seconds [09:05:14] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 0 seconds [09:05:41] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 0 seconds [09:27:35] PROBLEM - swift-container-auditor on ms-be2 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [09:30:44] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [09:31:47] RECOVERY - swift-container-auditor on ms-be2 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [09:32:50] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [09:39:44] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [09:39:44] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [09:48:26] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): /var/lib/ureadahead/debugfs 0 MB (0% inode=61%): [09:50:59] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): /var/lib/ureadahead/debugfs 0 MB (0% inode=61%): [09:50:59] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): /var/lib/ureadahead/debugfs 0 MB (0% inode=61%): [09:50:59] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 157 MB (2% inode=61%): /var/lib/ureadahead/debugfs 157 MB (2% inode=61%): [09:50:59] PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 176 MB (2% inode=61%): /var/lib/ureadahead/debugfs 176 MB (2% inode=61%): [09:50:59] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): /var/lib/ureadahead/debugfs 0 MB (0% inode=61%): [09:57:08] RECOVERY - Disk space on srv221 is OK: DISK OK [10:01:20] RECOVERY - Disk space on srv222 is OK: DISK OK [10:01:20] RECOVERY - Disk space on srv219 is OK: DISK OK [10:01:20] RECOVERY - Disk space on srv223 is OK: DISK OK [10:01:20] RECOVERY - Disk space on srv220 is OK: DISK OK [10:01:20] RECOVERY - Disk space on srv224 is OK: DISK OK [10:26:33] PROBLEM - swift-container-auditor on ms-be3 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [10:30:45] RECOVERY - swift-container-auditor on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [11:26:37] PROBLEM - swift-container-auditor on copper is CRITICAL: Connection refused by host [11:26:37] PROBLEM - swift-object-server on copper is CRITICAL: Connection refused by host [11:26:46] PROBLEM - swift-object-auditor on magnesium is CRITICAL: Connection refused by host [11:26:46] PROBLEM - swift-account-replicator on magnesium is CRITICAL: Connection refused by host [11:26:46] PROBLEM - swift-container-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:26:46] PROBLEM - swift-account-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:26:46] PROBLEM - swift-container-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:26:47] PROBLEM - swift-account-server on ms3 is CRITICAL: NRPE: Command check_swift_account_server not defined [11:26:47] PROBLEM - swift-object-updater on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:26:48] PROBLEM - swift-object-replicator on ms3 is CRITICAL: NRPE: Command check_swift_object_replicator not defined [11:27:04] PROBLEM - swift-container-replicator on copper is CRITICAL: Connection refused by host [11:27:04] PROBLEM - swift-object-updater on copper is CRITICAL: Connection refused by host [11:27:04] PROBLEM - swift-object-replicator on zinc is CRITICAL: Connection refused by host [11:27:04] PROBLEM - swift-account-server on zinc is CRITICAL: Connection refused by host [11:27:04] PROBLEM - swift-container-updater on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:04] PROBLEM - swift-account-reaper on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:05] PROBLEM - swift-account-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:05] PROBLEM - swift-object-server on ms3 is CRITICAL: NRPE: Command check_swift_object_server not defined [11:27:06] PROBLEM - swift-container-auditor on ms3 is CRITICAL: NRPE: Command check_swift_container_auditor not defined [11:27:06] PROBLEM - swift-container-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:22] PROBLEM - swift-account-server on magnesium is CRITICAL: Connection refused by host [11:27:22] PROBLEM - swift-container-server on copper is CRITICAL: Connection refused by host [11:27:22] PROBLEM - swift-account-auditor on copper is CRITICAL: Connection refused by host [11:27:22] PROBLEM - swift-container-auditor on zinc is CRITICAL: Connection refused by host [11:27:22] PROBLEM - swift-object-server on zinc is CRITICAL: Connection refused by host [11:27:31] PROBLEM - swift-object-server on magnesium is CRITICAL: Connection refused by host [11:27:31] PROBLEM - swift-object-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:31] PROBLEM - swift-account-reaper on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:31] PROBLEM - swift-container-auditor on magnesium is CRITICAL: Connection refused by host [11:27:31] PROBLEM - swift-container-updater on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:31] PROBLEM - swift-account-replicator on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:31] PROBLEM - swift-container-replicator on ms3 is CRITICAL: NRPE: Command check_swift_container_replicator not defined [11:27:32] PROBLEM - swift-object-updater on ms3 is CRITICAL: NRPE: Command check_swift_object_updater not defined [11:27:32] PROBLEM - swift-object-replicator on magnesium is CRITICAL: Connection refused by host [11:27:40] PROBLEM - swift-account-reaper on copper is CRITICAL: Connection refused by host [11:27:40] PROBLEM - swift-container-updater on copper is CRITICAL: Connection refused by host [11:27:49] PROBLEM - swift-object-updater on magnesium is CRITICAL: Connection refused by host [11:27:49] PROBLEM - swift-account-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:49] PROBLEM - swift-container-replicator on magnesium is CRITICAL: Connection refused by host [11:27:49] PROBLEM - swift-object-replicator on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:49] PROBLEM - swift-account-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:50] PROBLEM - swift-container-server on ms3 is CRITICAL: NRPE: Command check_swift_container_server not defined [11:27:50] PROBLEM - swift-object-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:27:51] PROBLEM - swift-account-auditor on ms3 is CRITICAL: NRPE: Command check_swift_account_auditor not defined [11:27:58] PROBLEM - swift-container-replicator on zinc is CRITICAL: Connection refused by host [11:28:07] PROBLEM - swift-object-auditor on copper is CRITICAL: Connection refused by host [11:28:07] PROBLEM - swift-account-replicator on copper is CRITICAL: Connection refused by host [11:28:07] PROBLEM - swift-container-server on zinc is CRITICAL: Connection refused by host [11:28:07] PROBLEM - swift-account-auditor on zinc is CRITICAL: Connection refused by host [11:28:16] PROBLEM - swift-account-auditor on magnesium is CRITICAL: Connection refused by host [11:28:16] PROBLEM - swift-container-server on magnesium is CRITICAL: Connection refused by host [11:28:16] PROBLEM - swift-container-auditor on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:28:16] PROBLEM - swift-object-server on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:28:16] PROBLEM - swift-object-replicator on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:28:17] PROBLEM - swift-container-updater on ms3 is CRITICAL: NRPE: Command check_swift_container_updater not defined [11:28:17] PROBLEM - swift-account-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:28:18] PROBLEM - swift-object-updater on zinc is CRITICAL: Connection refused by host [11:28:18] PROBLEM - swift-account-reaper on ms3 is CRITICAL: NRPE: Command check_swift_account_reaper not defined [11:28:25] PROBLEM - swift-account-server on copper is CRITICAL: Connection refused by host [11:28:25] PROBLEM - swift-object-replicator on copper is CRITICAL: Connection refused by host [11:28:25] PROBLEM - swift-account-reaper on zinc is CRITICAL: Connection refused by host [11:28:25] PROBLEM - swift-container-updater on zinc is CRITICAL: Connection refused by host [11:28:34] PROBLEM - swift-container-updater on magnesium is CRITICAL: Connection refused by host [11:28:34] PROBLEM - swift-account-reaper on magnesium is CRITICAL: Connection refused by host [11:28:34] PROBLEM - swift-object-updater on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:28:34] PROBLEM - swift-container-replicator on ms1 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:28:34] PROBLEM - swift-object-server on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:28:35] PROBLEM - swift-container-auditor on ms2 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [11:28:35] PROBLEM - swift-object-auditor on ms3 is CRITICAL: NRPE: Command check_swift_object_auditor not defined [11:28:36] PROBLEM - swift-account-replicator on ms3 is CRITICAL: NRPE: Command check_swift_account_replicator not defined [11:28:43] PROBLEM - swift-account-replicator on zinc is CRITICAL: Connection refused by host [11:28:43] PROBLEM - swift-object-auditor on zinc is CRITICAL: Connection refused by host [12:23:03] any coder with access to stewardswiki online? [12:39:10] PROBLEM - Puppet freshness on stafford is CRITICAL: Puppet has not run in the last 10 hours [12:45:47] got just some timeouts on deletions: from within function "LocalFile::delete". Database returned error "1205: Lock wait timeout exceeded; try restarting transaction (10.0.6.41)". [12:46:03] (at Wikimedia Commons) [12:46:20] retries were successful [12:50:32] same with me some hours ago! [12:51:03] needed some tries to delete *not only the page* but the file as well ... [12:51:25] i.e. the page has been deleted, but the file was still there and shown! [13:25:13] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [13:30:10] PROBLEM - Puppet freshness on cp1022 is CRITICAL: Puppet has not run in the last 10 hours [13:36:11] PROBLEM - Puppet freshness on cp1021 is CRITICAL: Puppet has not run in the last 10 hours [13:37:23] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [13:38:26] PROBLEM - Puppet freshness on cp1041 is CRITICAL: Puppet has not run in the last 10 hours [13:39:29] PROBLEM - Puppet freshness on cp1027 is CRITICAL: Puppet has not run in the last 10 hours [13:39:29] PROBLEM - Puppet freshness on cp1025 is CRITICAL: Puppet has not run in the last 10 hours [13:44:26] PROBLEM - Puppet freshness on cp1024 is CRITICAL: Puppet has not run in the last 10 hours [13:46:23] PROBLEM - Puppet freshness on cp1042 is CRITICAL: Puppet has not run in the last 10 hours [13:46:41] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:17] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:49:23] PROBLEM - Puppet freshness on cp1026 is CRITICAL: Puppet has not run in the last 10 hours [13:50:26] PROBLEM - Puppet freshness on cp1043 is CRITICAL: Puppet has not run in the last 10 hours [13:54:11] PROBLEM - swift-container-auditor on ms-be3 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [13:57:29] PROBLEM - Puppet freshness on cp1023 is CRITICAL: Puppet has not run in the last 10 hours [13:57:29] PROBLEM - Puppet freshness on cp1028 is CRITICAL: Puppet has not run in the last 10 hours [14:01:23] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 6.440 seconds [14:01:59] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.578 seconds [14:04:26] !log restarted swift-container-auditor on ms-be3, it had died for some reason [14:04:29] Logged the message, Master [14:04:41] RECOVERY - swift-container-auditor on ms-be3 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-auditor [14:06:11] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:08:08] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:08] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [14:10:23] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:32] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 8.622 seconds [14:22:50] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:04] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Fri Mar 16 14:46:59 UTC 2012 [14:50:58] RECOVERY - Puppet freshness on cp1041 is OK: puppet ran at Fri Mar 16 14:50:41 UTC 2012 [14:52:01] RECOVERY - Puppet freshness on cp1042 is OK: puppet ran at Fri Mar 16 14:51:46 UTC 2012 [14:53:22] PROBLEM - Varnish traffic logger on cp1021 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [14:54:52] PROBLEM - Varnish HTTP upload-backend on cp1021 is CRITICAL: Connection refused [14:55:16] !log reedy synchronized php-1.19/extensions/WikimediaMaintenance/cleanupBug31576.php [14:55:19] PROBLEM - DPKG on cp1021 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [14:55:19] Logged the message, Master [14:57:34] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Fri Mar 16 14:57:02 UTC 2012 [14:58:01] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Mar 16 14:57:56 UTC 2012 [14:59:04] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Fri Mar 16 14:58:53 UTC 2012 [14:59:04] RECOVERY - Varnish HTTP upload-backend on cp1021 is OK: HTTP OK HTTP/1.1 200 OK - 634 bytes in 0.055 seconds [15:04:35] !log root synchronized ufg.sql 'test sync to see if hume is fixed' [15:04:38] Logged the message, Master [15:04:55] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 183 seconds [15:06:34] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Fri Mar 16 15:06:14 UTC 2012 [15:06:52] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CRIT replication delay 202 seconds [15:07:28] PROBLEM - Host cp1021 is DOWN: PING CRITICAL - Packet loss = 100% [15:08:40] PROBLEM - Host cp1022 is DOWN: PING CRITICAL - Packet loss = 100% [15:09:07] RECOVERY - Host cp1021 is UP: PING OK - Packet loss = 0%, RTA = 26.75 ms [15:09:40] !log reedy synchronized stylize.php 'Test for hume' [15:09:43] Logged the message, Master [15:10:37] PROBLEM - Host cp1023 is DOWN: PING CRITICAL - Packet loss = 100% [15:10:46] RECOVERY - Host cp1022 is UP: PING OK - Packet loss = 0%, RTA = 26.58 ms [15:11:22] RECOVERY - Host cp1023 is UP: PING OK - Packet loss = 0%, RTA = 26.46 ms [15:13:46] PROBLEM - MySQL Slave Delay on db1033 is CRITICAL: CRIT replication delay 182 seconds [15:14:13] PROBLEM - Varnish HTTP upload-frontend on cp1021 is CRITICAL: Connection refused [15:15:07] PROBLEM - Varnish traffic logger on cp1022 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [15:15:25] PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 203 seconds [15:15:52] PROBLEM - Varnish traffic logger on cp1026 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [15:16:01] PROBLEM - Varnish traffic logger on cp1023 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [15:16:37] PROBLEM - Varnish traffic logger on cp1024 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [15:16:37] PROBLEM - Varnish HTTP upload-frontend on cp1028 is CRITICAL: Connection refused [15:16:55] PROBLEM - Varnish HTTP upload-frontend on cp1024 is CRITICAL: Connection refused [15:16:55] PROBLEM - Varnish HTTP upload-frontend on cp1025 is CRITICAL: Connection refused [15:16:55] PROBLEM - Varnish HTTP upload-frontend on cp1022 is CRITICAL: Connection refused [15:17:04] PROBLEM - Varnish traffic logger on cp1028 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [15:17:13] PROBLEM - Varnish traffic logger on cp1025 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [15:17:31] PROBLEM - Varnish HTTP upload-frontend on cp1026 is CRITICAL: Connection refused [15:17:40]