[00:18:33] blergh [00:18:37] found it, finally [00:19:02] 1.8.0.rc1~58^2 [01:57:16] (03PS1) 10Jforrester: Clean up how VisualEditor is configured [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87644 [01:57:17] (03PS1) 10Jforrester: Switch VisualEditor to secondary status on hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87645 [02:06:38] !log LocalisationUpdate completed (1.22wmf19) at Sat Oct 5 02:06:38 UTC 2013 [02:06:56] Logged the message, Master [02:19:16] !log LocalisationUpdate completed (1.22wmf20) at Sat Oct 5 02:19:16 UTC 2013 [02:19:27] Logged the message, Master [02:25:46] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 5 02:25:46 UTC 2013 [02:25:58] Logged the message, Master [03:49:07] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:49:57] RECOVERY - MySQL Processlist on db1021 is OK: OK 1 unauthenticated, 0 locked, 0 copy to table, 7 statistics [04:45:23] who [04:46:08] Is someone on the equinix ling being down? I don't think I have creds to the routers. [06:56:11] icinga-wm being so silent is very suspect [06:56:32] I reduced my downloaders but someone is not being equally nice to swift https://ganglia.wikimedia.org/latest/?r=week&cs=&ce=&m=network_report&s=by+name&c=Swift+pmtpa&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [07:01:32] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:02:22] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [07:05:25] hmm "starting swiftrepl process for pmtpa->eqiad (originals), running on copper " probably skews the numbers a bit :) [07:47:09] (03CR) 10Nikerabbit: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86889 (owner: 10Matanya) [08:27:39] PROBLEM - Disk space on labsdb1003 is CRITICAL: DISK CRITICAL - free space: /a 113619 MB (3% inode=99%): [09:13:05] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [09:22:25] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 15 copy to table, 273 statistics [09:23:25] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics [09:26:45] PROBLEM - MySQL Slave Delay on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:27:35] RECOVERY - MySQL Slave Delay on db1021 is OK: OK replication delay 16 seconds [09:28:35] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:29:25] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 26 copy to table, 1 statistics [09:35:25] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 65 copy to table, 4 statistics [09:41:28] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 3 copy to table, 5 statistics [10:14:26] PROBLEM - MySQL Idle Transactions on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:15:15] RECOVERY - MySQL Idle Transactions on db1021 is OK: OK longest blocking idle transaction sleeps for 0 seconds [10:21:35] PROBLEM - Disk space on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:21:36] PROBLEM - DPKG on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:21:36] PROBLEM - MySQL disk space on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:22:25] RECOVERY - Disk space on db1021 is OK: DISK OK [10:22:25] RECOVERY - MySQL disk space on db1021 is OK: DISK OK [10:22:25] RECOVERY - DPKG on db1021 is OK: All packages OK [10:25:35] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:26:35] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [11:35:19] PROBLEM - MySQL Replication Heartbeat on db53 is CRITICAL: CRIT replication delay 305 seconds [11:35:19] PROBLEM - MySQL Slave Delay on db53 is CRITICAL: CRIT replication delay 306 seconds [11:40:19] RECOVERY - MySQL Replication Heartbeat on db53 is OK: OK replication delay 69 seconds [11:40:19] RECOVERY - MySQL Slave Delay on db53 is OK: OK replication delay 66 seconds [13:12:04] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [13:12:24] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.59 ms [13:15:24] PROBLEM - Apache HTTP on mw31 is CRITICAL: Connection refused [13:16:24] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.250 second response time [13:21:14] PROBLEM - Host mw1085 is DOWN: PING CRITICAL - Packet loss = 100% [13:22:24] RECOVERY - Host mw1085 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [13:24:34] PROBLEM - Apache HTTP on mw1085 is CRITICAL: Connection refused [13:25:35] RECOVERY - Apache HTTP on mw1085 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.077 second response time [14:23:36] RECOVERY - search indices - check lucene status page on search19 is OK: HTTP OK: HTTP/1.1 200 OK - 60075 bytes in 0.117 second response time [16:08:28] PROBLEM - MySQL Idle Transactions on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:09:28] RECOVERY - MySQL Idle Transactions on db1021 is OK: OK longest blocking idle transaction sleeps for 0 seconds [16:09:48] PROBLEM - MySQL Slave Delay on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:10:39] RECOVERY - MySQL Slave Delay on db1021 is OK: OK replication delay 21 seconds [16:16:29] PROBLEM - MySQL disk space on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:16:29] PROBLEM - mysqld processes on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:16:29] PROBLEM - Full LVS Snapshot on db1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:17:19] RECOVERY - MySQL disk space on db1021 is OK: DISK OK [16:17:20] RECOVERY - Full LVS Snapshot on db1021 is OK: OK no full LVM snapshot volumes [16:17:20] RECOVERY - mysqld processes on db1021 is OK: PROCS OK: 1 process with command name mysqld [17:29:04] (03CR) 10MZMcBride: "Chad: any idea when you might be able to do that? :-)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/84743 (owner: 10QChris) [17:38:05] (03PS1) 10Yuvipanda: Add SPDY support to dynamicproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/87682 [17:38:13] (03CR) 10jenkins-bot: [V: 04-1] Add SPDY support to dynamicproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/87682 (owner: 10Yuvipanda) [17:38:29] (03PS2) 10Yuvipanda: Add SPDY support to dynamicproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/87682 [17:38:40] anyone to +2 that trivial patch? [17:43:26] * YuviPanda optimistically pings apergos [18:49:38] RECOVERY - Disk space on labsdb1003 is OK: DISK OK [19:13:18] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours