[00:12:30] New patchset: Asher; "adding db52/53 to s1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1971 [00:12:45] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1971 [00:12:51] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1971 [00:12:51] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1971 [00:16:25] PROBLEM - Squid on brewster is CRITICAL: Connection refused [00:16:26] PROBLEM - Squid on brewster is CRITICAL: Connection refused [00:20:52] !log changed vlan for virt1 eth0 [00:20:54] Logged the message, Mistress of the network gear. [00:26:15] RECOVERY - Squid on brewster is OK: TCP OK - 0.000 second response time on port 8080 [00:26:16] RECOVERY - Squid on brewster is OK: TCP OK - 0.000 second response time on port 8080 [00:34:37] New patchset: Ryan Lane; "Removing virt1.wikimedia.org and adding virt1.pmtpa.wmnet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1972 [00:34:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1972 [00:40:40] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1972 [00:40:41] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1972 [00:50:35] RECOVERY - Puppet freshness on virt1 is OK: puppet ran at Thu Jan 19 00:50:23 UTC 2012 [00:50:36] RECOVERY - Puppet freshness on virt1 is OK: puppet ran at Thu Jan 19 00:50:23 UTC 2012 [00:54:18] !log running a hot backup of db32, streaming to db52 [00:54:19] Logged the message, Master [01:12:53] New patchset: Asher; "removing extra frontend cache capacity from mobile" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1973 [01:13:08] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1973 [01:13:38] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1973 [01:13:38] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1973 [01:16:14] !log removing extra mobile varnish capacity - it wasn't needed [01:16:16] Logged the message, Master [01:20:11] New patchset: Ryan Lane; "Adding virt1 public cert" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1974 [01:20:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1974 [01:21:42] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1974 [01:21:43] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1974 [01:26:39] RECOVERY - ps1-d2-pmtpa-infeed-load-tower-A-phase-Y on ps1-d2-pmtpa is OK: ps1-d2-pmtpa-infeed-load-tower-A-phase-Y OK - 1200 [01:26:39] RECOVERY - ps1-d2-pmtpa-infeed-load-tower-A-phase-Y on ps1-d2-pmtpa is OK: ps1-d2-pmtpa-infeed-load-tower-A-phase-Y OK - 1200 [01:42:46] PROBLEM - Varnish HTTP mobile-frontend on cp1040 is CRITICAL: Connection refused [01:42:47] PROBLEM - Varnish HTTP mobile-frontend on cp1040 is CRITICAL: Connection refused [01:43:26] ^^^ those no longer run varnish.. need to run puppet on spence [01:46:35] PROBLEM - Varnish HTTP mobile-frontend on cp1039 is CRITICAL: Connection refused [01:46:36] PROBLEM - Varnish HTTP mobile-frontend on cp1039 is CRITICAL: Connection refused [01:46:55] PROBLEM - mobile traffic loggers on cp1040 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [01:46:56] PROBLEM - mobile traffic loggers on cp1040 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [01:49:15] PROBLEM - Varnish HTTP mobile-backend on cp1039 is CRITICAL: Connection refused [01:49:16] PROBLEM - Varnish HTTP mobile-backend on cp1039 is CRITICAL: Connection refused [01:50:20] Ryan_Lane: puppet on spence is failing on err: Could not retrieve catalog from remote server: Error 400 on SERVER: Exported resource Nagios_host[virt1] cannot override local resource on node spence.wikimedia.org [01:50:37] damn it [01:50:45] how do we purge crap from puppet's database again? [01:50:48] don't we have a script? [01:50:53] oh [01:50:55] PROBLEM - mobile traffic loggers on cp1039 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [01:50:55] PROBLEM - mobile traffic loggers on cp1039 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [01:50:55] ACKNOWLEDGEMENT - Varnish HTTP mobile-backend on cp1039 is CRITICAL: Connection refused asher waiting for puppet [01:50:56] ACKNOWLEDGEMENT - Varnish HTTP mobile-backend on cp1039 is CRITICAL: Connection refused asher waiting for puppet [01:51:00] hm [01:51:48] maybe virt1.wikimedia.org needs to get added to decommissioned [01:52:06] bah [01:52:10] it doesn't use FQDN [01:52:18] I need to purge the database [01:52:21] so fucking annoying [01:52:45] PROBLEM - Varnish HTTP mobile-backend on cp1040 is CRITICAL: Connection refused [01:52:46] PROBLEM - Varnish HTTP mobile-backend on cp1040 is CRITICAL: Connection refused [01:53:45] ACKNOWLEDGEMENT - mobile traffic loggers on cp1040 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa asher waiting for puppet [01:53:46] ACKNOWLEDGEMENT - mobile traffic loggers on cp1040 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa asher waiting for puppet [02:00:25] RECOVERY - mobile traffic loggers on cp1044 is OK: PROCS OK: 2 processes with command name varnishncsa [02:00:25] RECOVERY - mobile traffic loggers on cp1044 is OK: PROCS OK: 2 processes with command name varnishncsa [02:17:26] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1321s [02:17:26] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1321s [02:23:35] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1691s [02:23:35] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1691s [02:26:02] binasher: it looks like the big rush on the mobile caches never happened, whew... [02:26:49] yep, could have actually dropped some of the already existing servers instead of adding anything [02:27:48] it was so easy to circumvent the blackout, so no need for people to use the mobile site instead [02:30:05] PROBLEM - ps1-d2-sdtpa-infeed-load-tower-A-phase-Z on ps1-d2-sdtpa is CRITICAL: ps1-d2-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2488* [02:30:05] PROBLEM - ps1-d2-sdtpa-infeed-load-tower-A-phase-Z on ps1-d2-sdtpa is CRITICAL: ps1-d2-sdtpa-infeed-load-tower-A-phase-Z CRITICAL - *2488* [02:33:35] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 14s [02:33:35] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 14s [02:37:55] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:37:55] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [03:21:35] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Thu Jan 19 03:21:14 UTC 2012 [03:21:36] RECOVERY - Puppet freshness on db1045 is OK: puppet ran at Thu Jan 19 03:21:14 UTC 2012 [04:15:45] RECOVERY - Disk space on es1004 is OK: DISK OK [04:15:46] RECOVERY - Disk space on es1004 is OK: DISK OK [04:16:45] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:16:46] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:41:56] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [04:41:57] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [09:06:35] PROBLEM - Puppet freshness on mw1096 is CRITICAL: Puppet has not run in the last 10 hours [09:06:36] PROBLEM - Puppet freshness on mw1096 is CRITICAL: Puppet has not run in the last 10 hours [09:35:09] New patchset: ArielGlenn; "add snapshot1001-4 to site.pp and to download exports list" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1975 [09:41:30] New review: ArielGlenn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1975 [09:41:31] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1975 [09:50:05] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 424992 MB (3% inode=99%): [09:50:05] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 424992 MB (3% inode=99%): [09:50:15] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 424699 MB (3% inode=99%): [09:50:17] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 424699 MB (3% inode=99%): [10:16:35] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [10:16:35] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [10:36:15] RECOVERY - MySQL slave status on es1004 is OK: OK: [10:36:15] RECOVERY - MySQL slave status on es1004 is OK: OK: [10:36:35] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [10:36:36] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [10:41:25] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [10:41:26] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [11:06:20] New review: Dzahn; "looks like this is related to a new puppet problem on fenari:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1972 [11:32:52] New patchset: ArielGlenn; "em.. the new snaps are at equid, add to regexp in site.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1976 [11:37:26] New patchset: ArielGlenn; "em.. the new snaps are at equid, add to regexp in site.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1976 [11:39:48] hey mutante, I have a gerrit amend q if you are around [11:44:39] New review: ArielGlenn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1976 [11:44:40] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1976 [11:44:58] nevermind, figured out what the extra stuff was [12:35:20] hmm people are aware that they're migrating our linode instance right? [12:35:33] so wikitech (and whatever else might be over there) will be unavailable shortly, for a little while [13:20:52] afk for a little bit, running to bank. the heating oil fees are *ridiculous* [14:24:15] PROBLEM - Apache HTTP on srv263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:15] PROBLEM - Apache HTTP on srv263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:25] PROBLEM - Disk space on srv263 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:30:26] PROBLEM - Disk space on srv263 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:35:35] PROBLEM - DPKG on srv263 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:35:35] PROBLEM - RAID on srv263 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:35:36] PROBLEM - DPKG on srv263 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:35:36] PROBLEM - RAID on srv263 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:35:42] Hmm [14:35:54] sync-file is getting more apache errors than usual [14:35:59] and seems to be hanging [14:36:26] PROBLEM - SSH on srv263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:26] Ah [14:36:26] PROBLEM - SSH on srv263 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:33] srv275 timedout [14:45:15] PROBLEM - MySQL slave status on es2 is CRITICAL: CRITICAL: Connected threads = 1199 (1000) [14:45:16] PROBLEM - MySQL slave status on es2 is CRITICAL: CRITICAL: Connected threads = 1199 (1000) [14:45:35] PROBLEM - DPKG on srv259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:45:36] PROBLEM - Disk space on srv275 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:45:37] PROBLEM - DPKG on srv259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:45:37] PROBLEM - Disk space on srv275 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:45:46] PROBLEM - Disk space on srv259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:45:47] PROBLEM - Disk space on srv259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:46:15] PROBLEM - MySQL slave status on es4 is CRITICAL: CRITICAL: Connected threads = 1159 (1000) [14:46:16] PROBLEM - MySQL slave status on es4 is CRITICAL: CRITICAL: Connected threads = 1159 (1000) [14:47:05] PROBLEM - SSH on srv259 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:06] PROBLEM - SSH on srv259 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:55] PROBLEM - Apache HTTP on srv259 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:56] PROBLEM - Apache HTTP on srv259 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:25] PROBLEM - RAID on srv275 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:50:26] PROBLEM - RAID on srv275 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:50:45] PROBLEM - RAID on srv259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:50:48] PROBLEM - RAID on srv259 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:50:56] PROBLEM - SSH on srv275 is CRITICAL: Server answer: [14:50:57] PROBLEM - SSH on srv275 is CRITICAL: Server answer: [14:51:35] PROBLEM - SSH on srv286 is CRITICAL: Server answer: [14:51:36] PROBLEM - SSH on srv286 is CRITICAL: Server answer: [14:52:15] PROBLEM - DPKG on srv275 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:52:16] PROBLEM - DPKG on srv275 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:52:25] PROBLEM - Disk space on srv286 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:52:26] PROBLEM - Disk space on srv286 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:53:15] PROBLEM - Apache HTTP on srv275 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:53:16] PROBLEM - Apache HTTP on srv275 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:40] I don't really know why these hosts should be backed up and no others [14:55:07] (well 2 others, but the rest are fine) [14:55:57] PROBLEM - DPKG on srv286 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:55:58] PROBLEM - DPKG on srv286 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:57:55] PROBLEM - RAID on srv286 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:57:56] PROBLEM - RAID on srv286 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:58:55] PROBLEM - Apache HTTP on srv286 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:56] PROBLEM - Apache HTTP on srv286 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:35] !log Labs LDAP seems down [15:01:45] RECOVERY - SSH on srv286 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:01:46] RECOVERY - SSH on srv286 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:02:25] RECOVERY - Disk space on srv286 is OK: DISK OK [15:02:26] RECOVERY - Disk space on srv286 is OK: DISK OK [15:03:35] PROBLEM - DPKG on srv261 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:03:35] PROBLEM - Apache HTTP on srv230 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:36] PROBLEM - DPKG on srv261 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:03:36] PROBLEM - Apache HTTP on srv230 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:45] PROBLEM - Apache HTTP on srv211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:46] PROBLEM - Apache HTTP on srv211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:57] PROBLEM - Apache HTTP on mw54 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:58] PROBLEM - Apache HTTP on mw54 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:45] PROBLEM - Apache HTTP on mw30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:45] PROBLEM - Apache HTTP on srv270 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:46] PROBLEM - Apache HTTP on mw30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:46] PROBLEM - Apache HTTP on srv270 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:05] PROBLEM - Apache HTTP on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:06] PROBLEM - Apache HTTP on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:15] PROBLEM - Apache HTTP on mw28 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:16] PROBLEM - Apache HTTP on mw28 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:25] PROBLEM - Disk space on srv261 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:05:26] PROBLEM - Disk space on srv261 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:05:55] PROBLEM - Apache HTTP on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:55] RECOVERY - DPKG on srv286 is OK: All packages OK [15:05:56] PROBLEM - Apache HTTP on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:56] RECOVERY - DPKG on srv286 is OK: All packages OK [15:06:05] PROBLEM - Apache HTTP on srv190 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:06] PROBLEM - Apache HTTP on srv190 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:25] PROBLEM - Apache HTTP on srv205 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:25] PROBLEM - Apache HTTP on mw35 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:25] PROBLEM - Apache HTTP on mw42 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:26] PROBLEM - Apache HTTP on srv205 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:26] PROBLEM - Apache HTTP on mw35 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:26] PROBLEM - Apache HTTP on mw42 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:45] PROBLEM - Apache HTTP on srv271 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:46] PROBLEM - Apache HTTP on srv271 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:25] PROBLEM - Apache HTTP on srv261 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:26] PROBLEM - Apache HTTP on srv261 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:45] RECOVERY - RAID on srv286 is OK: OK: no RAID installed [15:07:45] PROBLEM - Apache HTTP on mw55 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:45] PROBLEM - RAID on srv279 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:07:46] RECOVERY - RAID on srv286 is OK: OK: no RAID installed [15:07:46] PROBLEM - Apache HTTP on mw55 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:46] PROBLEM - RAID on srv279 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:07:55] PROBLEM - Apache HTTP on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:55] PROBLEM - Apache HTTP on srv244 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:56] PROBLEM - Apache HTTP on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:56] PROBLEM - Apache HTTP on srv244 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:05] PROBLEM - Apache HTTP on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:06] PROBLEM - Apache HTTP on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:25] PROBLEM - Apache HTTP on mw27 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:26] PROBLEM - Apache HTTP on mw27 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:45] PROBLEM - Apache HTTP on srv279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:46] PROBLEM - Apache HTTP on srv279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:56] PROBLEM - SSH on srv279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:08:57] PROBLEM - SSH on srv279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:25] RECOVERY - Apache HTTP on srv286 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.063 second response time [15:09:26] RECOVERY - Apache HTTP on srv286 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.063 second response time [15:09:55] PROBLEM - DPKG on srv279 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:09:56] PROBLEM - DPKG on srv279 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:10:15] PROBLEM - Apache HTTP on srv232 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:16] PROBLEM - Apache HTTP on srv232 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:25] PROBLEM - Disk space on srv279 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:12:26] PROBLEM - Disk space on srv279 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:13:35] RECOVERY - DPKG on srv261 is OK: All packages OK [15:13:36] RECOVERY - DPKG on srv261 is OK: All packages OK [15:13:47] RECOVERY - Apache HTTP on srv211 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [15:13:47] RECOVERY - Apache HTTP on srv230 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.040 second response time [15:13:48] RECOVERY - Apache HTTP on srv211 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [15:13:48] RECOVERY - Apache HTTP on srv230 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.040 second response time [15:14:55] RECOVERY - Apache HTTP on srv270 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.698 second response time [15:14:56] RECOVERY - Apache HTTP on srv270 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 5.698 second response time [15:15:15] RECOVERY - Apache HTTP on mw28 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [15:15:16] RECOVERY - Apache HTTP on mw28 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [15:15:25] RECOVERY - Disk space on srv261 is OK: DISK OK [15:15:26] RECOVERY - Disk space on srv261 is OK: DISK OK [15:15:45] PROBLEM - MySQL slave status on es2 is CRITICAL: CRITICAL: Connected threads = 1016 (1000) [15:15:46] PROBLEM - MySQL slave status on es2 is CRITICAL: CRITICAL: Connected threads = 1016 (1000) [15:16:25] RECOVERY - Apache HTTP on srv205 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [15:16:26] RECOVERY - Apache HTTP on srv205 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [15:16:35] PROBLEM - SSH on srv268 is CRITICAL: Server answer: [15:16:36] PROBLEM - SSH on srv268 is CRITICAL: Server answer: [15:17:25] RECOVERY - Apache HTTP on srv261 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.037 second response time [15:17:26] RECOVERY - Apache HTTP on srv261 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 3.037 second response time [15:17:45] RECOVERY - Apache HTTP on mw55 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.035 second response time [15:17:46] RECOVERY - Apache HTTP on mw55 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.035 second response time [15:17:55] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.032 second response time [15:17:56] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.032 second response time [15:18:05] RECOVERY - Apache HTTP on mw1 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.401 second response time [15:18:06] RECOVERY - Apache HTTP on mw1 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.401 second response time [15:18:25] RECOVERY - Apache HTTP on mw27 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [15:18:26] RECOVERY - Apache HTTP on mw27 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [15:18:45] PROBLEM - Disk space on srv268 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:18:45] PROBLEM - DPKG on srv268 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:18:46] PROBLEM - Disk space on srv268 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:18:46] PROBLEM - DPKG on srv268 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:20:15] RECOVERY - Apache HTTP on srv232 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.029 second response time [15:20:16] RECOVERY - Apache HTTP on srv232 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.029 second response time [15:21:45] PROBLEM - RAID on srv268 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:21:46] PROBLEM - RAID on srv268 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:22:16] PROBLEM - Apache HTTP on srv268 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:16] PROBLEM - Apache HTTP on srv268 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:35] RECOVERY - Apache HTTP on mw54 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.055 second response time [15:24:36] RECOVERY - Apache HTTP on mw54 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.055 second response time [15:24:55] RECOVERY - Apache HTTP on mw30 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [15:24:56] RECOVERY - Apache HTTP on mw30 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [15:25:37] RECOVERY - Apache HTTP on mw11 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.029 second response time [15:25:38] RECOVERY - Apache HTTP on mw11 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.029 second response time [15:25:55] RECOVERY - Apache HTTP on mw12 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [15:25:56] RECOVERY - Apache HTTP on mw12 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [15:26:05] RECOVERY - Apache HTTP on srv190 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.972 second response time [15:26:06] RECOVERY - Apache HTTP on srv190 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 4.972 second response time [15:26:25] RECOVERY - Apache HTTP on mw35 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.038 second response time [15:26:26] RECOVERY - Apache HTTP on mw35 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.038 second response time [15:26:37] RECOVERY - Apache HTTP on mw42 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.616 second response time [15:26:37] RECOVERY - Apache HTTP on mw42 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.616 second response time [15:26:45] RECOVERY - Apache HTTP on srv271 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [15:26:46] RECOVERY - Apache HTTP on srv271 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [15:26:55] PROBLEM - MySQL slave status on es4 is CRITICAL: CRITICAL: Connected threads = 1038 (1000) [15:26:56] PROBLEM - MySQL slave status on es4 is CRITICAL: CRITICAL: Connected threads = 1038 (1000) [15:27:55] RECOVERY - Apache HTTP on srv244 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.033 second response time [15:27:56] RECOVERY - Apache HTTP on srv244 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.033 second response time [15:32:26] PROBLEM - Memcached on srv279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:27] PROBLEM - Memcached on srv279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:36:55] RECOVERY - DPKG on srv263 is OK: All packages OK [15:36:56] RECOVERY - DPKG on srv263 is OK: All packages OK [15:37:05] RECOVERY - Apache HTTP on srv263 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.601 second response time [15:37:06] RECOVERY - Apache HTTP on srv263 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.601 second response time [15:39:36] RECOVERY - SSH on srv263 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:39:36] RECOVERY - SSH on srv263 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:41:15] RECOVERY - Disk space on srv263 is OK: DISK OK [15:41:16] RECOVERY - Disk space on srv263 is OK: DISK OK [15:45:55] RECOVERY - MySQL slave status on es2 is OK: OK: [15:45:56] RECOVERY - MySQL slave status on es2 is OK: OK: [15:46:05] RECOVERY - Disk space on srv275 is OK: DISK OK [15:46:05] RECOVERY - DPKG on srv259 is OK: All packages OK [15:46:06] RECOVERY - Disk space on srv275 is OK: DISK OK [15:46:06] RECOVERY - DPKG on srv259 is OK: All packages OK [15:46:15] RECOVERY - Disk space on srv259 is OK: DISK OK [15:46:15] RECOVERY - RAID on srv263 is OK: OK: no RAID installed [15:46:16] RECOVERY - Disk space on srv259 is OK: DISK OK [15:46:16] RECOVERY - RAID on srv263 is OK: OK: no RAID installed [15:47:07] RECOVERY - MySQL slave status on es4 is OK: OK: [15:47:08] RECOVERY - MySQL slave status on es4 is OK: OK: [15:47:55] RECOVERY - RAID on srv279 is OK: OK: no RAID installed [15:47:56] RECOVERY - RAID on srv279 is OK: OK: no RAID installed [15:48:05] RECOVERY - SSH on srv259 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:48:06] RECOVERY - SSH on srv259 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:49:15] RECOVERY - SSH on srv279 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:49:16] RECOVERY - SSH on srv279 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:50:55] RECOVERY - DPKG on srv279 is OK: All packages OK [15:50:56] RECOVERY - DPKG on srv279 is OK: All packages OK [15:51:05] RECOVERY - RAID on srv275 is OK: OK: no RAID installed [15:51:06] RECOVERY - RAID on srv275 is OK: OK: no RAID installed [15:51:25] RECOVERY - SSH on srv275 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:51:26] RECOVERY - SSH on srv275 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:51:45] RECOVERY - RAID on srv259 is OK: OK: no RAID installed [15:51:46] RECOVERY - RAID on srv259 is OK: OK: no RAID installed [15:52:05] RECOVERY - RAID on srv268 is OK: OK: no RAID installed [15:52:06] RECOVERY - RAID on srv268 is OK: OK: no RAID installed [15:52:25] RECOVERY - Memcached on srv279 is OK: TCP OK - 0.002 second response time on port 11000 [15:52:26] RECOVERY - Memcached on srv279 is OK: TCP OK - 0.002 second response time on port 11000 [15:52:46] RECOVERY - DPKG on srv275 is OK: All packages OK [15:52:47] RECOVERY - DPKG on srv275 is OK: All packages OK [15:53:05] RECOVERY - Disk space on srv279 is OK: DISK OK [15:53:06] RECOVERY - Disk space on srv279 is OK: DISK OK [15:58:45] RECOVERY - DPKG on srv268 is OK: All packages OK [15:58:45] RECOVERY - Disk space on srv268 is OK: DISK OK [15:58:46] RECOVERY - DPKG on srv268 is OK: All packages OK [15:58:46] RECOVERY - Disk space on srv268 is OK: DISK OK [15:58:55] RECOVERY - Apache HTTP on srv279 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [15:58:56] RECOVERY - Apache HTTP on srv279 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [15:59:15] RECOVERY - SSH on srv268 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:59:16] RECOVERY - SSH on srv268 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [16:12:35] RECOVERY - Apache HTTP on srv268 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [16:12:36] RECOVERY - Apache HTTP on srv268 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [16:14:45] RECOVERY - Apache HTTP on srv275 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [16:14:46] RECOVERY - Apache HTTP on srv275 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [16:20:37] New patchset: Mark Bergsma; "Revert "Google and possibly others are rate limiting our new ip, so use the old server(s) for delayed messages (for now)"" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1977 [16:20:51] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1977 [16:20:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1977 [16:20:56] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1977 [16:20:57] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1977 [16:21:05] RECOVERY - Apache HTTP on srv259 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [16:21:06] RECOVERY - Apache HTTP on srv259 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [16:42:21] New patchset: Mark Bergsma; "Update MTA hostnames" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1978 [16:42:37] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1978 [16:43:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1978 [16:43:09] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1978 [16:45:35] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [16:45:35] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [16:52:39] !log Configured lily to send all mail to sodium to empty its mail spool, preparing for decommissioning [17:36:17] where are apache configs stored in git? [17:37:56] ah... https://gerrit.wikimedia.org/r/#change,1842 has it, I think [17:44:21] hexmode: noc.wm.o/conf mostly [17:55:27] i'm thinking if there's just a couple files changed or there's a few that have a common path, then gerrit-wm should say what those are [18:01:23] also, why does gerrit have a 1024 bit key? ;( [18:06:22] that seems tiny by today's standards [18:06:56] agreed [18:32:30] apergos: chris mentioned you needed something from me? [18:32:36] heya [18:32:54] yeah on snapshot1003 and 1004 I am seeing this error when I try to connect to mgmt: [18:33:03] Create CLPSession Instance Error.(1) CURL error: 7 [18:33:10] and then it slams the door. [18:33:11] awesome, its a firmware bug [18:33:13] pretty rude [18:33:23] can you kick them or something? :-D [18:34:24] it requires a firmware update, lemme see if i cannot get them fixed for ya real quick [18:34:32] that would be awesome [18:34:41] they get hung in some odd state [18:34:47] where a server that was workign, then gets curl error [18:34:49] its fubar. [18:34:50] I got the two others installed, and when you are not at eqiad I wanna talk about the paritioning. (i.e. not today) [18:34:56] yuck [18:35:04] 1.54 (Build 15) [18:35:11] we update to 1.7 build20 or soemthing [18:35:33] ahh damn it [18:35:45] wrong server, stupid proxy [18:35:54] hrmm, getting odder error on snapshot1003 [18:36:13] and same odd one on the other, lemme attach a console [18:36:20] apergos: i take it back, its some different error ;] [18:36:56] you wanna power one off hard and back on, see if we get anything better? [18:40:36] yea, setting up crash cart and the like [18:42:05] ok, thanks a lot dude [18:45:30] ok, poking at snapshot1003, trying just a plain power removal and restore first [18:45:34] before digging into it [18:47:53] huh, power restore and i can get into drac via ssh, but http has the errors [18:47:56] interestin [18:48:45] messing with it still [18:48:53] ok [18:50:52] its newest firmware already [18:51:00] well, newer than the curl error, which i am no longer getting [18:51:09] but it is having some drac http error, which is still an issue [18:51:37] ok [18:52:16] resetting drac to factory and redoing [18:52:50] ok [18:52:57] thanks for looking at this [18:53:05] a google search turned up a big fat nothing [19:00:22] oh god damn it. [19:00:28] it turns out this is a firefox error [19:00:31]