[00:01:20] New patchset: Asher; "class to install percona nagios monitors (just the files so far)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1723 [00:01:40] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1723 [00:05:20] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1723 [00:05:20] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1723 [00:40:34] LeslieCarr: Can you pop in #mediawiki for a second if you're not busy, I need your help [00:41:10] okay [00:48:36] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [01:03:06] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [01:04:01] New patchset: Lcarr; "Removing star from planet as it is already defined" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1724 [01:04:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1724 [01:04:21] anyone around to review ? [01:05:41] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1724 [01:05:41] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1724 [01:06:46] RECOVERY - Puppet freshness on singer is OK: puppet ran at Wed Dec 28 01:06:40 UTC 2011 [01:07:14] yay singer is happy again [02:13:38] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1667s [02:21:48] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 2157s [02:31:49] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:32:58] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [03:42:43] PROBLEM - mobile traffic loggers on cp1041 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [03:42:43] PROBLEM - mobile traffic loggers on cp1043 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [03:42:43] PROBLEM - mobile traffic loggers on cp1042 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [03:52:33] RECOVERY - mobile traffic loggers on cp1042 is OK: PROCS OK: 4 processes with args varnishncsa [04:02:13] RECOVERY - mobile traffic loggers on cp1043 is OK: PROCS OK: 3 processes with args varnishncsa [04:11:35] RECOVERY - mobile traffic loggers on cp1041 is OK: PROCS OK: 3 processes with args varnishncsa [04:36:55] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [06:59:19] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 272 MB (3% inode=60%): /var/lib/ureadahead/debugfs 272 MB (3% inode=60%): [07:09:19] RECOVERY - Disk space on srv221 is OK: DISK OK [07:29:18] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1687 [07:34:18] RECOVERY - Disk space on hume is OK: DISK OK [07:35:38] New patchset: Dzahn; "additional generic check_procs with -C option & fix "mobile traffic logger" checks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1687 [07:35:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1687 [07:36:16] !log live-hacked /usr/local/bin/copy_impression_logs_from_storage3.pl on hume, it was rsyncing everything into /a/static/uncompressed/2... do we need this job? there is also /usr/local/bin/offhost_backups on storage3 that seems to copy to the same dir, can whoever set this up take a look? [07:36:26] Logged the message, Master [07:36:26] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1687 [07:36:26] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1687 [07:38:00] morning apergos, just fixing that nagios check for mobile traffic loggers [07:38:08] yay [07:39:08] well in another 2 minutes I'll know if my typo fix on hume worked out [07:39:33] it would be nice not to have to panic about space over there for awhile [07:40:23] cool [07:40:31] guess I'm going to ask for some of those juniper credits [07:41:03] I don't necessarily want to become a juniper network equipment expert but if they have courses that teach more about networking that would be fine [07:42:08] yes, totally [07:43:13] yep looks like that took care of it... [07:44:54] nice apergos [07:46:38] and yes, i think it does teach a lot about networking [07:47:04] great [07:47:06] a friend of mine did the JNCIE cert, and i remember how he was learning for it, and all genereal routing and switching protocols [07:47:12] like here: http://www.juniper.net/us/en/training/certification/e_track.html#jncieent [07:49:10] hehe, a volunteer made this: class nagios::monitor::check_wiki_user_last_edit_time [07:49:18] monitor wiki users via Nagios? :p [07:49:54] !change 1712 [07:49:54] https://gerrit.wikimedia.org/r/1712 [07:49:56] :-D [07:51:27] guess I should look at those more closely later [07:52:25] do you already know details about the juniper credits? [07:52:39] like which classes you can use them for / how many / ... [08:13:57] apergos: found something out about puppet / nagios / duplicate service checks ..from official docs: [08:14:07] "You can purge Nagios resources using the resources type, but only in the default file locations. This is an architectural limitation." [08:15:51] I knew that. hmph [08:15:54] "By default, the statements will be added to /etc/nagios/nagios_service.cfg, but you can send them to a different file by setting their target attribute." [08:16:05] so since we don't use the default, we can't purge..ack [08:17:13] i thought we had something set up to work around that [08:17:56] somebody deleted .cfg files in between it seems [08:18:20] before: thousands of lines, after: just hundreds [08:18:38] huh [08:19:23] the work-around i would know about was just to fix permissions [08:19:42] that used to break nagios, and doesn't anymore [08:20:41] oh, yeah, and we have the purge script that is being started from the init script [08:20:59] what about that? [08:21:39] it uses NagiosPurgeFiles= .. and that has among other paths, also /etc/nagios/puppet_checks.d/* [08:22:51] it is not made for the job "remove duplicate service definitions for existing hosts", just for "remove services for hosts which do not exist" [08:22:58] bah [08:23:38] maybe it should just delete all those files ..hmm [08:24:28] hmm..or better, puppet should delete right before it re-creates each one [08:25:00] (or go back to one huge nagios_service.cfg file) [08:25:15] eww not one huge one [08:25:31] if it's going to create a new file then itcould move the old one out of the way [08:25:53] (this way after a test run you can compare the old and new one by hand and see if it sucks or you got what you wanted) [08:28:21] yeah,hmm, just that the file is created by the built-in "@@nagios_service" type, and not by a "file" ..checking [08:49:38] New patchset: Dzahn; "nagios: purge resources using puppet (instead of .py script), comments on duplicate definitions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1726 [08:49:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1726 [09:06:42] New patchset: Dzahn; "nagios: purge resources using puppet (instead of .py script), try to avoid duplicate definitions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1726 [09:54:03] RECOVERY - MySQL slave status on es1004 is OK: OK: [10:00:22] PROBLEM - mobile traffic loggers on cp1043 is CRITICAL: PROCS CRITICAL: 6 processes with args varnishncsa [10:00:22] PROBLEM - mobile traffic loggers on cp1041 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [10:00:22] PROBLEM - mobile traffic loggers on cp1042 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [10:28:55] RECOVERY - mobile traffic loggers on cp1043 is OK: PROCS OK: 2 processes with args varnishncsa [10:55:52] PROBLEM - mobile traffic loggers on cp1043 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [10:55:52] PROBLEM - mobile traffic loggers on cp1044 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [11:00:45] still weird..that's not what you get when checking manually..sigh [11:12:42] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [12:04:16] PROBLEM - Puppet freshness on srv191 is CRITICAL: Puppet has not run in the last 10 hours [12:47:27] PROBLEM - Auth DNS on ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:47:34] New patchset: Dzahn; "nagios - check procs via nrpe -checkcommands - can't use (broken) generic check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1727 [12:47:50] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1727 [12:48:47] New patchset: Dzahn; "nagios - check procs via nrpe -checkcommands - can't use (broken) generic check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1727 [12:49:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1727 [12:50:09] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [12:50:27] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1727 [12:50:27] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1727 [12:57:32] New patchset: Dzahn; "nagios proc checks - use nrpe commands, this just seemed to work..duh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1728 [12:57:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1728 [13:02:43] New patchset: Dzahn; "add nrpe to tarin and cp104[1-4] for process checks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1729 [13:02:56] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1729 [13:08:35] RECOVERY - Auth DNS on ns0.wikimedia.org is OK: DNS OK: 6.477 seconds response time. www.wikipedia.org returns 208.80.152.201 [13:15:10] PROBLEM - RAID on db1009 is CRITICAL: Connection refused by host [13:16:51] PROBLEM - Disk space on mw1113 is CRITICAL: Connection refused by host [13:17:05] PROBLEM - Disk space on srv225 is CRITICAL: Connection refused by host [13:17:11] PROBLEM - Disk space on mw1157 is CRITICAL: Connection refused by host [13:17:11] PROBLEM - DPKG on srv259 is CRITICAL: Connection refused by host [13:17:32] PROBLEM - Disk space on srv259 is CRITICAL: Connection refused by host [13:18:42] PROBLEM - RAID on es1003 is CRITICAL: Connection refused by host [13:19:51] PROBLEM - RAID on mw1015 is CRITICAL: Connection refused by host [13:20:32] PROBLEM - DPKG on ms1002 is CRITICAL: Connection refused by host [13:21:44] PROBLEM - MySQL disk space on es1003 is CRITICAL: Connection refused by host [13:21:54] PROBLEM - RAID on srv229 is CRITICAL: Connection refused by host [13:22:23] PROBLEM - Disk space on es1003 is CRITICAL: Connection refused by host [13:22:23] PROBLEM - DPKG on srv274 is CRITICAL: Connection refused by host [13:22:32] PROBLEM - Disk space on aluminium is CRITICAL: Connection refused by host [13:22:32] PROBLEM - DPKG on db1019 is CRITICAL: Connection refused by host [13:22:52] PROBLEM - DPKG on nfs1 is CRITICAL: Connection refused by host [13:22:52] PROBLEM - DPKG on mw1015 is CRITICAL: Connection refused by host [13:23:02] PROBLEM - DPKG on mw1088 is CRITICAL: Connection refused by host [13:23:22] PROBLEM - DPKG on db47 is CRITICAL: Connection refused by host [13:23:22] PROBLEM - RAID on mw1129 is CRITICAL: Connection refused by host [13:23:32] PROBLEM - RAID on mw1104 is CRITICAL: Connection refused by host [13:23:32] PROBLEM - RAID on mw1157 is CRITICAL: Connection refused by host [13:23:42] PROBLEM - Disk space on virt3 is CRITICAL: Connection refused by host [13:23:54] PROBLEM - RAID on srv259 is CRITICAL: Connection refused by host [13:24:02] PROBLEM - RAID on nfs1 is CRITICAL: Connection refused by host [13:24:24] PROBLEM - RAID on srv208 is CRITICAL: Connection refused by host [13:24:34] PROBLEM - MySQL disk space on db1019 is CRITICAL: Connection refused by host [13:24:42] PROBLEM - RAID on mw1112 is CRITICAL: Connection refused by host [13:24:42] PROBLEM - Disk space on db47 is CRITICAL: Connection refused by host [13:24:42] PROBLEM - Disk space on db1019 is CRITICAL: Connection refused by host [13:25:02] PROBLEM - DPKG on db1002 is CRITICAL: Connection refused by host [13:25:50] PROBLEM - Disk space on mw1088 is CRITICAL: Connection refused by host [13:28:10] PROBLEM - RAID on mw29 is CRITICAL: Connection refused by host [13:28:10] PROBLEM - Disk space on mw1076 is CRITICAL: Connection refused by host [13:28:33] PROBLEM - RAID on srv211 is CRITICAL: Connection refused by host [13:28:41] PROBLEM - Disk space on db1002 is CRITICAL: Connection refused by host [13:28:59] PROBLEM - MySQL disk space on db18 is CRITICAL: Connection refused by host [13:28:59] PROBLEM - RAID on srv240 is CRITICAL: Connection refused by host [13:29:09] PROBLEM - DPKG on db11 is CRITICAL: Connection refused by host [13:29:09] PROBLEM - RAID on srv235 is CRITICAL: Connection refused by host [13:29:18] PROBLEM - RAID on db16 is CRITICAL: Connection refused by host [13:29:49] PROBLEM - DPKG on mw1141 is CRITICAL: Connection refused by host [13:30:17] PROBLEM - DPKG on mw1116 is CRITICAL: Connection refused by host [13:30:18] PROBLEM - DPKG on mw1127 is CRITICAL: Connection refused by host [13:30:49] PROBLEM - Disk space on srv227 is CRITICAL: Connection refused by host [13:30:59] PROBLEM - DPKG on mw1144 is CRITICAL: Connection refused by host [13:31:09] PROBLEM - Disk space on db1022 is CRITICAL: Connection refused by host [13:31:10] PROBLEM - DPKG on snapshot4 is CRITICAL: Connection refused by host [13:31:20] PROBLEM - RAID on db1033 is CRITICAL: Connection refused by host [13:31:20] PROBLEM - DPKG on db1033 is CRITICAL: Connection refused by host [13:31:20] PROBLEM - DPKG on db1022 is CRITICAL: Connection refused by host [13:31:20] RECOVERY - RAID on db1009 is OK: OK: State is Optimal, checked 2 logical device(s) [13:31:30] PROBLEM - Disk space on mw1116 is CRITICAL: Connection refused by host [13:31:59] PROBLEM - Disk space on db25 is CRITICAL: Connection refused by host [13:31:59] PROBLEM - Disk space on mw1011 is CRITICAL: Connection refused by host [13:32:20] PROBLEM - Disk space on db34 is CRITICAL: Connection refused by host [13:32:23] PROBLEM - Disk space on srv273 is CRITICAL: Connection refused by host [13:32:38] PROBLEM - Disk space on mw1158 is CRITICAL: Connection refused by host [13:32:38] PROBLEM - Disk space on mw1144 is CRITICAL: Connection refused by host [13:32:38] RECOVERY - Disk space on mw1113 is OK: DISK OK [13:32:59] RECOVERY - Disk space on srv225 is OK: DISK OK [13:33:00] RECOVERY - Disk space on mw1157 is OK: DISK OK [13:33:00] RECOVERY - DPKG on srv259 is OK: All packages OK [13:33:11] PROBLEM - MySQL disk space on db34 is CRITICAL: Connection refused by host [13:33:11] PROBLEM - Disk space on srv230 is CRITICAL: Connection refused by host [13:33:20] RECOVERY - Disk space on srv259 is OK: DISK OK [13:33:20] PROBLEM - Disk space on mw1135 is CRITICAL: Connection refused by host [13:33:33] PROBLEM - DPKG on db13 is CRITICAL: Connection refused by host [13:33:42] PROBLEM - RAID on db25 is CRITICAL: Connection refused by host [13:33:42] PROBLEM - RAID on db1035 is CRITICAL: Connection refused by host [13:34:57] PROBLEM - RAID on db13 is CRITICAL: Connection refused by host [13:34:57] RECOVERY - RAID on es1003 is OK: OK: State is Optimal, checked 2 logical device(s) [13:35:16] RECOVERY - RAID on mw1015 is OK: OK: no RAID installed [13:35:26] RECOVERY - DPKG on ms1002 is OK: All packages OK [13:35:26] PROBLEM - DPKG on searchidx2 is CRITICAL: Connection refused by host [13:35:26] RECOVERY - MySQL disk space on es1003 is OK: DISK OK [13:35:35] RECOVERY - RAID on srv229 is OK: OK: no RAID installed [13:35:45] PROBLEM - Disk space on db13 is CRITICAL: Connection refused by host [13:35:45] PROBLEM - Disk space on es1001 is CRITICAL: Connection refused by host [13:36:06] RECOVERY - DPKG on srv274 is OK: All packages OK [13:36:06] RECOVERY - Disk space on es1003 is OK: DISK OK [13:36:06] PROBLEM - MySQL disk space on es1001 is CRITICAL: Connection refused by host [13:36:06] RECOVERY - Disk space on aluminium is OK: DISK OK [13:36:16] PROBLEM - RAID on mw1065 is CRITICAL: Connection refused by host [13:36:16] PROBLEM - RAID on srv283 is CRITICAL: Connection refused by host [13:36:16] PROBLEM - RAID on db1006 is CRITICAL: Connection refused by host [13:36:16] RECOVERY - DPKG on db1019 is OK: All packages OK [13:36:27] PROBLEM - DPKG on mw15 is CRITICAL: Connection refused by host [13:36:27] RECOVERY - DPKG on nfs1 is OK: All packages OK [13:36:35] PROBLEM - RAID on srv215 is CRITICAL: Connection refused by host [13:36:46] RECOVERY - DPKG on mw1015 is OK: All packages OK [13:36:46] RECOVERY - DPKG on mw1088 is OK: All packages OK [13:36:56] PROBLEM - DPKG on srv215 is CRITICAL: Connection refused by host [13:37:06] PROBLEM - RAID on db48 is CRITICAL: Connection refused by host [13:37:06] PROBLEM - DPKG on srv196 is CRITICAL: Connection refused by host [13:37:06] PROBLEM - RAID on srv234 is CRITICAL: Connection refused by host [13:37:06] RECOVERY - RAID on mw1129 is OK: OK: no RAID installed [13:37:06] RECOVERY - RAID on mw1104 is OK: OK: no RAID installed [13:37:06] RECOVERY - RAID on mw1157 is OK: OK: no RAID installed [13:37:16] RECOVERY - DPKG on db47 is OK: All packages OK [13:37:26] PROBLEM - DPKG on srv210 is CRITICAL: Connection refused by host [13:37:26] RECOVERY - RAID on srv259 is OK: OK: no RAID installed [13:37:36] RECOVERY - Disk space on virt3 is OK: DISK OK [13:37:36] PROBLEM - DPKG on srv234 is CRITICAL: Connection refused by host [13:37:36] RECOVERY - RAID on nfs1 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [13:37:56] PROBLEM - RAID on mw1156 is CRITICAL: Connection refused by host [13:38:06] PROBLEM - DPKG on db1029 is CRITICAL: Connection refused by host [13:38:06] PROBLEM - DPKG on mw65 is CRITICAL: Connection refused by host [13:38:16] RECOVERY - RAID on srv208 is OK: OK: no RAID installed [13:38:16] PROBLEM - DPKG on snapshot2 is CRITICAL: Connection refused by host [13:38:16] RECOVERY - RAID on mw1112 is OK: OK: no RAID installed [13:38:25] PROBLEM - RAID on snapshot2 is CRITICAL: Connection refused by host [13:38:25] PROBLEM - Disk space on db1029 is CRITICAL: Connection refused by host [13:38:25] RECOVERY - MySQL disk space on db1019 is OK: DISK OK [13:38:36] RECOVERY - Disk space on db47 is OK: DISK OK [13:38:36] PROBLEM - DPKG on bast1001 is CRITICAL: Connection refused by host [13:38:36] RECOVERY - Disk space on db1019 is OK: DISK OK [13:38:46] RECOVERY - DPKG on db1002 is OK: All packages OK [13:38:56] PROBLEM - Disk space on mw1059 is CRITICAL: Connection refused by host [13:38:56] PROBLEM - Disk space on mw1070 is CRITICAL: Connection refused by host [13:39:06] PROBLEM - RAID on es3 is CRITICAL: Connection refused by host [13:39:06] PROBLEM - DPKG on es3 is CRITICAL: Connection refused by host [13:39:06] PROBLEM - RAID on db44 is CRITICAL: Connection refused by host [13:39:06] RECOVERY - Disk space on mw1088 is OK: DISK OK [13:39:26] RECOVERY - Disk space on mw1076 is OK: DISK OK [13:39:36] RECOVERY - RAID on srv211 is OK: OK: no RAID installed [13:39:36] PROBLEM - Disk space on mw65 is CRITICAL: Connection refused by host [13:39:37] RECOVERY - RAID on mw29 is OK: OK: no RAID installed [13:39:46] PROBLEM - Disk space on mw1050 is CRITICAL: Connection refused by host [13:39:46] RECOVERY - Disk space on db1002 is OK: DISK OK [13:40:05] RECOVERY - MySQL disk space on db18 is OK: DISK OK [13:40:06] RECOVERY - RAID on srv240 is OK: OK: no RAID installed [13:40:16] PROBLEM - Disk space on mw1032 is CRITICAL: Connection refused by host [13:40:16] RECOVERY - RAID on srv235 is OK: OK: no RAID installed [13:40:26] PROBLEM - Disk space on mw1067 is CRITICAL: Connection refused by host [13:40:26] RECOVERY - DPKG on db11 is OK: All packages OK [13:40:36] RECOVERY - RAID on db16 is OK: OK: 1 logical device(s) checked [13:41:06] PROBLEM - DPKG on srv264 is CRITICAL: Connection refused by host [13:41:06] RECOVERY - DPKG on mw1141 is OK: All packages OK [13:41:16] RECOVERY - DPKG on mw1127 is OK: All packages OK [13:41:16] RECOVERY - DPKG on mw1116 is OK: All packages OK [13:41:36] RECOVERY - Disk space on srv227 is OK: DISK OK [13:41:36] PROBLEM - Disk space on srv264 is CRITICAL: Connection refused by host [13:41:36] PROBLEM - RAID on srv228 is CRITICAL: Connection refused by host [13:41:36] PROBLEM - DPKG on mw1153 is CRITICAL: Connection refused by host [13:41:46] RECOVERY - DPKG on mw1144 is OK: All packages OK [13:41:46] PROBLEM - RAID on srv271 is CRITICAL: Connection refused by host [13:41:56] RECOVERY - DPKG on snapshot4 is OK: All packages OK [13:41:56] RECOVERY - Disk space on db1022 is OK: DISK OK [13:42:06] RECOVERY - DPKG on db1022 is OK: All packages OK [13:42:06] they will all recover now.. was overload on nagios host [13:42:16] PROBLEM - DPKG on grosley is CRITICAL: Connection refused by host [13:42:16] RECOVERY - DPKG on db1033 is OK: All packages OK [13:42:16] RECOVERY - Disk space on mw1116 is OK: DISK OK [13:42:16] RECOVERY - RAID on db1033 is OK: OK: State is Optimal, checked 2 logical device(s) [13:42:26] PROBLEM - MySQL disk space on db1020 is CRITICAL: Connection refused by host [13:42:26] PROBLEM - Disk space on db1017 is CRITICAL: Connection refused by host [13:42:36] PROBLEM - Disk space on mw1153 is CRITICAL: Connection refused by host [13:42:46] RECOVERY - Disk space on mw1011 is OK: DISK OK [13:42:56] RECOVERY - Disk space on db25 is OK: DISK OK [13:43:06] RECOVERY - Disk space on srv273 is OK: DISK OK [13:43:26] RECOVERY - Disk space on db34 is OK: DISK OK [13:43:26] RECOVERY - Disk space on mw1158 is OK: DISK OK [13:43:36] PROBLEM - DPKG on db1005 is CRITICAL: Connection refused by host [13:43:36] RECOVERY - Disk space on mw1144 is OK: DISK OK [13:43:46] PROBLEM - RAID on db1005 is CRITICAL: Connection refused by host [13:43:56] PROBLEM - DPKG on srv228 is CRITICAL: Connection refused by host [13:44:06] RECOVERY - Disk space on mw1135 is OK: DISK OK [13:44:06] PROBLEM - RAID on es4 is CRITICAL: Connection refused by host [13:44:16] RECOVERY - Disk space on srv230 is OK: DISK OK [13:44:26] RECOVERY - MySQL disk space on db34 is OK: DISK OK [13:44:26] RECOVERY - RAID on db1035 is OK: OK: State is Optimal, checked 2 logical device(s) [13:44:26] RECOVERY - RAID on db25 is OK: OK: 1 logical device(s) checked [13:44:29] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1728 [13:44:30] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1728 [13:44:36] RECOVERY - DPKG on db13 is OK: All packages OK [13:44:36] RECOVERY - RAID on db13 is OK: OK: 1 logical device(s) checked [13:44:46] PROBLEM - RAID on es2 is CRITICAL: Connection refused by host [13:44:46] PROBLEM - Disk space on es4 is CRITICAL: Connection refused by host [13:44:46] PROBLEM - RAID on db1041 is CRITICAL: Connection refused by host [13:44:53] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1729 [13:44:53] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1729 [13:45:06] PROBLEM - RAID on srv207 is CRITICAL: Connection refused by host [13:45:06] RECOVERY - DPKG on searchidx2 is OK: All packages OK [13:45:16] PROBLEM - Disk space on es2 is CRITICAL: Connection refused by host [13:45:26] RECOVERY - Disk space on es1001 is OK: DISK OK [13:45:26] PROBLEM - RAID on mw7 is CRITICAL: Connection refused by host [13:45:26] RECOVERY - Disk space on db13 is OK: DISK OK [13:45:36] PROBLEM - DPKG on srv263 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - MySQL disk space on db1008 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - DPKG on db50 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - RAID on srv263 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - Disk space on snapshot1 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - RAID on srv272 is CRITICAL: Connection refused by host [13:45:46] PROBLEM - Disk space on srv260 is CRITICAL: Connection refused by host [13:45:46] RECOVERY - MySQL disk space on es1001 is OK: DISK OK [13:45:56] PROBLEM - DPKG on srv218 is CRITICAL: Connection refused by host [13:45:56] RECOVERY - RAID on srv283 is OK: OK: no RAID installed [13:46:06] PROBLEM - MySQL disk space on es2 is CRITICAL: Connection refused by host [13:46:06] PROBLEM - MySQL disk space on db1004 is CRITICAL: Connection refused by host [13:46:06] RECOVERY - DPKG on mw15 is OK: All packages OK [13:46:06] RECOVERY - RAID on mw1065 is OK: OK: no RAID installed [13:46:06] RECOVERY - RAID on db1006 is OK: OK: State is Optimal, checked 2 logical device(s) [13:46:16] PROBLEM - DPKG on db1004 is CRITICAL: Connection refused by host [13:46:26] PROBLEM - MySQL disk space on es4 is CRITICAL: Connection refused by host [13:46:26] RECOVERY - RAID on srv215 is OK: OK: no RAID installed [13:46:36] PROBLEM - DPKG on mw1026 is CRITICAL: Connection refused by host [13:46:56] PROBLEM - RAID on srv289 is CRITICAL: Connection refused by host [13:46:56] PROBLEM - RAID on db1007 is CRITICAL: Connection refused by host [13:46:56] PROBLEM - DPKG on mw1097 is CRITICAL: Connection refused by host [13:46:56] RECOVERY - DPKG on srv196 is OK: All packages OK [13:46:56] RECOVERY - RAID on srv234 is OK: OK: no RAID installed [13:46:56] RECOVERY - RAID on db48 is OK: OK: State is Optimal, checked 2 logical device(s) [13:46:56] PROBLEM - RAID on db42 is CRITICAL: Connection refused by host [13:46:57] PROBLEM - RAID on fenari is CRITICAL: Connection refused by host [13:46:57] RECOVERY - DPKG on srv215 is OK: All packages OK [13:47:16] PROBLEM - DPKG on snapshot1 is CRITICAL: Connection refused by host [13:47:16] PROBLEM - DPKG on srv289 is CRITICAL: Connection refused by host [13:47:16] RECOVERY - DPKG on srv234 is OK: All packages OK [13:47:26] PROBLEM - MySQL disk space on db42 is CRITICAL: Connection refused by host [13:47:26] PROBLEM - Disk space on srv218 is CRITICAL: Connection refused by host [13:47:26] RECOVERY - DPKG on srv210 is OK: All packages OK [13:47:36] PROBLEM - DPKG on mw1012 is CRITICAL: Connection refused by host [13:47:36] PROBLEM - Disk space on db50 is CRITICAL: Connection refused by host [13:47:46] RECOVERY - DPKG on mw65 is OK: All packages OK [13:47:56] PROBLEM - RAID on srv280 is CRITICAL: Connection refused by host [13:47:56] RECOVERY - DPKG on snapshot2 is OK: All packages OK [13:47:56] RECOVERY - RAID on mw1156 is OK: OK: no RAID installed [13:48:06] RECOVERY - Disk space on db1029 is OK: DISK OK [13:48:16] PROBLEM - DPKG on db1038 is CRITICAL: Connection refused by host [13:48:16] PROBLEM - RAID on srv247 is CRITICAL: Connection refused by host [13:48:16] RECOVERY - DPKG on bast1001 is OK: All packages OK [13:48:16] RECOVERY - RAID on snapshot2 is OK: OK: no RAID installed [13:48:17] New patchset: Dzahn; "fix duplicate definition of monitor_service for mobile traffic loggers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1730 [13:48:26] PROBLEM - DPKG on mw7 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - MySQL disk space on db50 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - DPKG on db42 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - DPKG on fenari is CRITICAL: Connection refused by host [13:48:26] PROBLEM - Disk space on mw1026 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - DPKG on srv272 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - RAID on mw30 is CRITICAL: Connection refused by host [13:48:36] PROBLEM - RAID on db1038 is CRITICAL: Connection refused by host [13:48:36] RECOVERY - DPKG on db1029 is OK: All packages OK [13:48:44] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1730 [13:48:44] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1730 [13:48:46] RECOVERY - Disk space on mw1070 is OK: DISK OK [13:48:46] RECOVERY - RAID on db44 is OK: OK: State is Optimal, checked 2 logical device(s) [13:48:56] PROBLEM - DPKG on mw30 is CRITICAL: Connection refused by host [13:49:16] RECOVERY - Disk space on mw65 is OK: DISK OK [13:49:16] RECOVERY - Disk space on mw1059 is OK: DISK OK [13:49:26] PROBLEM - DPKG on srv280 is CRITICAL: Connection refused by host [13:49:26] RECOVERY - Disk space on mw1050 is OK: DISK OK [13:49:26] PROBLEM - Disk space on mw7 is CRITICAL: Connection refused by host [13:49:26] RECOVERY - DPKG on es3 is OK: All packages OK [13:49:36] RECOVERY - RAID on es3 is OK: OK: State is Optimal, checked 2 logical device(s) [13:50:06] RECOVERY - Disk space on mw1032 is OK: DISK OK [13:50:16] PROBLEM - Disk space on srv289 is CRITICAL: Connection refused by host [13:50:16] PROBLEM - RAID on locke is CRITICAL: Connection refused by host [13:50:16] PROBLEM - RAID on snapshot1 is CRITICAL: Connection refused by host [13:50:26] RECOVERY - Disk space on mw1067 is OK: DISK OK [13:50:46] RECOVERY - DPKG on srv264 is OK: All packages OK [13:51:15] RECOVERY - Disk space on srv264 is OK: DISK OK [13:51:15] RECOVERY - RAID on srv228 is OK: OK: no RAID installed [13:51:15] RECOVERY - DPKG on mw1153 is OK: All packages OK [13:51:26] PROBLEM - Disk space on fenari is CRITICAL: Connection refused by host [13:51:36] PROBLEM - DPKG on srv247 is CRITICAL: Connection refused by host [13:51:46] RECOVERY - RAID on srv271 is OK: OK: no RAID installed [13:51:56] RECOVERY - DPKG on grosley is OK: All packages OK [13:52:06] RECOVERY - MySQL disk space on db1020 is OK: DISK OK [13:52:06] RECOVERY - Disk space on db1017 is OK: DISK OK [13:52:16] PROBLEM - Disk space on mw1012 is CRITICAL: Connection refused by host [13:52:16] RECOVERY - Disk space on mw1153 is OK: DISK OK [13:53:06] PROBLEM - Disk space on srv247 is CRITICAL: Connection refused by host [13:53:16] RECOVERY - DPKG on db1005 is OK: All packages OK [13:53:26] RECOVERY - RAID on db1005 is OK: OK: State is Optimal, checked 2 logical device(s) [13:53:36] RECOVERY - DPKG on srv228 is OK: All packages OK [13:53:46] RECOVERY - RAID on es4 is OK: OK: State is Optimal, checked 2 logical device(s) [13:54:26] RECOVERY - RAID on db1041 is OK: OK: State is Optimal, checked 2 logical device(s) [13:54:46] RECOVERY - RAID on srv207 is OK: OK: no RAID installed [13:54:46] RECOVERY - Disk space on es4 is OK: DISK OK [13:54:56] RECOVERY - Disk space on es2 is OK: DISK OK [13:55:06] RECOVERY - RAID on mw7 is OK: OK: no RAID installed [13:55:06] RECOVERY - RAID on es2 is OK: OK: State is Optimal, checked 2 logical device(s) [13:55:16] RECOVERY - DPKG on db50 is OK: All packages OK [13:55:16] RECOVERY - RAID on srv263 is OK: OK: no RAID installed [13:55:16] RECOVERY - Disk space on snapshot1 is OK: DISK OK [13:55:16] RECOVERY - RAID on srv272 is OK: OK: no RAID installed [13:55:26] RECOVERY - Disk space on srv260 is OK: DISK OK [13:55:36] RECOVERY - DPKG on srv218 is OK: All packages OK [13:55:36] RECOVERY - DPKG on srv263 is OK: All packages OK [13:55:36] RECOVERY - MySQL disk space on db1008 is OK: DISK OK [13:55:46] RECOVERY - MySQL disk space on es2 is OK: DISK OK [13:55:46] RECOVERY - MySQL disk space on db1004 is OK: DISK OK [13:55:56]