[00:01:20] New patchset: Asher; "class to install percona nagios monitors (just the files so far)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1723 [00:01:40] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1723 [00:05:20] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1723 [00:05:20] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1723 [00:48:36] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [01:03:06] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [01:04:02] New patchset: Lcarr; "Removing star from planet as it is already defined" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1724 [01:04:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1724 [01:05:41] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1724 [01:05:41] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1724 [01:06:46] RECOVERY - Puppet freshness on singer is OK: puppet ran at Wed Dec 28 01:06:40 UTC 2011 [02:04:37] !log LocalisationUpdate completed (1.18) at Wed Dec 28 02:04:37 UTC 2011 [02:04:49] Logged the message, Master [02:13:38] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1667s [02:21:48] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 2157s [02:31:48] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:32:58] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [03:42:43] PROBLEM - mobile traffic loggers on cp1041 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [03:42:43] PROBLEM - mobile traffic loggers on cp1043 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [03:42:43] PROBLEM - mobile traffic loggers on cp1042 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [03:52:33] RECOVERY - mobile traffic loggers on cp1042 is OK: PROCS OK: 4 processes with args varnishncsa [04:02:13] RECOVERY - mobile traffic loggers on cp1043 is OK: PROCS OK: 3 processes with args varnishncsa [04:11:35] RECOVERY - mobile traffic loggers on cp1041 is OK: PROCS OK: 3 processes with args varnishncsa [04:36:55] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [06:59:19] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 272 MB (3% inode=60%): /var/lib/ureadahead/debugfs 272 MB (3% inode=60%): [07:06:22] !log maybe the cron job on these image scalers should run more often... cleaned up /tmp on srv221 [07:06:32] Logged the message, Master [07:09:19] RECOVERY - Disk space on srv221 is OK: DISK OK [07:29:18] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1687 [07:34:18] RECOVERY - Disk space on hume is OK: DISK OK [07:35:38] New patchset: Dzahn; "additional generic check_procs with -C option & fix "mobile traffic logger" checks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1687 [07:35:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1687 [07:36:26] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1687 [07:36:27] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1687 [08:49:39] New patchset: Dzahn; "nagios: purge resources using puppet (instead of .py script), comments on duplicate definitions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1726 [08:49:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1726 [09:06:42] New patchset: Dzahn; "nagios: purge resources using puppet (instead of .py script), try to avoid duplicate definitions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1726 [09:17:34] hi, found a deadlink on https://github.com/wikimedia/WikipediaMobile pointing to https://github.com/nitobi/Wikipedia (dead) [09:54:03] RECOVERY - MySQL slave status on es1004 is OK: OK: [10:00:22] PROBLEM - mobile traffic loggers on cp1043 is CRITICAL: PROCS CRITICAL: 6 processes with args varnishncsa [10:00:22] PROBLEM - mobile traffic loggers on cp1041 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [10:00:22] PROBLEM - mobile traffic loggers on cp1042 is CRITICAL: PROCS CRITICAL: 7 processes with args varnishncsa [10:28:54] RECOVERY - mobile traffic loggers on cp1043 is OK: PROCS OK: 2 processes with args varnishncsa [10:55:52] PROBLEM - mobile traffic loggers on cp1043 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [10:55:52] PROBLEM - mobile traffic loggers on cp1044 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [10:56:30] hundfred: as far as i can tell /nitobi/Wikipedia has been merged into /wikimedia/WikipediaMobile [10:57:55] "Merge branch 'master' of git://github.com/nitobi/Wikipedia " October 28, 2011 [11:12:42] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [12:04:15] PROBLEM - Puppet freshness on srv191 is CRITICAL: Puppet has not run in the last 10 hours [12:47:27] PROBLEM - Auth DNS on ns0.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:47:35] New patchset: Dzahn; "nagios - check procs via nrpe -checkcommands - can't use (broken) generic check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1727 [12:47:50] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1727 [12:48:47] New patchset: Dzahn; "nagios - check procs via nrpe -checkcommands - can't use (broken) generic check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1727 [12:49:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1727 [12:50:09] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [12:50:27] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1727 [12:50:28] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1727 [12:57:33] New patchset: Dzahn; "nagios proc checks - use nrpe commands, this just seemed to work..duh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1728 [12:57:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1728 [13:02:44] New patchset: Dzahn; "add nrpe to tarin and cp104[1-4] for process checks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1729 [13:02:56] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1729 [13:08:35] RECOVERY - Auth DNS on ns0.wikimedia.org is OK: DNS OK: 6.477 seconds response time. www.wikipedia.org returns 208.80.152.201 [13:15:10] PROBLEM - RAID on db1009 is CRITICAL: Connection refused by host [13:16:51] PROBLEM - Disk space on mw1113 is CRITICAL: Connection refused by host [13:17:04] PROBLEM - Disk space on srv225 is CRITICAL: Connection refused by host [13:17:10] PROBLEM - Disk space on mw1157 is CRITICAL: Connection refused by host [13:17:11] PROBLEM - DPKG on srv259 is CRITICAL: Connection refused by host [13:17:31] PROBLEM - Disk space on srv259 is CRITICAL: Connection refused by host [13:18:42] PROBLEM - RAID on es1003 is CRITICAL: Connection refused by host [13:19:51] PROBLEM - RAID on mw1015 is CRITICAL: Connection refused by host [13:20:31] PROBLEM - DPKG on ms1002 is CRITICAL: Connection refused by host [13:21:44] PROBLEM - MySQL disk space on es1003 is CRITICAL: Connection refused by host [13:21:54] PROBLEM - RAID on srv229 is CRITICAL: Connection refused by host [13:22:23] PROBLEM - Disk space on es1003 is CRITICAL: Connection refused by host [13:22:23] PROBLEM - DPKG on srv274 is CRITICAL: Connection refused by host [13:22:32] PROBLEM - Disk space on aluminium is CRITICAL: Connection refused by host [13:22:32] PROBLEM - DPKG on db1019 is CRITICAL: Connection refused by host [13:22:52] PROBLEM - DPKG on nfs1 is CRITICAL: Connection refused by host [13:22:52] PROBLEM - DPKG on mw1015 is CRITICAL: Connection refused by host [13:23:02] PROBLEM - DPKG on mw1088 is CRITICAL: Connection refused by host [13:23:22] PROBLEM - DPKG on db47 is CRITICAL: Connection refused by host [13:23:22] PROBLEM - RAID on mw1129 is CRITICAL: Connection refused by host [13:23:32] PROBLEM - RAID on mw1104 is CRITICAL: Connection refused by host [13:23:32] PROBLEM - RAID on mw1157 is CRITICAL: Connection refused by host [13:23:42] PROBLEM - Disk space on virt3 is CRITICAL: Connection refused by host [13:23:54] PROBLEM - RAID on srv259 is CRITICAL: Connection refused by host [13:24:02] PROBLEM - RAID on nfs1 is CRITICAL: Connection refused by host [13:24:24] PROBLEM - RAID on srv208 is CRITICAL: Connection refused by host [13:24:34] PROBLEM - MySQL disk space on db1019 is CRITICAL: Connection refused by host [13:24:42] PROBLEM - RAID on mw1112 is CRITICAL: Connection refused by host [13:24:42] PROBLEM - Disk space on db47 is CRITICAL: Connection refused by host [13:24:42] PROBLEM - Disk space on db1019 is CRITICAL: Connection refused by host [13:25:02] PROBLEM - DPKG on db1002 is CRITICAL: Connection refused by host [13:25:50] PROBLEM - Disk space on mw1088 is CRITICAL: Connection refused by host [13:28:09] PROBLEM - RAID on mw29 is CRITICAL: Connection refused by host [13:28:10] PROBLEM - Disk space on mw1076 is CRITICAL: Connection refused by host [13:28:32] PROBLEM - RAID on srv211 is CRITICAL: Connection refused by host [13:28:41] PROBLEM - Disk space on db1002 is CRITICAL: Connection refused by host [13:28:59] PROBLEM - MySQL disk space on db18 is CRITICAL: Connection refused by host [13:28:59] PROBLEM - RAID on srv240 is CRITICAL: Connection refused by host [13:29:08] PROBLEM - DPKG on db11 is CRITICAL: Connection refused by host [13:29:09] PROBLEM - RAID on srv235 is CRITICAL: Connection refused by host [13:29:18] PROBLEM - RAID on db16 is CRITICAL: Connection refused by host [13:29:49] PROBLEM - DPKG on mw1141 is CRITICAL: Connection refused by host [13:30:17] PROBLEM - DPKG on mw1116 is CRITICAL: Connection refused by host [13:30:18] PROBLEM - DPKG on mw1127 is CRITICAL: Connection refused by host [13:30:48] PROBLEM - Disk space on srv227 is CRITICAL: Connection refused by host [13:30:58] PROBLEM - DPKG on mw1144 is CRITICAL: Connection refused by host [13:31:08] PROBLEM - Disk space on db1022 is CRITICAL: Connection refused by host [13:31:09] PROBLEM - DPKG on snapshot4 is CRITICAL: Connection refused by host [13:31:19] PROBLEM - RAID on db1033 is CRITICAL: Connection refused by host [13:31:20] PROBLEM - DPKG on db1033 is CRITICAL: Connection refused by host [13:31:20] PROBLEM - DPKG on db1022 is CRITICAL: Connection refused by host [13:31:20] RECOVERY - RAID on db1009 is OK: OK: State is Optimal, checked 2 logical device(s) [13:31:29] PROBLEM - Disk space on mw1116 is CRITICAL: Connection refused by host [13:31:58] PROBLEM - Disk space on db25 is CRITICAL: Connection refused by host [13:31:59] PROBLEM - Disk space on mw1011 is CRITICAL: Connection refused by host [13:32:19] PROBLEM - Disk space on db34 is CRITICAL: Connection refused by host [13:32:22] PROBLEM - Disk space on srv273 is CRITICAL: Connection refused by host [13:32:38] PROBLEM - Disk space on mw1158 is CRITICAL: Connection refused by host [13:32:38] PROBLEM - Disk space on mw1144 is CRITICAL: Connection refused by host [13:32:38] RECOVERY - Disk space on mw1113 is OK: DISK OK [13:32:58] RECOVERY - Disk space on srv225 is OK: DISK OK [13:32:59] RECOVERY - Disk space on mw1157 is OK: DISK OK [13:33:00] RECOVERY - DPKG on srv259 is OK: All packages OK [13:33:11] PROBLEM - MySQL disk space on db34 is CRITICAL: Connection refused by host [13:33:11] PROBLEM - Disk space on srv230 is CRITICAL: Connection refused by host [13:33:19] RECOVERY - Disk space on srv259 is OK: DISK OK [13:33:20] PROBLEM - Disk space on mw1135 is CRITICAL: Connection refused by host [13:33:32] PROBLEM - DPKG on db13 is CRITICAL: Connection refused by host [13:33:41] PROBLEM - RAID on db25 is CRITICAL: Connection refused by host [13:33:42] PROBLEM - RAID on db1035 is CRITICAL: Connection refused by host [13:34:56] PROBLEM - RAID on db13 is CRITICAL: Connection refused by host [13:34:57] RECOVERY - RAID on es1003 is OK: OK: State is Optimal, checked 2 logical device(s) [13:35:15] RECOVERY - RAID on mw1015 is OK: OK: no RAID installed [13:35:25] RECOVERY - DPKG on ms1002 is OK: All packages OK [13:35:26] PROBLEM - DPKG on searchidx2 is CRITICAL: Connection refused by host [13:35:26] RECOVERY - MySQL disk space on es1003 is OK: DISK OK [13:35:35] RECOVERY - RAID on srv229 is OK: OK: no RAID installed [13:35:45] PROBLEM - Disk space on db13 is CRITICAL: Connection refused by host [13:35:45] PROBLEM - Disk space on es1001 is CRITICAL: Connection refused by host [13:36:06] RECOVERY - DPKG on srv274 is OK: All packages OK [13:36:06] RECOVERY - Disk space on es1003 is OK: DISK OK [13:36:06] PROBLEM - MySQL disk space on es1001 is CRITICAL: Connection refused by host [13:36:06] RECOVERY - Disk space on aluminium is OK: DISK OK [13:36:15] PROBLEM - RAID on mw1065 is CRITICAL: Connection refused by host [13:36:16] PROBLEM - RAID on srv283 is CRITICAL: Connection refused by host [13:36:16] PROBLEM - RAID on db1006 is CRITICAL: Connection refused by host [13:36:16] RECOVERY - DPKG on db1019 is OK: All packages OK [13:36:26] PROBLEM - DPKG on mw15 is CRITICAL: Connection refused by host [13:36:27] RECOVERY - DPKG on nfs1 is OK: All packages OK [13:36:35] PROBLEM - RAID on srv215 is CRITICAL: Connection refused by host [13:36:45] RECOVERY - DPKG on mw1015 is OK: All packages OK [13:36:46] RECOVERY - DPKG on mw1088 is OK: All packages OK [13:36:55] PROBLEM - DPKG on srv215 is CRITICAL: Connection refused by host [13:37:05] PROBLEM - RAID on db48 is CRITICAL: Connection refused by host [13:37:06] PROBLEM - DPKG on srv196 is CRITICAL: Connection refused by host [13:37:06] PROBLEM - RAID on srv234 is CRITICAL: Connection refused by host [13:37:06] RECOVERY - RAID on mw1129 is OK: OK: no RAID installed [13:37:06] RECOVERY - RAID on mw1104 is OK: OK: no RAID installed [13:37:06] RECOVERY - RAID on mw1157 is OK: OK: no RAID installed [13:37:16] RECOVERY - DPKG on db47 is OK: All packages OK [13:37:25] PROBLEM - DPKG on srv210 is CRITICAL: Connection refused by host [13:37:26] RECOVERY - RAID on srv259 is OK: OK: no RAID installed [13:37:35] RECOVERY - Disk space on virt3 is OK: DISK OK [13:37:36] PROBLEM - DPKG on srv234 is CRITICAL: Connection refused by host [13:37:36] RECOVERY - RAID on nfs1 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [13:37:55] PROBLEM - RAID on mw1156 is CRITICAL: Connection refused by host [13:38:05] PROBLEM - DPKG on db1029 is CRITICAL: Connection refused by host [13:38:06] PROBLEM - DPKG on mw65 is CRITICAL: Connection refused by host [13:38:15] RECOVERY - RAID on srv208 is OK: OK: no RAID installed [13:38:16] PROBLEM - DPKG on snapshot2 is CRITICAL: Connection refused by host [13:38:16] RECOVERY - RAID on mw1112 is OK: OK: no RAID installed [13:38:25] PROBLEM - RAID on snapshot2 is CRITICAL: Connection refused by host [13:38:25] PROBLEM - Disk space on db1029 is CRITICAL: Connection refused by host [13:38:25] RECOVERY - MySQL disk space on db1019 is OK: DISK OK [13:38:35] RECOVERY - Disk space on db47 is OK: DISK OK [13:38:36] PROBLEM - DPKG on bast1001 is CRITICAL: Connection refused by host [13:38:36] RECOVERY - Disk space on db1019 is OK: DISK OK [13:38:45] RECOVERY - DPKG on db1002 is OK: All packages OK [13:38:55] PROBLEM - Disk space on mw1059 is CRITICAL: Connection refused by host [13:38:56] PROBLEM - Disk space on mw1070 is CRITICAL: Connection refused by host [13:39:05] PROBLEM - RAID on es3 is CRITICAL: Connection refused by host [13:39:06] PROBLEM - DPKG on es3 is CRITICAL: Connection refused by host [13:39:06] PROBLEM - RAID on db44 is CRITICAL: Connection refused by host [13:39:06] RECOVERY - Disk space on mw1088 is OK: DISK OK [13:39:26] RECOVERY - Disk space on mw1076 is OK: DISK OK [13:39:35] RECOVERY - RAID on srv211 is OK: OK: no RAID installed [13:39:36] PROBLEM - Disk space on mw65 is CRITICAL: Connection refused by host [13:39:36] RECOVERY - RAID on mw29 is OK: OK: no RAID installed [13:39:46] PROBLEM - Disk space on mw1050 is CRITICAL: Connection refused by host [13:39:46] RECOVERY - Disk space on db1002 is OK: DISK OK [13:40:05] RECOVERY - MySQL disk space on db18 is OK: DISK OK [13:40:06] RECOVERY - RAID on srv240 is OK: OK: no RAID installed [13:40:15] PROBLEM - Disk space on mw1032 is CRITICAL: Connection refused by host [13:40:16] RECOVERY - RAID on srv235 is OK: OK: no RAID installed [13:40:25] PROBLEM - Disk space on mw1067 is CRITICAL: Connection refused by host [13:40:26] RECOVERY - DPKG on db11 is OK: All packages OK [13:40:35] RECOVERY - RAID on db16 is OK: OK: 1 logical device(s) checked [13:41:05] PROBLEM - DPKG on srv264 is CRITICAL: Connection refused by host [13:41:06] RECOVERY - DPKG on mw1141 is OK: All packages OK [13:41:15] RECOVERY - DPKG on mw1127 is OK: All packages OK [13:41:16] RECOVERY - DPKG on mw1116 is OK: All packages OK [13:41:35] RECOVERY - Disk space on srv227 is OK: DISK OK [13:41:36] PROBLEM - Disk space on srv264 is CRITICAL: Connection refused by host [13:41:36] PROBLEM - RAID on srv228 is CRITICAL: Connection refused by host [13:41:36] PROBLEM - DPKG on mw1153 is CRITICAL: Connection refused by host [13:41:45] RECOVERY - DPKG on mw1144 is OK: All packages OK [13:41:46] PROBLEM - RAID on srv271 is CRITICAL: Connection refused by host [13:41:55] RECOVERY - DPKG on snapshot4 is OK: All packages OK [13:41:56] RECOVERY - Disk space on db1022 is OK: DISK OK [13:42:05] RECOVERY - DPKG on db1022 is OK: All packages OK [13:42:15] PROBLEM - DPKG on grosley is CRITICAL: Connection refused by host [13:42:16] RECOVERY - DPKG on db1033 is OK: All packages OK [13:42:16] RECOVERY - Disk space on mw1116 is OK: DISK OK [13:42:16] RECOVERY - RAID on db1033 is OK: OK: State is Optimal, checked 2 logical device(s) [13:42:25] PROBLEM - MySQL disk space on db1020 is CRITICAL: Connection refused by host [13:42:26] PROBLEM - Disk space on db1017 is CRITICAL: Connection refused by host [13:42:35] PROBLEM - Disk space on mw1153 is CRITICAL: Connection refused by host [13:42:45] RECOVERY - Disk space on mw1011 is OK: DISK OK [13:42:55] RECOVERY - Disk space on db25 is OK: DISK OK [13:43:06] RECOVERY - Disk space on srv273 is OK: DISK OK [13:43:25] RECOVERY - Disk space on db34 is OK: DISK OK [13:43:26] RECOVERY - Disk space on mw1158 is OK: DISK OK [13:43:35] PROBLEM - DPKG on db1005 is CRITICAL: Connection refused by host [13:43:36] RECOVERY - Disk space on mw1144 is OK: DISK OK [13:43:45] PROBLEM - RAID on db1005 is CRITICAL: Connection refused by host [13:43:55] PROBLEM - DPKG on srv228 is CRITICAL: Connection refused by host [13:44:05] RECOVERY - Disk space on mw1135 is OK: DISK OK [13:44:06] PROBLEM - RAID on es4 is CRITICAL: Connection refused by host [13:44:15] RECOVERY - Disk space on srv230 is OK: DISK OK [13:44:25] RECOVERY - MySQL disk space on db34 is OK: DISK OK [13:44:26] RECOVERY - RAID on db1035 is OK: OK: State is Optimal, checked 2 logical device(s) [13:44:26] RECOVERY - RAID on db25 is OK: OK: 1 logical device(s) checked [13:44:29] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1728 [13:44:30] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1728 [13:44:35] RECOVERY - DPKG on db13 is OK: All packages OK [13:44:36] RECOVERY - RAID on db13 is OK: OK: 1 logical device(s) checked [13:44:46] PROBLEM - RAID on es2 is CRITICAL: Connection refused by host [13:44:46] PROBLEM - Disk space on es4 is CRITICAL: Connection refused by host [13:44:46] PROBLEM - RAID on db1041 is CRITICAL: Connection refused by host [13:44:53] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1729 [13:44:53] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1729 [13:45:05] PROBLEM - RAID on srv207 is CRITICAL: Connection refused by host [13:45:06] RECOVERY - DPKG on searchidx2 is OK: All packages OK [13:45:15] PROBLEM - Disk space on es2 is CRITICAL: Connection refused by host [13:45:26] RECOVERY - Disk space on es1001 is OK: DISK OK [13:45:26] PROBLEM - RAID on mw7 is CRITICAL: Connection refused by host [13:45:26] RECOVERY - Disk space on db13 is OK: DISK OK [13:45:36] PROBLEM - DPKG on srv263 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - MySQL disk space on db1008 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - DPKG on db50 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - RAID on srv263 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - Disk space on snapshot1 is CRITICAL: Connection refused by host [13:45:36] PROBLEM - RAID on srv272 is CRITICAL: Connection refused by host [13:45:45] PROBLEM - Disk space on srv260 is CRITICAL: Connection refused by host [13:45:46] RECOVERY - MySQL disk space on es1001 is OK: DISK OK [13:45:56] PROBLEM - DPKG on srv218 is CRITICAL: Connection refused by host [13:45:56] RECOVERY - RAID on srv283 is OK: OK: no RAID installed [13:46:05] PROBLEM - MySQL disk space on es2 is CRITICAL: Connection refused by host [13:46:06] PROBLEM - MySQL disk space on db1004 is CRITICAL: Connection refused by host [13:46:06] RECOVERY - DPKG on mw15 is OK: All packages OK [13:46:06] RECOVERY - RAID on mw1065 is OK: OK: no RAID installed [13:46:06] RECOVERY - RAID on db1006 is OK: OK: State is Optimal, checked 2 logical device(s) [13:46:15] PROBLEM - DPKG on db1004 is CRITICAL: Connection refused by host [13:46:26] PROBLEM - MySQL disk space on es4 is CRITICAL: Connection refused by host [13:46:26] RECOVERY - RAID on srv215 is OK: OK: no RAID installed [13:46:35] PROBLEM - DPKG on mw1026 is CRITICAL: Connection refused by host [13:46:55] PROBLEM - RAID on srv289 is CRITICAL: Connection refused by host [13:46:56] PROBLEM - RAID on db1007 is CRITICAL: Connection refused by host [13:46:56] PROBLEM - DPKG on mw1097 is CRITICAL: Connection refused by host [13:46:56] RECOVERY - DPKG on srv196 is OK: All packages OK [13:46:56] RECOVERY - RAID on srv234 is OK: OK: no RAID installed [13:46:56] RECOVERY - RAID on db48 is OK: OK: State is Optimal, checked 2 logical device(s) [13:46:56] PROBLEM - RAID on db42 is CRITICAL: Connection refused by host [13:46:57] PROBLEM - RAID on fenari is CRITICAL: Connection refused by host [13:46:57] RECOVERY - DPKG on srv215 is OK: All packages OK [13:47:15] PROBLEM - DPKG on snapshot1 is CRITICAL: Connection refused by host [13:47:16] PROBLEM - DPKG on srv289 is CRITICAL: Connection refused by host [13:47:16] RECOVERY - DPKG on srv234 is OK: All packages OK [13:47:25] PROBLEM - MySQL disk space on db42 is CRITICAL: Connection refused by host [13:47:26] PROBLEM - Disk space on srv218 is CRITICAL: Connection refused by host [13:47:26] RECOVERY - DPKG on srv210 is OK: All packages OK [13:47:35] PROBLEM - DPKG on mw1012 is CRITICAL: Connection refused by host [13:47:36] PROBLEM - Disk space on db50 is CRITICAL: Connection refused by host [13:47:45] RECOVERY - DPKG on mw65 is OK: All packages OK [13:47:56] PROBLEM - RAID on srv280 is CRITICAL: Connection refused by host [13:47:56] RECOVERY - DPKG on snapshot2 is OK: All packages OK [13:47:56] RECOVERY - RAID on mw1156 is OK: OK: no RAID installed [13:48:05] RECOVERY - Disk space on db1029 is OK: DISK OK [13:48:16] PROBLEM - DPKG on db1038 is CRITICAL: Connection refused by host [13:48:16] PROBLEM - RAID on srv247 is CRITICAL: Connection refused by host [13:48:16] RECOVERY - DPKG on bast1001 is OK: All packages OK [13:48:16] RECOVERY - RAID on snapshot2 is OK: OK: no RAID installed [13:48:18] New patchset: Dzahn; "fix duplicate definition of monitor_service for mobile traffic loggers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1730 [13:48:25] PROBLEM - DPKG on mw7 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - MySQL disk space on db50 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - DPKG on db42 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - DPKG on fenari is CRITICAL: Connection refused by host [13:48:26] PROBLEM - Disk space on mw1026 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - DPKG on srv272 is CRITICAL: Connection refused by host [13:48:26] PROBLEM - RAID on mw30 is CRITICAL: Connection refused by host [13:48:35] PROBLEM - RAID on db1038 is CRITICAL: Connection refused by host [13:48:36] RECOVERY - DPKG on db1029 is OK: All packages OK [13:48:44] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1730 [13:48:44] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1730 [13:48:46] RECOVERY - Disk space on mw1070 is OK: DISK OK [13:48:46] RECOVERY - RAID on db44 is OK: OK: State is Optimal, checked 2 logical device(s) [13:48:55] PROBLEM - DPKG on mw30 is CRITICAL: Connection refused by host [13:49:15] RECOVERY - Disk space on mw65 is OK: DISK OK [13:49:16] RECOVERY - Disk space on mw1059 is OK: DISK OK [13:49:26] PROBLEM - DPKG on srv280 is CRITICAL: Connection refused by host [13:49:26] RECOVERY - Disk space on mw1050 is OK: DISK OK [13:49:26] PROBLEM - Disk space on mw7 is CRITICAL: Connection refused by host [13:49:26] RECOVERY - DPKG on es3 is OK: All packages OK [13:49:36] RECOVERY - RAID on es3 is OK: OK: State is Optimal, checked 2 logical device(s) [13:50:06] RECOVERY - Disk space on mw1032 is OK: DISK OK [13:50:15] PROBLEM - Disk space on srv289 is CRITICAL: Connection refused by host [13:50:16] PROBLEM - RAID on locke is CRITICAL: Connection refused by host [13:50:16] PROBLEM - RAID on snapshot1 is CRITICAL: Connection refused by host [13:50:25] RECOVERY - Disk space on mw1067 is OK: DISK OK [13:50:45] RECOVERY - DPKG on srv264 is OK: All packages OK [13:51:15] RECOVERY - Disk space on srv264 is OK: DISK OK [13:51:15] RECOVERY - RAID on srv228 is OK: OK: no RAID installed [13:51:15] RECOVERY - DPKG on mw1153 is OK: All packages OK [13:51:25] PROBLEM - Disk space on fenari is CRITICAL: Connection refused by host [13:51:35] PROBLEM - DPKG on srv247 is CRITICAL: Connection refused by host [13:51:46] RECOVERY - RAID on srv271 is OK: OK: no RAID installed [13:51:55] RECOVERY - DPKG on grosley is OK: All packages OK [13:52:05] RECOVERY - MySQL disk space on db1020 is OK: DISK OK [13:52:06] RECOVERY - Disk space on db1017 is OK: DISK OK [13:52:16] PROBLEM - Disk space on mw1012 is CRITICAL: Connection refused by host [13:52:16] RECOVERY - Disk space on mw1153 is OK: DISK OK [13:53:05] PROBLEM - Disk space on srv247 is CRITICAL: Connection refused by host [13:53:15] RECOVERY - DPKG on db1005 is OK: All packages OK [13:53:25] RECOVERY - RAID on db1005 is OK: OK: State is Optimal, checked 2 logical device(s) [13:53:35] RECOVERY - DPKG on srv228 is OK: All packages OK [13:53:45] RECOVERY - RAID on es4 is OK: OK: State is Optimal, checked 2 logical device(s) [13:54:25] RECOVERY - RAID on db1041 is OK: OK: State is Optimal, checked 2 logical device(s) [13:54:45] RECOVERY - RAID on srv207 is OK: OK: no RAID installed [13:54:46] RECOVERY - Disk space on es4 is OK: DISK OK [13:54:55] RECOVERY - Disk space on es2 is OK: DISK OK [13:55:06] RECOVERY - RAID on mw7 is OK: OK: no RAID installed [13:55:06] RECOVERY - RAID on es2 is OK: OK: State is Optimal, checked 2 logical device(s) [13:55:16] RECOVERY - DPKG on db50 is OK: All packages OK [13:55:16] RECOVERY - RAID on srv263 is OK: OK: no RAID installed [13:55:16] RECOVERY - Disk space on snapshot1 is OK: DISK OK [13:55:16] RECOVERY - RAID on srv272 is OK: OK: no RAID installed [13:55:25] RECOVERY - Disk space on srv260 is OK: DISK OK [13:55:35] RECOVERY - DPKG on srv218 is OK: All packages OK [13:55:36] RECOVERY - DPKG on srv263 is OK: All packages OK [13:55:36] RECOVERY - MySQL disk space on db1008 is OK: DISK OK [13:55:45] RECOVERY - MySQL disk space on es2 is OK: DISK OK [13:55:46] RECOVERY - MySQL disk space on db1004 is OK: DISK OK [13:55:56] RECOVERY - DPKG on db1004 is OK: All packages OK [13:56:15] RECOVERY - DPKG on mw1026 is OK: All packages OK [13:56:36] RECOVERY - RAID on db42 is OK: OK: State is Optimal, checked 2 logical device(s) [13:56:36] RECOVERY - DPKG on mw1097 is OK: All packages OK [13:56:36] RECOVERY - RAID on fenari is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [13:56:36] RECOVERY - RAID on db1007 is OK: OK: State is Optimal, checked 2 logical device(s) [13:56:56] RECOVERY - DPKG on srv289 is OK: All packages OK [13:57:06] RECOVERY - Disk space on srv218 is OK: DISK OK [13:57:06] RECOVERY - DPKG on snapshot1 is OK: All packages OK [13:57:06] RECOVERY - RAID on srv289 is OK: OK: no RAID installed [13:57:15] RECOVERY - Disk space on db50 is OK: DISK OK [13:57:16] RECOVERY - MySQL disk space on db42 is OK: DISK OK [13:57:16] RECOVERY - DPKG on mw1012 is OK: All packages OK [13:57:16] RECOVERY - MySQL disk space on es4 is OK: DISK OK [13:57:45] RECOVERY - RAID on srv280 is OK: OK: no RAID installed [13:57:56] RECOVERY - RAID on srv247 is OK: OK: no RAID installed [13:57:56] RECOVERY - DPKG on db1038 is OK: All packages OK [13:58:06] RECOVERY - DPKG on fenari is OK: All packages OK [13:58:06] RECOVERY - RAID on mw30 is OK: OK: no RAID installed [13:58:06] RECOVERY - DPKG on srv272 is OK: All packages OK [13:58:06] RECOVERY - Disk space on mw1026 is OK: DISK OK [13:58:15] RECOVERY - DPKG on db42 is OK: All packages OK [13:58:25] RECOVERY - MySQL disk space on db50 is OK: DISK OK [13:58:26] RECOVERY - RAID on db1038 is OK: OK: State is Optimal, checked 2 logical device(s) [13:58:36] RECOVERY - DPKG on mw7 is OK: All packages OK [13:58:36] RECOVERY - DPKG on mw30 is OK: All packages OK [13:59:05] RECOVERY - Disk space on mw7 is OK: DISK OK [13:59:15] RECOVERY - DPKG on srv280 is OK: All packages OK [13:59:55] RECOVERY - RAID on snapshot1 is OK: OK: no RAID installed [13:59:56] RECOVERY - RAID on locke is OK: OK: State is Optimal, checked 8 logical device(s) [14:00:05] RECOVERY - Disk space on srv289 is OK: DISK OK [14:01:06] RECOVERY - Disk space on fenari is OK: DISK OK [14:01:36] RECOVERY - DPKG on srv247 is OK: All packages OK [14:02:46] RECOVERY - Disk space on mw1012 is OK: DISK OK [14:03:55] RECOVERY - Disk space on srv247 is OK: DISK OK [14:33:39] RECOVERY - mobile traffic loggers on cp1041 is OK: PROCS OK: 2 processes with command name varnishncsa [15:04:22] Reedy, regarding bug 33319, will the user pages need touching/null-editing? [15:29:28] New patchset: Demon; "Use https so we don't complain about mixed content" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1731 [15:29:41] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1731 [15:30:30] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1731 [15:30:31] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1731 [15:54:10] New patchset: Dzahn; "integration.mw - add SSLCACertificateFile to fix cert verification" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1732 [15:54:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1732 [15:55:59] New review: Dzahn; "this makes openssl say: Verify return code: 0 (ok)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1732 [15:55:59] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1732 [16:11:49] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 187 MB (2% inode=60%): /var/lib/ureadahead/debugfs 187 MB (2% inode=60%): [16:16:28]