[19:52:29] blah [19:53:21] @logoff [20:53:25] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1542 [20:53:25] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1542 [21:4:11] New patchset: Pyoungmeister; "removing "temp" nagios config" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1543 [21:4:42] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1543 [21:4:43] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1543 [21:22:29] PROBLEM - Puppet freshness on mw1055 is CRITICAL: Puppet has not run in the last 10 hours [21:22:29] PROBLEM - Puppet freshness on mw1062 is CRITICAL: Puppet has not run in the last 10 hours [21:22:29] PROBLEM - Puppet freshness on mw1063 is CRITICAL: Puppet has not run in the last 10 hours [21:22:29] PROBLEM - Puppet freshness on mw1083 is CRITICAL: Puppet has not run in the last 10 hours [21:22:29] PROBLEM - Puppet freshness on mw1122 is CRITICAL: Puppet has not run in the last 10 hours [21:22:30] PROBLEM - Puppet freshness on mw1121 is CRITICAL: Puppet has not run in the last 10 hours [21:22:30] PROBLEM - Puppet freshness on mw1156 is CRITICAL: Puppet has not run in the last 10 hours [21:22:31] PROBLEM - Puppet freshness on mw1140 is CRITICAL: Puppet has not run in the last 10 hours [21:22:31] PROBLEM - Puppet freshness on mw1148 is CRITICAL: Puppet has not run in the last 10 hours [21:22:32] PROBLEM - Puppet freshness on mw1159 is CRITICAL: Puppet has not run in the last 10 hours [21:22:32] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: Puppet has not run in the last 10 hours [21:22:33] PROBLEM - Puppet freshness on virt1 is CRITICAL: Puppet has not run in the last 10 hours [21:22:33] PROBLEM - Puppet freshness on virt2 is CRITICAL: Puppet has not run in the last 10 hours [21:25:15] RECOVERY - SSH on mw29 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:25:15] RECOVERY - Disk space on mw26 is OK: DISK OK [21:25:15] RECOVERY - Disk space on mw30 is OK: DISK OK [21:25:15] RECOVERY - Apache HTTP on mw27 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [21:25:15] RECOVERY - RAID on mw27 is OK: OK: no RAID installed [21:25:16] RECOVERY - DPKG on mw29 is OK: All packages OK [21:25:25] RECOVERY - Disk space on mw59 is OK: DISK OK [21:25:25] RECOVERY - Apache HTTP on mw6 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [21:25:25] RECOVERY - RAID on mw6 is OK: OK: no RAID installed [21:25:35] RECOVERY - ps1-a3-sdtpa-infeed-load-tower-A-phase-Z on ps1-a3-sdtpa is OK: ps1-a3-sdtpa-infeed-load-tower-A-phase-Z OK - 1713 [21:25:35] RECOVERY - ps1-a2-sdtpa-infeed-load-tower-B-phase-Y on ps1-a2-sdtpa is OK: ps1-a2-sdtpa-infeed-load-tower-B-phase-Y OK - 1063 [21:25:35] RECOVERY - ps1-a3-eqiad-infeed-load-tower-B-phase-X on ps1-a3-eqiad is OK: ps1-a3-eqiad-infeed-load-tower-B-phase-X OK - 775 [21:25:35] RECOVERY - ps1-a2-eqiad-infeed-load-tower-A-phase-X on ps1-a2-eqiad is OK: ps1-a2-eqiad-infeed-load-tower-A-phase-X OK - 613 [21:25:35] RECOVERY - ps1-a5-eqiad-infeed-load-tower-A-phase-X on ps1-a5-eqiad is OK: ps1-a5-eqiad-infeed-load-tower-A-phase-X OK - 900 [21:25:36] RECOVERY - ps1-a2-eqiad-infeed-load-tower-B-phase-Z on ps1-a2-eqiad is OK: ps1-a2-eqiad-infeed-load-tower-B-phase-Z OK - 600 [21:25:36] RECOVERY - ps1-a5-eqiad-infeed-load-tower-B-phase-Z on ps1-a5-eqiad is OK: ps1-a5-eqiad-infeed-load-tower-B-phase-Z OK - 1088 [21:26:5] RECOVERY - SSH on srv190 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:26:5] RECOVERY - DPKG on srv190 is OK: All packages OK [21:26:25] RECOVERY - SSH on srv243 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:26:25] RECOVERY - Disk space on srv245 is OK: DISK OK [21:26:25] RECOVERY - Apache HTTP on srv246 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.022 second response time [21:26:25] RECOVERY - Apache HTTP on srv241 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.028 second response time [21:26:25] RECOVERY - DPKG on srv243 is OK: All packages OK [21:26:26] RECOVERY - RAID on srv241 is OK: OK: no RAID installed [21:26:35] RECOVERY - RAID on mw21 is OK: OK: no RAID installed [21:26:55] RECOVERY - SSH on mw24 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:26:55] RECOVERY - Apache HTTP on mw36 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.022 second response time [21:27:15] RECOVERY - Disk space on mw20 is OK: DISK OK [21:27:15] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [21:27:15] RECOVERY - Apache HTTP on mw17 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [21:27:15] RECOVERY - DPKG on mw19 is OK: All packages OK [21:28:25] RECOVERY - RAID on mw36 is OK: OK: no RAID installed [21:29:15] RECOVERY - Apache HTTP on mw12 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.034 second response time [21:29:35] RECOVERY - SSH on mw5 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:29:35] RECOVERY - Apache HTTP on mw48 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.030 second response time [21:29:35] RECOVERY - Apache HTTP on mw53 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [21:29:35] RECOVERY - RAID on mw48 is OK: OK: no RAID installed [21:29:35] RECOVERY - Disk space on mw52 is OK: DISK OK [21:29:36] RECOVERY - DPKG on mw5 is OK: All packages OK [21:29:45] RECOVERY - DPKG on srv193 is OK: All packages OK [21:30:5] RECOVERY - ps1-b5-sdtpa-infeed-load-tower-A-phase-Y on ps1-b5-sdtpa is OK: ps1-b5-sdtpa-infeed-load-tower-A-phase-Y OK - 1900 [21:30:5] RECOVERY - ps1-c1-sdtpa-infeed-load-tower-B-phase-Z on ps1-c1-sdtpa is OK: ps1-c1-sdtpa-infeed-load-tower-B-phase-Z OK - 638 [21:30:5] RECOVERY - ps1-c1-sdtpa-infeed-load-tower-A-phase-X on ps1-c1-sdtpa is OK: ps1-c1-sdtpa-infeed-load-tower-A-phase-X OK - 525 [21:30:5] RECOVERY - ps1-b8-eqiad-infeed-load-tower-A-phase-Y on ps1-b8-eqiad is OK: ps1-b8-eqiad-infeed-load-tower-A-phase-Y OK - 588 [21:30:5] RECOVERY - ps1-b7-eqiad-infeed-load-tower-A-phase-Z on ps1-b7-eqiad is OK: ps1-b7-eqiad-infeed-load-tower-A-phase-Z OK - 650 [21:30:6] RECOVERY - ps1-b5-eqiad-infeed-load-tower-A-phase-Z on ps1-b5-eqiad is OK: ps1-b5-eqiad-infeed-load-tower-A-phase-Z OK - 638 [21:30:6] RECOVERY - ps1-b6-eqiad-infeed-load-tower-B-phase-X on ps1-b6-eqiad is OK: ps1-b6-eqiad-infeed-load-tower-B-phase-X OK - 200 [21:30:35] RECOVERY - SSH on mw57 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:30:35] RECOVERY - SSH on mw8 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:30:35] RECOVERY - ps1-d3-pmtpa-infeed-load-tower-B-phase-Z on ps1-d3-pmtpa is OK: ps1-d3-pmtpa-infeed-load-tower-B-phase-Z OK - 25 [21:30:35] RECOVERY - RAID on mw40 is OK: OK: no RAID installed [21:30:35] RECOVERY - DPKG on mw52 is OK: All packages OK [21:31:5] RECOVERY - DPKG on mw8 is OK: All packages OK [21:31:25] RECOVERY - SSH on srv231 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:31:25] RECOVERY - Disk space on srv233 is OK: DISK OK [21:31:25] RECOVERY - Apache HTTP on srv234 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [21:31:25] RECOVERY - RAID on srv234 is OK: OK: no RAID installed [21:31:25] RECOVERY - DPKG on srv231 is OK: All packages OK [21:31:35] RECOVERY - Disk space on srv265 is OK: DISK OK [21:31:35] RECOVERY - Apache HTTP on srv267 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.019 second response time [21:31:55] RECOVERY - RAID on srv267 is OK: OK: no RAID installed [21:32:25] RECOVERY - SSH on mw42 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:32:45] RECOVERY - Disk space on srv235 is OK: DISK OK [21:32:55] RECOVERY - Disk space on srv262 is OK: DISK OK [21:32:55] RECOVERY - DPKG on srv260 is OK: All packages OK [21:33:24] RECOVERY - ps1-a6-eqiad-infeed-load-tower-A-phase-Y on ps1-a6-eqiad is OK: ps1-a6-eqiad-infeed-load-tower-A-phase-Y OK - 575 [21:33:25] RECOVERY - ps1-a8-eqiad-infeed-load-tower-B-phase-Y on ps1-a8-eqiad is OK: ps1-a8-eqiad-infeed-load-tower-B-phase-Y OK - 392 [21:33:35] RECOVERY - Apache HTTP on srv226 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [21:33:35] RECOVERY - RAID on srv279 is OK: OK: no RAID installed [21:34:5] RECOVERY - ps1-d1-sdtpa-infeed-load-tower-A-phase-X on ps1-d1-sdtpa is OK: ps1-d1-sdtpa-infeed-load-tower-A-phase-X OK - 1838 [21:34:5] RECOVERY - ps1-a1-sdtpa-infeed-load-tower-A-phase-Y on ps1-a1-sdtpa is OK: ps1-a1-sdtpa-infeed-load-tower-A-phase-Y OK - 538 [21:34:35] RECOVERY - SSH on mw17 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:35:35] RECOVERY - Disk space on mw17 is OK: DISK OK [21:35:35] RECOVERY - SSH on mw15 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:35:35] RECOVERY - Apache HTTP on mw13 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.017 second response time [21:35:35] RECOVERY - Apache HTTP on mw18 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [21:35:35] RECOVERY - RAID on mw13 is OK: OK: no RAID installed [21:35:36] RECOVERY - DPKG on mw15 is OK: All packages OK [21:35:45] RECOVERY - ps1-a5-sdtpa-infeed-load-tower-A-phase-X on ps1-a5-sdtpa is OK: ps1-a5-sdtpa-infeed-load-tower-A-phase-X OK - 2125 [21:35:45] RECOVERY - ps1-a8-eqiad-infeed-load-tower-B-phase-Z on ps1-a8-eqiad is OK: ps1-a8-eqiad-infeed-load-tower-B-phase-Z OK - 334 [21:35:45] RECOVERY - ps1-a4-eqiad-infeed-load-tower-B-phase-Z on ps1-a4-eqiad is OK: ps1-a4-eqiad-infeed-load-tower-B-phase-Z OK - 588 [21:35:45] RECOVERY - ps1-a6-eqiad-infeed-load-tower-A-phase-Z on ps1-a6-eqiad is OK: ps1-a6-eqiad-infeed-load-tower-A-phase-Z OK - 638 [21:35:55] RECOVERY - SSH on srv263 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:35:55] RECOVERY - RAID on srv244 is OK: OK: no RAID installed [21:36:5] RECOVERY - SSH on mw34 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:36:5] RECOVERY - Disk space on mw36 is OK: DISK OK [21:36:5] RECOVERY - Apache HTTP on mw37 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [21:36:5] RECOVERY - DPKG on mw34 is OK: All packages OK [21:36:5] RECOVERY - RAID on mw32 is OK: OK: no RAID installed [21:36:6] RECOVERY - RAID on mw37 is OK: OK: no RAID installed [21:36:15] RECOVERY - Apache HTTP on mw21 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [21:36:25] RECOVERY - Disk space on srv193 is OK: DISK OK [21:36:25] RECOVERY - DPKG on srv196 is OK: All packages OK [21:36:35] RECOVERY - SSH on srv193 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:36:45] RECOVERY - ps1-a2-sdtpa-infeed-load-tower-A-phase-Y on ps1-a2-sdtpa is OK: ps1-a2-sdtpa-infeed-load-tower-A-phase-Y OK - 1113 [21:36:45] RECOVERY - ps1-a1-sdtpa-infeed-load-tower-B-phase-X on ps1-a1-sdtpa is OK: ps1-a1-sdtpa-infeed-load-tower-B-phase-X OK - 600 [21:36:45] RECOVERY - ps1-a3-eqiad-infeed-load-tower-A-phase-X on ps1-a3-eqiad is OK: ps1-a3-eqiad-infeed-load-tower-A-phase-X OK - 700 [21:36:45] RECOVERY - ps1-a2-eqiad-infeed-load-tower-A-phase-Z on ps1-a2-eqiad is OK: ps1-a2-eqiad-infeed-load-tower-A-phase-Z OK - 613 [21:36:45] RECOVERY - ps1-a3-eqiad-infeed-load-tower-B-phase-Z on ps1-a3-eqiad is OK: ps1-a3-eqiad-infeed-load-tower-B-phase-Z OK - 738 [21:36:46] RECOVERY - ps1-a1-eqiad-infeed-load-tower-B-phase-Y on ps1-a1-eqiad is OK: ps1-a1-eqiad-infeed-load-tower-B-phase-Y OK - 401 [21:36:55] RECOVERY - RAID on mw9 is OK: OK: no RAID installed [21:36:55] RECOVERY - ps1-a8-eqiad-infeed-load-tower-A-phase-X on ps1-a8-eqiad is OK: ps1-a8-eqiad-infeed-load-tower-A-phase-X OK - 399 [21:36:55] RECOVERY - ps1-b3-eqiad-infeed-load-tower-A-phase-X on ps1-b3-eqiad is OK: ps1-b3-eqiad-infeed-load-tower-A-phase-X OK - 950 [21:37:5] RECOVERY - Disk space on srv243 is OK: DISK OK [21:37:5] RECOVERY - Apache HTTP on srv261 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [21:37:15] RECOVERY - DPKG on mw27 is OK: All packages OK [21:37:35] RECOVERY - RAID on srv286 is OK: OK: no RAID installed [21:37:45] RECOVERY - DPKG on mw14 is OK: All packages OK [21:38:5] RECOVERY - SSH on srv210 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:38:5] RECOVERY - Disk space on srv212 is OK: DISK OK [21:38:5] RECOVERY - Apache HTTP on srv213 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [21:38:5] RECOVERY - RAID on srv213 is OK: OK: no RAID installed [21:38:5] RECOVERY - DPKG on srv210 is OK: All packages OK [21:38:25] RECOVERY - Disk space on mw8 is OK: DISK OK [21:38:35] RECOVERY - RAID on mw45 is OK: OK: no RAID installed [21:38:35] RECOVERY - ps1-b2-eqiad-infeed-load-tower-A-phase-Y on ps1-b2-eqiad is OK: ps1-b2-eqiad-infeed-load-tower-A-phase-Y OK - 713 [21:38:55] RECOVERY - Apache HTTP on mw45 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [21:38:55] RECOVERY - RAID on srv277 is OK: OK: no RAID installed [21:39:5] RECOVERY - RAID on mw34 is OK: OK: no RAID installed [21:39:15] RECOVERY - Disk space on mw57 is OK: DISK OK [21:39:15] RECOVERY - Disk space on mw12 is OK: DISK OK [21:39:15] RECOVERY - Disk space on srv273 is OK: DISK OK [21:39:15] RECOVERY - RAID on mw18 is OK: OK: no RAID installed [21:39:45] RECOVERY - Apache HTTP on srv236 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [21:39:45] RECOVERY - ps1-a5-eqiad-infeed-load-tower-A-phase-Y on ps1-a5-eqiad is OK: ps1-a5-eqiad-infeed-load-tower-A-phase-Y OK - 988 [21:39:45] RECOVERY - RAID on mw25 is OK: OK: no RAID installed [21:39:55] RECOVERY - SSH on mw19 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:40:35] RECOVERY - SSH on mw43 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:40:35] RECOVERY - Disk space on mw27 is OK: DISK OK [21:40:35] RECOVERY - RAID on mw22 is OK: OK: no RAID installed [21:40:35] RECOVERY - RAID on mw46 is OK: OK: no RAID installed [21:40:55] RECOVERY - SSH on srv286 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:40:55] RECOVERY - SSH on srv269 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:40:55] RECOVERY - Disk space on srv288 is OK: DISK OK [21:40:55] RECOVERY - RAID on srv289 is OK: OK: no RAID installed [21:41:14] RECOVERY - Apache HTTP on mw10 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [21:41:54] RECOVERY - DPKG on mw12 is OK: All packages OK [21:42:35] RECOVERY - Apache HTTP on srv201 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [21:42:35] RECOVERY - DPKG on srv282 is OK: All packages OK [21:42:35] RECOVERY - DPKG on mw1 is OK: All packages OK [21:42:45] RECOVERY - RAID on srv196 is OK: OK: no RAID installed [21:42:45] RECOVERY - ps1-a7-eqiad-infeed-load-tower-A-phase-Z on ps1-a7-eqiad is OK: ps1-a7-eqiad-infeed-load-tower-A-phase-Z OK - 625 [21:42:45] RECOVERY - ps1-b5-eqiad-infeed-load-tower-B-phase-X on ps1-b5-eqiad is OK: ps1-b5-eqiad-infeed-load-tower-B-phase-X OK - 638 [21:42:54] RECOVERY - SSH on srv234 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:42:54] RECOVERY - Disk space on srv241 is OK: DISK OK [21:42:54] RECOVERY - DPKG on srv198 is OK: All packages OK [21:42:55] RECOVERY - RAID on srv227 is OK: OK: no RAID installed [21:42:55] RECOVERY - DPKG on srv229 is OK: All packages OK [21:43:25] RECOVERY - Apache HTTP on srv272 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.027 second response time [21:43:25] RECOVERY - Disk space on mw4 is OK: DISK OK [21:43:25] RECOVERY - Disk space on srv271 is OK: DISK OK [21:43:25] RECOVERY - SSH on mw45 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:43:25] RECOVERY - DPKG on srv279 is OK: All packages OK [21:43:35] RECOVERY - Disk space on srv211 is OK: DISK OK [21:43:45] RECOVERY - SSH on mw40 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:44:5] RECOVERY - Disk space on srv190 is OK: DISK OK [21:44:5] RECOVERY - Disk space on srv209 is OK: DISK OK [21:44:5] RECOVERY - RAID on srv226 is OK: OK: no RAID installed [21:44:15] RECOVERY - ps1-d2-pmtpa-infeed-load-tower-B-phase-Y on ps1-d2-pmtpa is OK: ps1-d2-pmtpa-infeed-load-tower-B-phase-Y OK - 50 [21:44:25] RECOVERY - Disk space on srv238 is OK: DISK OK [21:44:25] RECOVERY - SSH on srv267 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:44:25] RECOVERY - DPKG on srv263 is OK: All packages OK [21:44:35] RECOVERY - SSH on mw37 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:44:35] RECOVERY - Disk space on mw34 is OK: DISK OK [21:44:35] RECOVERY - Disk space on mw39 is OK: DISK OK [21:44:35] RECOVERY - Apache HTTP on mw35 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [21:44:35] RECOVERY - RAID on mw35 is OK: OK: no RAID installed [21:44:36] RECOVERY - DPKG on mw37 is OK: All packages OK [21:44:45] RECOVERY - Disk space on mw44 is OK: DISK OK [21:44:45] RECOVERY - Apache HTTP on srv207 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.035 second response time [21:44:45] RECOVERY - DPKG on mw42 is OK: All packages OK [21:44:45] RECOVERY - Disk space on mw24 is OK: DISK OK [21:44:55] RECOVERY - ps1-b2-sdtpa-infeed-load-tower-A-phase-X on ps1-b2-sdtpa is OK: ps1-b2-sdtpa-infeed-load-tower-A-phase-X OK - 1088 [21:44:55] RECOVERY - Disk space on srv276 is OK: DISK OK [21:45:5] RECOVERY - SSH on srv236 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:45:5] RECOVERY - Apache HTTP on srv194 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [21:45:5] RECOVERY - Disk space on srv225 is OK: DISK OK [21:45:5] RECOVERY - RAID on mw10 is OK: OK: no RAID installed [21:45:15] RECOVERY - Disk space on mw49 is OK: DISK OK [21:45:15] RECOVERY - Disk space on mw29 is OK: DISK OK [21:45:25] RECOVERY - Disk space on mw9 is OK: DISK OK [21:45:45] RECOVERY - Apache HTTP on srv263 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.022 second response time [21:45:45] RECOVERY - RAID on mw12 is OK: OK: no RAID installed [21:45:45] RECOVERY - DPKG on srv209 is OK: All packages OK [21:46:5] RECOVERY - Apache HTTP on mw9 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.018 second response time [21:46:25] RECOVERY - Apache HTTP on mw3 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.016 second response time [21:46:35] RECOVERY - Disk space on srv231 is OK: DISK OK [21:46:35] RECOVERY - DPKG on srv228 is OK: All packages OK [21:46:45] RECOVERY - ps1-b5-sdtpa-infeed-load-tower-A-phase-X on ps1-b5-sdtpa is OK: ps1-b5-sdtpa-infeed-load-tower-A-phase-X OK - 1650 [21:46:54] RECOVERY - SSH on mw33 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:46:55] RECOVERY - DPKG on mw58 is OK: All packages OK [21:47:5] RECOVERY - Disk space on srv194 is OK: DISK OK [21:47:5] RECOVERY - Apache HTTP on srv195 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.022 second response time [21:47:15] RECOVERY - Apache HTTP on mw15 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.029 second response time [21:47:24] RECOVERY - Disk space on srv274 is OK: DISK OK [21:47:24] RECOVERY - SSH on srv283 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:47:24] RECOVERY - DPKG on srv272 is OK: All packages OK [21:47:24] RECOVERY - DPKG on srv283 is OK: All packages OK [21:47:25] RECOVERY - Apache HTTP on srv280 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.034 second response time [21:47:25] RECOVERY - DPKG on srv277 is OK: All packages OK [21:47:55] RECOVERY - SSH on srv247 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:47:55] RECOVERY - DPKG on srv247 is OK: All packages OK [21:47:55] RECOVERY - RAID on srv245 is OK: OK: no RAID installed [21:48:5] RECOVERY - SSH on srv204 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:48:5] RECOVERY - RAID on srv210 is OK: OK: no RAID installed [21:48:5] RECOVERY - ps1-a2-eqiad-infeed-load-tower-A-phase-Y on ps1-a2-eqiad is OK: ps1-a2-eqiad-infeed-load-tower-A-phase-Y OK - 650 [21:48:15] RECOVERY - DPKG on mw45 is OK: All packages OK [21:48:25] RECOVERY - DPKG on mw10 is OK: All packages OK [21:48:45] RECOVERY - SSH on srv275 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:48:45] RECOVERY - Disk space on srv277 is OK: DISK OK [21:48:45] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [21:48:45] RECOVERY - RAID on srv278 is OK: OK: no RAID installed [21:48:45] RECOVERY - RAID on srv273 is OK: OK: no RAID installed [21:48:46] RECOVERY - DPKG on srv275 is OK: All packages OK [21:48:55] RECOVERY - Disk space on srv201 is OK: DISK OK [21:48:55] RECOVERY - ps1-b4-eqiad-infeed-load-tower-A-phase-Y on ps1-b4-eqiad is OK: ps1-b4-eqiad-infeed-load-tower-A-phase-Y OK - 675 [21:49:24] RECOVERY - Disk space on srv263 is OK: DISK OK [21:49:24] RECOVERY - DPKG on mw40 is OK: All packages OK [21:50:5] RECOVERY - ps1-c2-sdtpa-infeed-load-tower-A-phase-X on ps1-c2-sdtpa is OK: ps1-c2-sdtpa-infeed-load-tower-A-phase-X OK - 1000 [21:50:15] RECOVERY - Disk space on srv205 is OK: DISK OK [21:50:44] RECOVERY - ps1-b1-sdtpa-infeed-load-tower-A-phase-Z on ps1-b1-sdtpa is OK: ps1-b1-sdtpa-infeed-load-tower-A-phase-Z OK - 659 [21:50:55] RECOVERY - ps1-d1-pmtpa-infeed-load-tower-B-phase-X on ps1-d1-pmtpa is OK: ps1-d1-pmtpa-infeed-load-tower-B-phase-X OK - 163 [21:50:55] RECOVERY - RAID on mw7 is OK: OK: no RAID installed [21:50:55] RECOVERY - Apache HTTP on mw51 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [21:51:5] RECOVERY - ps1-d1-pmtpa-infeed-load-tower-A-phase-Y on ps1-d1-pmtpa is OK: ps1-d1-pmtpa-infeed-load-tower-A-phase-Y OK - 113 [21:51:5] RECOVERY - DPKG on mw18 is OK: All packages OK [21:51:35] RECOVERY - SSH on mw48 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:51:39] New patchset: Asher; "exim: use db48/49 to check incoming otrs mail" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1544 [21:51:44] RECOVERY - SSH on srv198 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:51:45] RECOVERY - Apache HTTP on srv229 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [21:51:45] RECOVERY - RAID on mw11 is OK: OK: no RAID installed [21:52:24] RECOVERY - SSH on srv238 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:52:25] RECOVERY - RAID on srv212 is OK: OK: no RAID installed [21:52:35] RECOVERY - Apache HTTP on mw58 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.024 second response time [21:52:35] RECOVERY - DPKG on mw41 is OK: All packages OK [21:52:35] RECOVERY - DPKG on mw22 is OK: All packages OK [21:52:35] RECOVERY - DPKG on mw46 is OK: All packages OK [21:52:44] RECOVERY - ps1-d3-sdtpa-infeed-load-tower-A-phase-Y on ps1-d3-sdtpa is OK: ps1-d3-sdtpa-infeed-load-tower-A-phase-Y OK - 1363 [21:52:44] RECOVERY - RAID on srv197 is OK: OK: no RAID installed [21:52:55] RECOVERY - SSH on srv196 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:53:15] RECOVERY - Disk space on mw35 is OK: DISK OK [21:53:15] RECOVERY - ps1-a7-eqiad-infeed-load-tower-A-phase-Y on ps1-a7-eqiad is OK: ps1-a7-eqiad-infeed-load-tower-A-phase-Y OK - 588 [21:53:24] RECOVERY - Disk space on srv226 is OK: DISK OK [21:53:24] RECOVERY - DPKG on mw6 is OK: All packages OK [21:53:35] RECOVERY - Disk space on srv195 is OK: DISK OK [21:53:35] RECOVERY - Disk space on srv199 is OK: DISK OK [21:53:35] RECOVERY - RAID on srv205 is OK: OK: no RAID installed [21:53:44] RECOVERY - Disk space on mw11 is OK: DISK OK [21:53:44] RECOVERY - Apache HTTP on srv208 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [21:53:44] RECOVERY - DPKG on srv270 is OK: All packages OK [21:53:44] RECOVERY - RAID on srv268 is OK: OK: no RAID installed [21:54:15] RECOVERY - SSH on mw2 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:54:24] RECOVERY - ps1-b1-sdtpa-infeed-load-tower-B-phase-Y on ps1-b1-sdtpa is OK: ps1-b1-sdtpa-infeed-load-tower-B-phase-Y OK - 593 [21:54:25] RECOVERY - Apache HTTP on mw39 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [21:54:35] RECOVERY - SSH on srv261 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:54:35] RECOVERY - RAID on srv287 is OK: OK: no RAID installed [21:54:45] RECOVERY - SSH on mw30 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:54:45] RECOVERY - SSH on mw35 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:54:45] RECOVERY - Disk space on mw32 is OK: DISK OK [21:54:45] RECOVERY - Apache HTTP on mw33 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [21:54:45] RECOVERY - RAID on mw33 is OK: OK: no RAID installed [21:54:46] RECOVERY - DPKG on mw35 is OK: All packages OK [21:55:5] RECOVERY - Disk space on mw40 is OK: DISK OK [21:55:5] RECOVERY - SSH on mw7 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:55:5] RECOVERY - ps1-b2-sdtpa-infeed-load-tower-B-phase-X on ps1-b2-sdtpa is OK: ps1-b2-sdtpa-infeed-load-tower-B-phase-X OK - 1113 [21:55:5] RECOVERY - ps1-c1-sdtpa-infeed-load-tower-A-phase-Z on ps1-c1-sdtpa is OK: ps1-c1-sdtpa-infeed-load-tower-A-phase-Z OK - 588 [21:55:5] RECOVERY - ps1-b6-eqiad-infeed-load-tower-A-phase-X on ps1-b6-eqiad is OK: ps1-b6-eqiad-infeed-load-tower-A-phase-X OK - 213 [21:55:35] RECOVERY - Disk space on mw33 is OK: DISK OK [21:55:44] RECOVERY - RAID on mw8 is OK: OK: no RAID installed [21:55:55] RECOVERY - SSH on srv229 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:55:55] RECOVERY - SSH on mw14 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:55:55] RECOVERY - Apache HTTP on srv279 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [21:55:55] RECOVERY - RAID on srv269 is OK: OK: no RAID installed [21:55:56] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1544 [21:55:56] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1544 [21:56:5] RECOVERY - Disk space on mw6 is OK: DISK OK [21:56:5] RECOVERY - Apache HTTP on srv197 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [21:56:5] RECOVERY - ps1-b4-eqiad-infeed-load-tower-A-phase-Z on ps1-b4-eqiad is OK: ps1-b4-eqiad-infeed-load-tower-A-phase-Z OK - 575 [21:56:15] RECOVERY - Disk space on srv240 is OK: DISK OK [21:56:24] RECOVERY - SSH on mw13 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:56:24] RECOVERY - Disk space on srv282 is OK: DISK OK [21:56:25] RECOVERY - DPKG on srv289 is OK: All packages OK [21:56:35] RECOVERY - ps1-d1-pmtpa-infeed-load-tower-B-phase-Z on ps1-d1-pmtpa is OK: ps1-d1-pmtpa-infeed-load-tower-B-phase-Z OK - 250 [21:56:35] RECOVERY - ps1-d3-pmtpa-infeed-load-tower-B-phase-Y on ps1-d3-pmtpa is OK: ps1-d3-pmtpa-infeed-load-tower-B-phase-Y OK - 50 [21:56:35] RECOVERY - ps1-d1-pmtpa-infeed-load-tower-A-phase-X on ps1-d1-pmtpa is OK: ps1-d1-pmtpa-infeed-load-tower-A-phase-X OK - 225 [21:56:55] RECOVERY - ps1-d2-pmtpa-infeed-load-tower-B-phase-Z on ps1-d2-pmtpa is OK: ps1-d2-pmtpa-infeed-load-tower-B-phase-Z OK - 0 [21:56:55] RECOVERY - DPKG on mw33 is OK: All packages OK [21:57:5] RECOVERY - SSH on srv227 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:57:5] RECOVERY - Disk space on mw42 is OK: DISK OK [21:57:5] RECOVERY - Apache HTTP on srv262 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.018 second response time [21:57:5] RECOVERY - ps1-b1-eqiad-infeed-load-tower-B-phase-X on ps1-b1-eqiad is OK: ps1-b1-eqiad-infeed-load-tower-B-phase-X OK - 600 [21:57:15] RECOVERY - SSH on mw32 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:57:15] RECOVERY - ps1-a1-sdtpa-infeed-load-tower-A-phase-Z on ps1-a1-sdtpa is OK: ps1-a1-sdtpa-infeed-load-tower-A-phase-Z OK - 613 [21:57:25] RECOVERY - SSH on mw52 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:57:35] RECOVERY - Apache HTTP on srv190 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [21:57:35] RECOVERY - RAID on srv193 is OK: OK: no RAID installed [21:57:35] RECOVERY - RAID on srv190 is OK: OK: no RAID installed [21:57:35] RECOVERY - DPKG on srv195 is OK: All packages OK [21:58:4] RECOVERY - SSH on srv207 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:58:4] RECOVERY - RAID on srv258 is OK: OK: no RAID installed [21:58:14] RECOVERY - ps1-a4-sdtpa-infeed-load-tower-A-phase-X on ps1-a4-sdtpa is OK: ps1-a4-sdtpa-infeed-load-tower-A-phase-X OK - 1325 [21:58:25] RECOVERY - SSH on srv225 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:58:25] RECOVERY - SSH on srv230 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [21:58:25] RECOVERY - Disk space on srv227 is OK: DISK OK [21:58:25] RECOVERY - RAID on srv228 is OK: OK: no RAID installed [21:58:25] RECOVERY - Apache HTTP on srv228 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.022 second response time [21:58:26] RECOVERY - DPKG on srv230 is OK: All packages OK [21:58:35] RECOVERY - Apache HTTP on srv260 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.020 second response time [21:58:44] RECOVERY - ps1-b6-eqiad-infeed-load-tower-A-phase-Z on ps1-b6-eqiad is OK: ps1-b6-eqiad-infeed-load-tower-A-phase-Z OK - 250 [21:58:44] RECOVERY - RAID on srv263 is OK: OK: no RAID installed [21:58:44] RECOVERY - DPKG on srv207 is OK: All packages OK [22:4:44] !log exim on mchenry is now using db48/49 for otrs mail verification [22:4:52] Logged the message, Master [22:13:38] !log otrs on williams migrated to db48 [22:13:46] Logged the message, Master [22:17:34] New patchset: Asher; "db48 is now a masterdb48 is now a masterdb48 is now a masterdb48 is now a master" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1545 [22:17:57] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1545 [22:17:57] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1545 [22:19:21] anyone have an idea why there are so many (Return code of 127 is out of bounds - plugin may be missing) errors ? [22:22:43] New patchset: Pyoungmeister; "adding https to noc.w.o" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1546 [22:25:27] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: Puppet has not run in the last 10 hours [22:25:27] PROBLEM - Puppet freshness on ssl2 is CRITICAL: Puppet has not run in the last 10 hours [22:25:27] PROBLEM - Puppet freshness on ssl3004 is CRITICAL: Puppet has not run in the last 10 hours [22:31:22] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1546 [22:35:36] !log dropping otrs from db9 [22:35:44] Logged the message, Master [22:36:28] grr, the 127's are back [22:37:58] !log ibdata1 on db9 now has 208GB of free space (InnoDB free: 218228736 kB) [22:38:6] Logged the message, Master [22:38:56] Omh [22:44:32] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1546 [22:44:53] 127's are same as last time, nagios is failing to spawn checks with E2BIG errors again [23:0:46] RECOVERY - ps1-b1-sdtpa-infeed-load-tower-A-phase-Y on ps1-b1-sdtpa is OK: ps1-b1-sdtpa-infeed-load-tower-A-phase-Y OK - 654 [23:0:46] RECOVERY - ps1-b4-eqiad-infeed-load-tower-B-phase-Y on ps1-b4-eqiad is OK: ps1-b4-eqiad-infeed-load-tower-B-phase-Y OK - 675 [23:0:46] RECOVERY - ps1-b5-eqiad-infeed-load-tower-A-phase-X on ps1-b5-eqiad is OK: ps1-b5-eqiad-infeed-load-tower-A-phase-X OK - 638 [23:0:46] RECOVERY - ps1-a7-eqiad-infeed-load-tower-B-phase-Y on ps1-a7-eqiad is OK: ps1-a7-eqiad-infeed-load-tower-B-phase-Y OK - 613 [23:0:47] RECOVERY - ps1-a3-eqiad-infeed-load-tower-A-phase-Z on ps1-a3-eqiad is OK: ps1-a3-eqiad-infeed-load-tower-A-phase-Z OK - 700 [23:2:7] RECOVERY - SSH on mw20 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:2:7] RECOVERY - Disk space on mw22 is OK: DISK OK [23:2:7] RECOVERY - RAID on mw19 is OK: OK: no RAID installed [23:2:7] RECOVERY - DPKG on mw20 is OK: All packages OK [23:2:7] RECOVERY - Apache HTTP on mw24 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.040 second response time [23:2:7] RECOVERY - RAID on mw24 is OK: OK: no RAID installed [23:2:17] RECOVERY - SSH on mw49 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:2:17] RECOVERY - Disk space on mw51 is OK: DISK OK [23:2:17] RECOVERY - SSH on mw54 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:2:17] RECOVERY - Apache HTTP on mw52 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.036 second response time [23:2:17] RECOVERY - RAID on mw52 is OK: OK: no RAID installed [23:2:18] RECOVERY - DPKG on mw54 is OK: All packages OK [23:2:26] !log restarted nagios with enable_environment_macros = 0 [23:2:26] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.022 second response time [23:2:34] Logged the message, Master [23:3:7] RECOVERY - RAID on srv203 is OK: OK: no RAID installed [23:3:7] RECOVERY - SSH on srv205 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:3:7] RECOVERY - Disk space on srv208 is OK: DISK OK [23:3:7] RECOVERY - Apache HTTP on srv209 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [23:3:7] RECOVERY - DPKG on srv205 is OK: All packages OK [23:3:7] RECOVERY - RAID on srv209 is OK: OK: no RAID installed [23:3:37] RECOVERY - RAID on mw14 is OK: OK: no RAID installed [23:3:48] New patchset: Asher; "more complete fix for checks failing with argument list too big errors in large host groups" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1547 [23:4:7] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1547 [23:4:7] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1547 [23:5:7] RECOVERY - ps1-b3-eqiad-infeed-load-tower-B-phase-X on ps1-b3-eqiad is OK: ps1-b3-eqiad-infeed-load-tower-B-phase-X OK - 950 [23:5:7] RECOVERY - ps1-a6-eqiad-infeed-load-tower-A-phase-X on ps1-a6-eqiad is OK: ps1-a6-eqiad-infeed-load-tower-A-phase-X OK - 600 [23:5:7] RECOVERY - ps1-b8-eqiad-infeed-load-tower-B-phase-Y on ps1-b8-eqiad is OK: ps1-b8-eqiad-infeed-load-tower-B-phase-Y OK - 625 [23:7:21] RECOVERY - DPKG on mw26 is OK: All packages OK [23:7:31] RECOVERY - Disk space on mw56 is OK: DISK OK [23:7:31] RECOVERY - Apache HTTP on mw57 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [23:8:21] RECOVERY - Disk space on srv213 is OK: DISK OK [23:8:31] RECOVERY - Apache HTTP on srv233 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.025 second response time [23:8:31] RECOVERY - SSH on srv235 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:8:31] RECOVERY - RAID on srv233 is OK: OK: no RAID installed [23:8:31] RECOVERY - Apache HTTP on srv238 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.021 second response time [23:8:31] RECOVERY - DPKG on srv235 is OK: All packages OK [23:8:31] RECOVERY - Disk space on srv237 is OK: DISK OK [23:8:51] RECOVERY - SSH on srv285 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [23:8:51] RECOVERY - Apache HTTP on srv282 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.026 second response time [23:8:51] RECOVERY - RAID on srv282 is OK: OK: no RAID installed [23:8:51] RECOVERY - Disk space on srv287 is OK: DISK OK [23:8:51] RECOVERY - DPKG on srv285 is OK: All packages OK [23:9:31]