[03:26:45] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 852.37 seconds
[04:04:55] <icinga-wm>	 RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 143.91 seconds
[07:13:19] <icinga-wm>	 ACKNOWLEDGEMENT - HP RAID on ms-be1017 is CRITICAL: CRITICAL: Slot 1: OK: 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, 2I:4:1, 2I:4:2 - Controller: OK - Cache: Permanently Disabled - Cable Error - Battery/Capacitor: Recharging nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T172054
[07:13:23] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on ms-be1017 - https://phabricator.wikimedia.org/T172054#3484167 (10ops-monitoring-bot)
[07:42:54] <wikibugs>	 (03PS1) 10Elukey: statistics::packages: package 'virtualenv' not available on trusty [puppet] - 10https://gerrit.wikimedia.org/r/368576
[07:48:19] <wikibugs>	 (03CR) 10Elukey: [C: 032] statistics::packages: package 'virtualenv' not available on trusty [puppet] - 10https://gerrit.wikimedia.org/r/368576 (owner: 10Elukey)
[07:51:15] <icinga-wm>	 RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:51:25] <elukey>	 \o/
[07:54:14] <wikibugs>	 (03PS1) 10Elukey: admin::data::data.yaml: remove ironholds from absented users [puppet] - 10https://gerrit.wikimedia.org/r/368577 (https://phabricator.wikimedia.org/T171696)
[10:13:45] <wikibugs>	 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10Hindi-Sites, and 2 others: Create Wikiversity Hindi - https://phabricator.wikimedia.org/T168765#3484219 (10Urbanecm) Okay.
[10:17:32] <wikibugs>	 (03PS10) 10Urbanecm: Initial configuration for hiwikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368165 (https://phabricator.wikimedia.org/T168765)
[10:18:49] <wikibugs>	 (03PS1) 10Urbanecm: Remove expired throttle rules [mediawiki-config] - 10https://gerrit.wikimedia.org/r/368581
[10:20:30] <wikibugs>	 10Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 10MW-1.30-release-notes (WMF-deploy-2017-07-25_(1.30.0-wmf.11)), and 2 others: Create Dinka Wikipedia - https://phabricator.wikimedia.org/T168518#3484234 (10Urbanecm) 05Open>03Resolved Seems to be done - reopen if anything else needs...
[10:35:45] <icinga-wm>	 PROBLEM - HHVM jobrunner on mw1300 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 473 bytes in 0.001 second response time
[10:36:45] <icinga-wm>	 RECOVERY - HHVM jobrunner on mw1300 is OK: HTTP OK: HTTP/1.1 200 OK - 202 bytes in 0.002 second response time
[11:07:22] <wikibugs>	 (03Abandoned) 10Paladox: Update npm to 4.x and nodejs to 6.x [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/303370 (owner: 10Paladox)
[11:16:57] <wikibugs>	 (03PS5) 10Paladox: Gerrit: Enable logstash by default for prod gerrit [puppet] - 10https://gerrit.wikimedia.org/r/332531 (https://phabricator.wikimedia.org/T141324)
[11:17:41] <wikibugs>	 (03PS6) 10Paladox: Gerrit: Enable logstash by default for prod gerrit [puppet] - 10https://gerrit.wikimedia.org/r/332531 (https://phabricator.wikimedia.org/T141324)
[14:38:25] <icinga-wm>	 RECOVERY - Debian mirror in sync with upstream on sodium is OK: /srv/mirrors/debian is over 0 hours old.
[15:33:21] <icinga-wm>	 ACKNOWLEDGEMENT - HP RAID on ms-be1017 is CRITICAL: CRITICAL: Slot 1: OK: 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, 2I:4:1, 2I:4:2 - Controller: OK - Cache: Permanently Disabled - Cable Error - Battery/Capacitor: Recharging nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T172062
[15:33:25] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on ms-be1017 - https://phabricator.wikimedia.org/T172062#3484416 (10ops-monitoring-bot)
[17:28:45] <icinga-wm>	 PROBLEM - Upload HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [1000.0]
[17:29:45] <icinga-wm>	 PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0]
[17:35:45] <icinga-wm>	 RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[17:36:45] <icinga-wm>	 RECOVERY - Upload HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0]
[21:00:59] <wikibugs>	 (03PS1) 10Zhuyifei1999: Quarry: Add package 'python-xlsxwriter' [puppet] - 10https://gerrit.wikimedia.org/r/368597 (https://phabricator.wikimedia.org/T76126)