[03:29:12] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 794.70 seconds [03:33:22] PROBLEM - puppet last run on mw2135 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIPCity.dat.gz] [03:55:21] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 192.70 seconds [04:00:51] RECOVERY - puppet last run on mw2135 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:08:11] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=742.90 Read Requests/Sec=3012.80 Write Requests/Sec=18.80 KBytes Read/Sec=43111.20 KBytes_Written/Sec=1618.80 [04:15:21] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=126.50 Read Requests/Sec=48.00 Write Requests/Sec=30.00 KBytes Read/Sec=590.60 KBytes_Written/Sec=330.40 [06:18:21] PROBLEM - nova-compute process on labvirt1011 is CRITICAL: PROCS CRITICAL: 2 processes with regex args ^/usr/bin/python /usr/bin/nova-compute [06:19:21] RECOVERY - nova-compute process on labvirt1011 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/nova-compute [09:23:12] (03PS2) 10Amire80: Add din to InterwikiSortOrders [mediawiki-config] - 10https://gerrit.wikimedia.org/r/365451 (https://phabricator.wikimedia.org/T168518) (owner: 10Reedy) [13:18:32] (03PS1) 10Lucas Werkmeister (WMDE): Add sandboxing directives to wdqs-blazegraph.service [puppet] - 10https://gerrit.wikimedia.org/r/365518 [13:22:06] (03CR) 10Lucas Werkmeister (WMDE): "Just a suggestion :) I tried this out locally, and the query service seems to run fine. Updating the data also works, at least with loadRe" [puppet] - 10https://gerrit.wikimedia.org/r/365518 (owner: 10Lucas Werkmeister (WMDE)) [13:53:37] PROBLEM - Check whether ferm is active by checking the default input chain on elastic1017 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [13:54:46] RECOVERY - Check whether ferm is active by checking the default input chain on elastic1017 is OK: OK ferm input default policy is set [15:40:58] 10Operations, 10Cassandra, 10Mobile-Content-Service, 10Reading-Infrastructure-Team-Backlog, 10Services (done): mobileapps 500s following reboot of restbase1007 - https://phabricator.wikimedia.org/T138314#3442061 (10mobrovac) 05Open>03Resolved a:03mobrovac Indeed. [23:09:41] (03PS1) 10Framawiki: Create 'rollbacker' user group in frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/365538 (https://phabricator.wikimedia.org/T170780) [23:49:53] 10Operations, 10Performance-Team, 10TemplateStyles, 10Traffic, and 3 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410#3442349 (10Johan) wikitech-l rather than wikitech-ambassadors-l, I'd say, they're more of the target audience for mediawiki.org and Wikitech. Poss...