[00:12:30] (03PS1) 10Dzahn: piwik: add support for stretch/PHP7 [puppet] - 10https://gerrit.wikimedia.org/r/453553 [00:19:11] (03PS1) 10Dzahn: ci::website: convert apache to httpd [puppet] - 10https://gerrit.wikimedia.org/r/453554 [00:50:14] (03PS1) 10Andrew Bogott: wmcs pdns-recursor: support a list of reverse-lookup zones [puppet] - 10https://gerrit.wikimedia.org/r/453557 (https://phabricator.wikimedia.org/T199578) [00:50:55] (03CR) 10jerkins-bot: [V: 04-1] wmcs pdns-recursor: support a list of reverse-lookup zones [puppet] - 10https://gerrit.wikimedia.org/r/453557 (https://phabricator.wikimedia.org/T199578) (owner: 10Andrew Bogott) [00:53:42] (03PS2) 10Andrew Bogott: wmcs pdns-recursor: support a list of reverse-lookup zones [puppet] - 10https://gerrit.wikimedia.org/r/453557 (https://phabricator.wikimedia.org/T199578) [00:54:20] (03CR) 10jerkins-bot: [V: 04-1] wmcs pdns-recursor: support a list of reverse-lookup zones [puppet] - 10https://gerrit.wikimedia.org/r/453557 (https://phabricator.wikimedia.org/T199578) (owner: 10Andrew Bogott) [00:57:19] (03PS3) 10Andrew Bogott: wmcs pdns-recursor: support a list of reverse-lookup zones [puppet] - 10https://gerrit.wikimedia.org/r/453557 (https://phabricator.wikimedia.org/T199578) [01:13:41] (03PS4) 10Andrew Bogott: wmcs pdns-recursor: support a list of reverse-lookup zones [puppet] - 10https://gerrit.wikimedia.org/r/453557 (https://phabricator.wikimedia.org/T199578) [01:20:51] (03CR) 10Andrew Bogott: [C: 032] wmcs pdns-recursor: support a list of reverse-lookup zones [puppet] - 10https://gerrit.wikimedia.org/r/453557 (https://phabricator.wikimedia.org/T199578) (owner: 10Andrew Bogott) [01:25:07] (03PS1) 10Andrew Bogott: eqiad1 pdns: fix copy/paste error [puppet] - 10https://gerrit.wikimedia.org/r/453558 (https://phabricator.wikimedia.org/T199578) [01:25:46] (03CR) 10Andrew Bogott: [C: 032] eqiad1 pdns: fix copy/paste error [puppet] - 10https://gerrit.wikimedia.org/r/453558 (https://phabricator.wikimedia.org/T199578) (owner: 10Andrew Bogott) [03:25:29] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 793.25 seconds [03:48:39] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 151.47 seconds [04:37:54] (03CR) 10Legoktm: [C: 032] php72: Add more missing extensions that php5.6 had [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/453064 (https://phabricator.wikimedia.org/T188318) (owner: 10Legoktm) [04:38:17] (03Merged) 10jenkins-bot: php72: Add more missing extensions that php5.6 had [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/453064 (https://phabricator.wikimedia.org/T188318) (owner: 10Legoktm) [05:46:46] (03CR) 10Krinkle: "Intended difference:" [puppet] - 10https://gerrit.wikimedia.org/r/452744 (owner: 10Krinkle) [06:02:42] (03CR) 10Krinkle: [C: 031] ci::website: convert apache to httpd [puppet] - 10https://gerrit.wikimedia.org/r/453554 (owner: 10Dzahn) [06:03:29] PROBLEM - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is alerting: 70% GET drop in 30min alert. [06:04:38] RECOVERY - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is not alerting. [06:27:59] PROBLEM - puppet last run on dbproxy1010 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/phaste] [06:28:18] PROBLEM - puppet last run on mw1323 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/mysql-ps1.sh] [06:40:58] PROBLEM - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is alerting: 70% GET drop in 30min alert. [06:41:59] RECOVERY - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is not alerting. [06:57:19] PROBLEM - HHVM jobrunner on mw1301 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:18] RECOVERY - HHVM jobrunner on mw1301 is OK: HTTP OK: HTTP/1.1 200 OK - 206 bytes in 0.002 second response time [06:58:18] RECOVERY - puppet last run on dbproxy1010 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:58:28] RECOVERY - puppet last run on mw1323 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:06:23] (03CR) 10Volans: "Second thought inline ;)" (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/453373 (https://phabricator.wikimedia.org/T199079) (owner: 10Volans) [07:08:07] (03CR) 10Volans: "reply inline" (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/451254 (https://phabricator.wikimedia.org/T199079) (owner: 10Volans) [07:47:29] PROBLEM - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is alerting: 70% GET drop in 30min alert. [07:49:39] RECOVERY - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is not alerting. [07:54:59] PROBLEM - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is alerting: 70% GET drop in 30min alert. [07:57:18] RECOVERY - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is not alerting. [09:40:49] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [09:45:09] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [09:54:38] (03CR) 10MarcoAurelio: [C: 031] "Apparently (per the "Cannot Merge" message) this'd need a [manual?] rebase." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/450450 (owner: 10Gergő Tisza) [10:00:36] (03PS3) 10Urbanecm: Allow all bureaucrats to remove interface-admin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/450450 (owner: 10Gergő Tisza) [10:02:35] (03CR) 10Urbanecm: [C: 031] "Almost all patches in this repository need a rebase during merging, usually automatical rebase is enough :) (as in this case)." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/450450 (owner: 10Gergő Tisza) [10:56:55] (03CR) 10Matěj Suchánek: "Can this get +1 for the rest?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/449017 (owner: 10Matěj Suchánek) [13:06:57] 10Operations, 10Traffic, 10Wikidata, 10wikiba.se, 10Patch-For-Review: [Task] move wikiba.se webhosting to wikimedia misc-cluster - https://phabricator.wikimedia.org/T99531 (10Addshore) What's the status here? Is this still blocked on a decision? Can we try to move forward with 2? Ping @BBlack & @faidon [13:43:00] 10Operations, 10Traffic, 10Wikidata, 10wikiba.se, 10Patch-For-Review: [Task] move wikiba.se webhosting to wikimedia misc-cluster - https://phabricator.wikimedia.org/T99531 (10BBlack) There are plans underway at this point to support multiple LE certs on our standard cache terminators via the work in T199... [14:09:53] (03CR) 10Rxy: [C: 031] Allow all bureaucrats to remove interface-admin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/450450 (owner: 10Gergő Tisza) [15:10:39] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [15:12:49] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [16:24:59] PROBLEM - Check health of redis instance on 6382 on rdb1004 is CRITICAL: CRITICAL ERROR - Redis Library - can not ping 127.0.0.1 on port 6382 [16:26:18] RECOVERY - Check health of redis instance on 6382 on rdb1004 is OK: OK: REDIS 2.8.17 on 127.0.0.1:6382 has 1 databases (db0) with 6774982 keys, up 45 days 15 hours [17:33:58] PROBLEM - MediaWiki memcached error rate on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [5000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [17:38:09] RECOVERY - MediaWiki memcached error rate on graphite1001 is OK: OK: Less than 40.00% above the threshold [1000.0] https://grafana.wikimedia.org/dashboard/db/mediawiki-graphite-alerts?orgId=1&panelId=1&fullscreen [19:59:35] 10Operations, 10Wikimedia-Mailing-lists: Wikimedia Community User Group Albania mailing list request - https://phabricator.wikimedia.org/T201670 (10Sidorela) Hi Dzahn, Thankyou for you help. Can I ask a last thing. It seems to have an issue with subscription. When someone wants to subscribe there is a 403 For... [22:35:41] 10Operations, 10Wikimedia-Mailing-lists: Wikimedia Community User Group Albania mailing list request - https://phabricator.wikimedia.org/T201670 (10Dzahn) Hi @Sidorela how are they subscribing? I can't really confirm that behaviour: When i fill out the "subscribe" from on https://lists.wikimedia.org/mailma... [22:36:16] 10Operations, 10Wikimedia-Mailing-lists: Wikimedia Community User Group Albania mailing list request - https://phabricator.wikimedia.org/T201670 (10Dzahn) also: as an admin you can use a form in the admin interface to mass subscribe people or even lists of people all at once