[00:14:55] !log bromine - scheduled downtime, reboot for reinstall, upgrade to stretch, misc_static_services switched to codfw (T188163) [00:15:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:15:03] T188163: create codfw-equivalent of bromine, make webserver_misc_static active/active in misc varnish - https://phabricator.wikimedia.org/T188163 [00:18:07] (03PS1) 10Dzahn: static_bugzilla: reverse rsync direction after bromine reinstall [puppet] - 10https://gerrit.wikimedia.org/r/424727 (https://phabricator.wikimedia.org/T188163) [00:27:07] (03CR) 10Dzahn: [C: 032] static_bugzilla: reverse rsync direction after bromine reinstall [puppet] - 10https://gerrit.wikimedia.org/r/424727 (https://phabricator.wikimedia.org/T188163) (owner: 10Dzahn) [00:27:17] 10Operations, 10puppet-compiler, 10Release-Engineering-Team (Watching / External): Integrate the puppet compiler in the puppet CI pipeline - https://phabricator.wikimedia.org/T166066#4113673 (10EddieGP) [00:39:33] (03PS1) 10Dzahn: static_bugzilla: ensure absence of rsyncd after maintenance [puppet] - 10https://gerrit.wikimedia.org/r/424730 [00:40:11] (03CR) 10Dzahn: [C: 032] static_bugzilla: ensure absence of rsyncd after maintenance [puppet] - 10https://gerrit.wikimedia.org/r/424730 (owner: 10Dzahn) [00:42:06] (03PS1) 10Dzahn: Revert "misc_static_sites: temp disable bromine backend for reinstall" [puppet] - 10https://gerrit.wikimedia.org/r/424731 [00:43:56] (03PS2) 10Dzahn: Revert "misc_static_sites: temp disable bromine backend for reinstall" [puppet] - 10https://gerrit.wikimedia.org/r/424731 [00:44:38] (03PS3) 10Dzahn: Revert "misc_static_sites: temp disable bromine backend for reinstall" [puppet] - 10https://gerrit.wikimedia.org/r/424731 (https://phabricator.wikimedia.org/T188163) [00:44:57] (03CR) 10Dzahn: [C: 032] Revert "misc_static_sites: temp disable bromine backend for reinstall" [puppet] - 10https://gerrit.wikimedia.org/r/424731 (https://phabricator.wikimedia.org/T188163) (owner: 10Dzahn) [00:47:21] 10Operations, 10Availability, 10Patch-For-Review: create codfw-equivalent of bromine, make webserver_misc_static active/active in misc varnish - https://phabricator.wikimedia.org/T188163#4113698 (10Dzahn) [00:48:02] 10Operations, 10Availability, 10Patch-For-Review: create codfw-equivalent of bromine, make webserver_misc_static active/active in misc varnish - https://phabricator.wikimedia.org/T188163#3998458 (10Dzahn) 05Open>03Resolved All done! - we are active-active - both eqiad and codfw are on stretch [01:08:57] (03PS1) 10Dzahn: planet: update some moved URLs/more https for en.planet [puppet] - 10https://gerrit.wikimedia.org/r/424732 [01:09:38] (03PS2) 10Dzahn: planet: update some moved URLs/more https for en.planet [puppet] - 10https://gerrit.wikimedia.org/r/424732 [01:09:43] (03CR) 10Dzahn: [C: 032] planet: update some moved URLs/more https for en.planet [puppet] - 10https://gerrit.wikimedia.org/r/424732 (owner: 10Dzahn) [03:28:02] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 756.98 seconds [04:08:11] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 184.30 seconds [04:14:21] PROBLEM - Nginx local proxy to apache on mw2201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:11] RECOVERY - Nginx local proxy to apache on mw2201 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.206 second response time [06:29:42] PROBLEM - puppet last run on mw1323 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/field.sh] [06:32:31] PROBLEM - puppet last run on labvirt1010 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/sudoers] [06:57:31] RECOVERY - puppet last run on labvirt1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:59:41] RECOVERY - puppet last run on mw1323 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [07:28:34] !log disabled and cleaned up spam from @Farjksn on Phabricator [07:28:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:22:02] (03PS2) 10Gilles: Remove obsolete imagescaler logic from swift proxy [puppet] - 10https://gerrit.wikimedia.org/r/424594 (https://phabricator.wikimedia.org/T188062) (owner: 10Muehlenhoff) [08:22:32] (03CR) 10jerkins-bot: [V: 04-1] Remove obsolete imagescaler logic from swift proxy [puppet] - 10https://gerrit.wikimedia.org/r/424594 (https://phabricator.wikimedia.org/T188062) (owner: 10Muehlenhoff) [08:25:56] (03PS3) 10Gilles: Remove obsolete imagescaler logic from swift proxy [puppet] - 10https://gerrit.wikimedia.org/r/424594 (https://phabricator.wikimedia.org/T188062) (owner: 10Muehlenhoff) [08:35:35] (03PS2) 10Gilles: Fix $wgLocalFileRepo definition [mediawiki-config] - 10https://gerrit.wikimedia.org/r/424618 (https://phabricator.wikimedia.org/T191643) [09:42:31] (03PS1) 10Jcrespo: mariadb: Repool es2019 after maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/424745 (https://phabricator.wikimedia.org/T153440) [10:09:53] (03CR) 10Jcrespo: [C: 031] Make WMFMariaDB.py flake8 compliant [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/424558 (owner: 10Rduran) [12:23:48] (03PS1) 10Urbanecm: Add adm.dp.gov.ua to wgCopyUploadDomains, change if.gov.ua to www.if.gov.ua [mediawiki-config] - 10https://gerrit.wikimedia.org/r/424756 (https://phabricator.wikimedia.org/T191692) [12:27:03] (03PS1) 10Urbanecm: Enable RelatedArticles for vector at hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/424757 (https://phabricator.wikimedia.org/T191573) [12:52:21] PROBLEM - HHVM rendering on mw2211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:21] RECOVERY - HHVM rendering on mw2211 is OK: HTTP OK: HTTP/1.1 200 OK - 74119 bytes in 1.798 second response time [13:19:58] (03PS1) 10Imarlier: coal: Fix property name that indicates an oversample [puppet] - 10https://gerrit.wikimedia.org/r/424761 (https://phabricator.wikimedia.org/T191239) [13:21:33] (03PS2) 10Imarlier: coal: Fix property name that indicates an oversample [puppet] - 10https://gerrit.wikimedia.org/r/424761 (https://phabricator.wikimedia.org/T191239) [13:48:54] (03CR) 10Krinkle: [C: 031] coal: Fix property name that indicates an oversample [puppet] - 10https://gerrit.wikimedia.org/r/424761 (https://phabricator.wikimedia.org/T191239) (owner: 10Imarlier) [13:51:06] Anyone here who can merge a small change that's in the puppet repository for me? https://gerrit.wikimedia.org/r/#/c/424761/ -- it's a change to a utility that hte performance team runs, that is deployed via puppet [14:17:43] (03CR) 10Guycn2: [C: 031] "> Uploaded patch set 1." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/424757 (https://phabricator.wikimedia.org/T191573) (owner: 10Urbanecm) [15:00:17] Anyone here who can merge https://gerrit.wikimedia.org/r/#/c/424761/ for me, in the puppet repo? [15:00:48] marlier hi, i think many of the ops are out for the weekend. [15:02:12] Yeah, I know...just hoping someone might drop in who has +2 on ops/puppet.... [15:02:26] Can't hurt to ask, right? [15:02:31] yep [15:16:58] 10Puppet, 10Beta-Cluster-Infrastructure: deployment-secureredirexperiment puppet error - https://phabricator.wikimedia.org/T191663#4114240 (10MarcoAurelio) [15:18:57] argh [15:20:01] deployment-puppetdb01 is down, maybe it just needs to be deleted? [15:20:10] if so, where/how? [15:54:27] I poked Krenair about it on the task (T187736). As the creator of the instance he should know whether it's still needed. [15:54:28] T187736: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736 [17:22:47] 10Operations, 10Ops-Access-Requests: I would like access to the deployment hosts - https://phabricator.wikimedia.org/T191704#4114326 (10Imarlier) [18:13:39] 10Operations, 10Ops-Access-Requests: Access to the deployment hosts for Imarlier - https://phabricator.wikimedia.org/T191704#4114355 (10Aklapper) [20:06:27] Another try....anyone around who is able to merge a change to operations/puppet? [20:14:24] (03Draft1) 10MarcoAurelio: admin: change ssh key for Sharvaniharan [puppet] - 10https://gerrit.wikimedia.org/r/424778 [20:14:28] (03PS2) 10MarcoAurelio: admin: change ssh key for Sharvaniharan [puppet] - 10https://gerrit.wikimedia.org/r/424778 [20:15:32] (03PS3) 10MarcoAurelio: admin: change ssh key for Sharvaniharan [puppet] - 10https://gerrit.wikimedia.org/r/424778 (https://phabricator.wikimedia.org/T191673) [20:15:38] (03PS1) 10ArielGlenn: add ipv6 for rsyncs from your.org [puppet] - 10https://gerrit.wikimedia.org/r/424779 [20:16:33] 10Operations, 10Patch-For-Review: New ssh key for production - https://phabricator.wikimedia.org/T191673#4113459 (10MarcoAurelio) oops, sorry @MoritzMuehlenhoff - didn't saw you were the asignee for this task. [20:20:44] (03CR) 10ArielGlenn: [C: 032] add ipv6 for rsyncs from your.org [puppet] - 10https://gerrit.wikimedia.org/r/424779 (owner: 10ArielGlenn) [23:44:27] !log OATHAuth disabled for Wikimedia SUL global account Barek (T191708) [23:44:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:44:34] T191708: Disable Two-factor authentication for user Barek on en.wiki - https://phabricator.wikimedia.org/T191708