[04:26:49] (03CR) 10Marostegui: [C: 03+2] install_server: Do not format /srv on dbstore1006 [puppet] - 10https://gerrit.wikimedia.org/r/693296 (https://phabricator.wikimedia.org/T283125) (owner: 10Marostegui) [04:27:15] (03PS1) 10Marostegui: Revert "db1142: Disable notifications." [puppet] - 10https://gerrit.wikimedia.org/r/693648 [04:28:08] (03CR) 10Marostegui: [C: 03+2] Revert "db1142: Disable notifications." [puppet] - 10https://gerrit.wikimedia.org/r/693648 (owner: 10Marostegui) [04:36:31] (03PS1) 10Marostegui: alias.yaml.erb: Fix new clouddb* replicas alias [puppet] - 10https://gerrit.wikimedia.org/r/693635 (https://phabricator.wikimedia.org/T282662) [04:41:47] (03CR) 10Marostegui: [C: 03+2] alias.yaml.erb: Fix new clouddb* replicas alias [puppet] - 10https://gerrit.wikimedia.org/r/693635 (https://phabricator.wikimedia.org/T282662) (owner: 10Marostegui) [04:52:49] 10SRE, 10ops-codfw, 10User-fgiunchedi: Degraded RAID on ms-be2038 - https://phabricator.wikimedia.org/T283401 (10Marostegui) p:05Triage→03Medium This doesn't look like a failed disk but a BBU failure. cc @fgiunchedi as service owner :) [04:53:50] 10ops-eqiad, 10DC-Ops: Relabel db1125 to be dbstore1006 - https://phabricator.wikimedia.org/T283300 (10Marostegui) p:05Triage→03Medium [04:59:09] 10SRE, 10Wikimedia-Mailing-lists, 10Wikisource-Community-User-Group: I lost the password for the Wikisource-l mailing list - https://phabricator.wikimedia.org/T282997 (10Marostegui) 05Open→03Resolved Closing this, please re-open if it still not fixed. [05:00:54] 10SRE, 10LDAP-Access-Requests, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users, Kerberos, and LDAP for ODimitrijevic - https://phabricator.wikimedia.org/T282836 (10Marostegui) 05Open→03Resolved a:03elukey Closing - please re-open if needed! [05:03:42] (03PS1) 10Marostegui: data.yaml: Replace kzeta's key [puppet] - 10https://gerrit.wikimedia.org/r/693636 (https://phabricator.wikimedia.org/T283383) [05:17:33] 10SRE, 10Wikimedia-Mailing-lists: Mailman3 bounce runner is running very slowly - https://phabricator.wikimedia.org/T282348 (10Legoktm) I have no idea what's wrong this time, but the bounce handler is back to taking ~20s per message... [05:34:37] 10SRE, 10Wikimedia-Mailing-lists: Mailman3 bounce runner is running very slowly - https://phabricator.wikimedia.org/T282348 (10Legoktm) I applied https://gitlab.com/mailman/mailman/-/merge_requests/869 on lists1001 - my previous patch just didn't cover all the code paths. [05:44:43] (03PS3) 10Giuseppe Lavagetto: docker::baseimages: add script to build debian-slim [puppet] - 10https://gerrit.wikimedia.org/r/693175 (https://phabricator.wikimedia.org/T281596) [05:49:50] (03CR) 10Giuseppe Lavagetto: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29653/console" [puppet] - 10https://gerrit.wikimedia.org/r/693175 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [05:52:19] (03CR) 10Giuseppe Lavagetto: [V: 03+1 C: 03+2] docker::baseimages: add script to build debian-slim [puppet] - 10https://gerrit.wikimedia.org/r/693175 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [06:37:09] (03PS1) 10Giuseppe Lavagetto: baseimages: add preferences file, bullseye [puppet] - 10https://gerrit.wikimedia.org/r/693643 [06:38:37] (03CR) 10Giuseppe Lavagetto: [C: 03+2] baseimages: add preferences file, bullseye [puppet] - 10https://gerrit.wikimedia.org/r/693643 (owner: 10Giuseppe Lavagetto) [06:58:12] (03PS1) 10Elukey: Add base kubeflow kfserving images and kube-rbac-proxy [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/693644 (https://phabricator.wikimedia.org/T272919) [06:59:11] (03PS2) 10Elukey: Add base kubeflow kfserving images and kube-rbac-proxy [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/693644 (https://phabricator.wikimedia.org/T272919) [07:01:38] 10SRE, 10ops-codfw, 10User-fgiunchedi: Degraded RAID on ms-be2038 - https://phabricator.wikimedia.org/T283401 (10fgiunchedi) a:03Papaul Indeed! Looks like BBU failure to me too, @papaul please replace; host can be powered off at any time [07:05:13] (03CR) 10Filippo Giunchedi: [C: 03+1] logstash: remove references to hosts logstash102[012] [puppet] - 10https://gerrit.wikimedia.org/r/693436 (https://phabricator.wikimedia.org/T281266) (owner: 10Herron) [07:06:49] (03CR) 10Filippo Giunchedi: [C: 03+1] Revert "Revert "prometheus: Migrate node_file_count cron to systemd timer"" [puppet] - 10https://gerrit.wikimedia.org/r/691317 (owner: 10Ladsgroup) [07:07:18] 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Replace SSH key for kzimmerman - https://phabricator.wikimedia.org/T283383 (10Marostegui) p:05Triage→03Medium a:03Marostegui [07:13:21] (03PS1) 10Marostegui: aliases.yaml.erb: Remove wikireplicas aliases [puppet] - 10https://gerrit.wikimedia.org/r/693666 (https://phabricator.wikimedia.org/T282662) [07:14:52] (03CR) 10Marostegui: [C: 03+2] aliases.yaml.erb: Remove wikireplicas aliases [puppet] - 10https://gerrit.wikimedia.org/r/693666 (https://phabricator.wikimedia.org/T282662) (owner: 10Marostegui) [07:16:04] 10SRE, 10netops, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10Marostegui) p:05Triage→03Medium [07:24:28] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Marostegui) [07:29:20] 10SRE, 10Wikimedia-Mailing-lists, 10Upstream: In Mailman3, users cannot change their display name from the web - https://phabricator.wikimedia.org/T283128 (10Legoktm) This is fixed in the next release apparently: https://gitlab.com/mailman/postorius/-/issues/451#note_579219934 [07:37:58] 10SRE, 10serviceops: dropped packets to echostore.svc.eqiad 8082/tcp - https://phabricator.wikimedia.org/T238789 (10Marostegui) 05Open→03Declined Closing as it is sort of tracked T238823 and there's been no activity here for more than a year [07:38:03] 10SRE, 10netops, 10User-jbond: Sporatic RST drops in the ulogd logs - https://phabricator.wikimedia.org/T238823 (10Marostegui) [07:40:47] 10SRE, 10Wikimedia-Mailing-lists, 10Upstream: Mailman mailing list messages should link to the web archive version in the footer - https://phabricator.wikimedia.org/T64949 (10Marostegui) 05Open→03Resolved Per the above message. [07:47:31] 10SRE, 10netops, 10User-jbond: Sporadic RST drops in the ulogd logs - https://phabricator.wikimedia.org/T238823 (10Nemo_bis) [07:48:32] 10SRE, 10SRE-Access-Requests: Requesting access to production analytics data and cluster for htriedman - https://phabricator.wikimedia.org/T283368 (10Marostegui) p:05Triage→03Medium [07:50:48] 10SRE, 10Discovery, 10Discovery-Search, 10Elasticsearch: Setup backups of elasticsearch indices - https://phabricator.wikimedia.org/T91404 (10Marostegui) Is this still needed? [07:52:31] 10SRE, 10Wikimedia-Mailing-lists: All WMF mailing lists should be publicly listed - https://phabricator.wikimedia.org/T124324 (10Marostegui) @Ladsgroup @Legoktm is this something that can be / has been done with the migration to mailman3? [07:57:37] (03PS1) 10Elukey: Add Jetstack's cert-manager base go images. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/693826 (https://phabricator.wikimedia.org/T280661) [08:16:00] 10SRE, 10serviceops: Publish wikimedia-bullseye base docker image - https://phabricator.wikimedia.org/T281596 (10Joe) a:05Legoktm→03Joe [08:18:59] 10SRE, 10Wikimedia-Hackathon-2021, 10Wikimedia-Mailing-lists, 10Upstream: Add OAuth login to mailman for accessing list memberships/archive viewing - https://phabricator.wikimedia.org/T249678 (10Legoktm) The logos appear to come from https://gitlab.com/mailman/django-mailman3/-/tree/master/django_mailman3/... [08:19:51] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Joe) [08:29:54] 10SRE: decrease negative cache TTL for lookups from MTA to google (WMF google apps) - https://phabricator.wikimedia.org/T84600 (10Marostegui) 05Open→03Resolved Going to close this as it's been more than 5 years. Please reopen if needed. [08:31:31] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Joe) [08:37:14] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Marostegui) [08:43:44] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Marostegui) [08:44:34] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10elukey) [08:48:00] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Joe) [08:49:46] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10diego) [08:51:02] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Joe) [08:52:58] 10SRE, 10Sustainability (Incident Followup): September 2019 DoS attacks [Public] - https://phabricator.wikimedia.org/T232224 (10Marostegui) 05Open→03Resolved I am going to close this as resolved. If someone feels this needs to stay open, please reopen. [08:53:54] 10SRE, 10MediaWiki-General: Elevated php7 latency during mw deploy - https://phabricator.wikimedia.org/T230934 (10Marostegui) 05Open→03Resolved Closing this as it's been almost 2 years, I don't think this can be investigated anymore. If someone feels this needs to stay open, please reopen! [08:56:01] 10SRE, 10Icinga, 10observability: Have a link to the alert in the icinga alert email - https://phabricator.wikimedia.org/T231274 (10Marostegui) @fgiunchedi do we really want to work on this? or should we close this and hope to get rid of icinga soon? :) [08:57:34] 10SRE, 10observability, 10CAS-SSO: thanos u/i gives errors if left idle for a few hours - https://phabricator.wikimedia.org/T268233 (10fgiunchedi) Ok so for thanos UI the symptom is still that an open page at https://thanos.wikimedia.org left idle for a little bit (I think an hour, judging by the values at t... [08:58:39] 10SRE, 10Icinga, 10observability: Have a link to the alert in the icinga alert email - https://phabricator.wikimedia.org/T231274 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Boldly resolving, emails from AM do have a link to the UI. [09:01:41] (03PS1) 10KartikMistry: Update cxserver to 2021-05-15-034540-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/693842 (https://phabricator.wikimedia.org/T276214) [09:03:54] (03PS4) 10Jbond: (WIP) create a logout.d profile for managing logout scripts [puppet] - 10https://gerrit.wikimedia.org/r/693149 [09:05:22] (03CR) 10jerkins-bot: [V: 04-1] (WIP) create a logout.d profile for managing logout scripts [puppet] - 10https://gerrit.wikimedia.org/r/693149 (owner: 10Jbond) [09:07:18] 10SRE: Investigate idle appservers in codfw - https://phabricator.wikimedia.org/T133093 (10Marostegui) 05Open→03Resolved Closing this as these hosts no longer exist. [09:10:24] 10SRE, 10DBA, 10Pybal, 10Sustainability: Create a backend check for pybal to monitor the MySQL protocol being up - https://phabricator.wikimedia.org/T165677 (10Marostegui) 05Open→03Declined I am going to close this - I don't think we are considering pybal for now or anytime soon. We can reopen if needed. [09:11:52] 10SRE, 10IRCecho: Make ircecho much better - https://phabricator.wikimedia.org/T95052 (10Marostegui) 05Open→03Invalid Closing this task as it doesn't provide much details on what actually needs to be done. If someone feels this needs to stay open, please reopen. [09:12:29] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (100xkaywong) [09:13:49] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10Marostegui) p:05Triage→03Medium [09:14:02] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10Marostegui) a:03Marostegui [09:14:27] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10Marostegui) I can see @0xkaywong signature on L3. [09:15:50] 10SRE, 10DBA, 10Pybal, 10Sustainability: Create a backend check for pybal to monitor the MySQL protocol being up - https://phabricator.wikimedia.org/T165677 (10jcrespo) Cool to me- I think this was a suggestion we got from traffic that I just put in written form. [09:22:53] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10Marostegui) [09:23:06] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10Marostegui) @KFrancis I cannot find Kay_Wong's name on the NDA sheet, can you confirm the NDA is signed? Thank you! [09:24:04] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10Marostegui) @diego @Ottomata I would need your comment with the specific approval for this. [09:39:22] 10SRE, 10SRE-tools: Cookbook to failover the Ganeti master - https://phabricator.wikimedia.org/T283320 (10Marostegui) p:05Triage→03Medium [09:39:47] 10SRE: Cookbooks for Ganeti maintenance tasks - https://phabricator.wikimedia.org/T283319 (10Marostegui) p:05Triage→03Medium [09:47:31] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10diego) hi @Marostegui , @0xkaywong should be signing her contract today. We will ping you here when the contract is signed. [09:48:58] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10Marostegui) 05Open→03Stalled Excellent thanks [09:51:00] (03PS1) 10Jcrespo: reimage: Avoid automatic reimage of backup[12]00[123] [puppet] - 10https://gerrit.wikimedia.org/r/693856 (https://phabricator.wikimedia.org/T282249) [09:51:26] (03PS2) 10Jcrespo: reimage: Avoid automatic reimage of backup[12]00[123] [puppet] - 10https://gerrit.wikimedia.org/r/693856 (https://phabricator.wikimedia.org/T282249) [09:52:14] (03PS3) 10Jcrespo: reimage: Avoid automatic reimage of backup[12]00[123] [puppet] - 10https://gerrit.wikimedia.org/r/693856 (https://phabricator.wikimedia.org/T282249) [09:57:14] (03PS1) 10Jbond: C:netbox: update SSL config to work with pki.discovery.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/693863 (https://phabricator.wikimedia.org/T283483) [09:59:23] (03CR) 10Jbond: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29655/console" [puppet] - 10https://gerrit.wikimedia.org/r/693863 (https://phabricator.wikimedia.org/T283483) (owner: 10Jbond) [10:02:48] (03CR) 10Jcrespo: [C: 03+2] reimage: Avoid automatic reimage of backup[12]00[123] [puppet] - 10https://gerrit.wikimedia.org/r/693856 (https://phabricator.wikimedia.org/T282249) (owner: 10Jcrespo) [10:06:35] (03CR) 10Volans: [C: 03+1] "LGTM but needs a change to the netbox-extras repo too to update the reports accordingly." [puppet] - 10https://gerrit.wikimedia.org/r/693863 (https://phabricator.wikimedia.org/T283483) (owner: 10Jbond) [10:08:20] (03PS1) 10Jbond: reports/puppetdb: use system certificate store for tls verification [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/693864 (https://phabricator.wikimedia.org/T283483) [10:09:23] 10SRE, 10Traffic: Let's Encrypt issuance chains update - https://phabricator.wikimedia.org/T283164 (10Marostegui) p:05Triage→03Medium [10:10:58] (03PS2) 10Jbond: reports/puppetdb: use system certificate store for tls verification [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/693864 (https://phabricator.wikimedia.org/T283483) [10:11:29] (03CR) 10Volans: [C: 03+1] "LGTM, thanks" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/693864 (https://phabricator.wikimedia.org/T283483) (owner: 10Jbond) [10:15:25] 10SRE, 10ops-codfw, 10DC-Ops, 10Patch-For-Review: (Need By: 2021-04-30) rack/setup/install backup200[4-7] - https://phabricator.wikimedia.org/T277323 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on cumin2001.codfw.wmnet for hosts: ` backup2004.codfw.wmnet ` The log can be found in `... [10:15:28] (03CR) 10Jbond: [V: 03+1 C: 03+2] C:netbox: update SSL config to work with pki.discovery.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/693863 (https://phabricator.wikimedia.org/T283483) (owner: 10Jbond) [10:16:31] (03CR) 10Volans: [C: 03+1] "LGTM if the key is verified, indentation nit inline." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/693636 (https://phabricator.wikimedia.org/T283383) (owner: 10Marostegui) [10:17:34] (03CR) 10Jbond: [C: 03+2] reports/puppetdb: use system certificate store for tls verification [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/693864 (https://phabricator.wikimedia.org/T283483) (owner: 10Jbond) [10:19:27] (03CR) 10Volans: "Sorry but I'm not able to review this because I'm not familiar with the current plans for pybal, including in which branch the current dev" [debs/pybal] - 10https://gerrit.wikimedia.org/r/644050 (owner: 10Ladsgroup) [10:21:18] 10SRE, 10serviceops: Publish wikimedia-bullseye base docker image - https://phabricator.wikimedia.org/T281596 (10Joe) p:05Triage→03Medium [10:21:42] (03PS1) 10Giuseppe Lavagetto: docker::baseimages: improvements to build-bare-slim [puppet] - 10https://gerrit.wikimedia.org/r/693865 (https://phabricator.wikimedia.org/T281596) [10:22:11] (03CR) 10jerkins-bot: [V: 04-1] docker::baseimages: improvements to build-bare-slim [puppet] - 10https://gerrit.wikimedia.org/r/693865 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [10:22:15] (03CR) 10Marostegui: "thanks - waiting for them to wake up so I can verify it :)" [puppet] - 10https://gerrit.wikimedia.org/r/693636 (https://phabricator.wikimedia.org/T283383) (owner: 10Marostegui) [10:26:00] 10SRE, 10Wikimedia-Hackathon-2021, 10Wikimedia-Mailing-lists, 10Upstream: Add OAuth login to mailman for accessing list memberships/archive viewing - https://phabricator.wikimedia.org/T249678 (10Tgr) >>! In T249678#7106765, @Legoktm wrote: > Is there a standard list of test edge cases to try out? Account... [10:26:44] 10SRE, 10Wikimedia-Hackathon-2021, 10Wikimedia-Mailing-lists, 10Upstream: Add OAuth login to mailman for accessing list memberships/archive viewing - https://phabricator.wikimedia.org/T249678 (10Tgr) Do we want some sort of formal security review for this, BTW? [10:27:37] 10SRE, 10Wikimedia-Mailing-lists: All WMF mailing lists should be publicly listed - https://phabricator.wikimedia.org/T124324 (10Ladsgroup) This is not done as part of migration, technically it's possible to do in both mm2 and mm3 (maybe easier in mm3 though) but the problem is the social part. It requires com... [10:27:44] (03Abandoned) 10Filippo Giunchedi: cassandra: check for heap dumps [puppet] - 10https://gerrit.wikimedia.org/r/227956 (https://phabricator.wikimedia.org/T106346) (owner: 10Filippo Giunchedi) [10:28:31] (03CR) 10Hnowlan: [C: 03+1] "This does have the side effect of lumping all GETs and all POSTs together as opposed to just those for mw API endpoints, but for now I am " [deployment-charts] - 10https://gerrit.wikimedia.org/r/693017 (owner: 10Ppchelko) [10:28:43] (03Abandoned) 10Filippo Giunchedi: swift: fix user/group for log files/dirs [puppet] - 10https://gerrit.wikimedia.org/r/465127 (https://phabricator.wikimedia.org/T205974) (owner: 10Filippo Giunchedi) [10:29:30] (03CR) 10Hnowlan: [V: 03+1 C: 03+2] maps: DB performance improvements [puppet] - 10https://gerrit.wikimedia.org/r/685743 (owner: 10MSantos) [10:30:05] jan_drewniak: Time to snap out of that daydream and deploy Wikimedia Portals Update. Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210524T1030). [10:31:02] 10SRE, 10ops-codfw, 10DC-Ops, 10Patch-For-Review: (Need By: 2021-04-30) rack/setup/install backup200[4-7] - https://phabricator.wikimedia.org/T277323 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['backup2004.codfw.wmnet'] ` Of which those **FAILED**: ` ['backup2004.codfw.wmnet'] ` [10:31:59] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] toolforge: update static proxy config for toolforge.org migration [puppet] - 10https://gerrit.wikimedia.org/r/693513 (owner: 10BryanDavis) [10:37:49] (03PS2) 10Giuseppe Lavagetto: docker::baseimages: improvements to build-bare-slim [puppet] - 10https://gerrit.wikimedia.org/r/693865 (https://phabricator.wikimedia.org/T281596) [10:42:25] (03PS1) 10Urbanecm: Deploy Growth features to newcomers on bgwiki, urwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693869 (https://phabricator.wikimedia.org/T280824) [10:48:35] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Marostegui) p:05Triage→03High Setting to high so we can move away from freenode asap [10:48:39] 10SRE, 10Analytics-Radar, 10LDAP-Access-Requests, 10SRE-Access-Requests: Account setup issues for jmixter-ctr - https://phabricator.wikimedia.org/T283250 (10Marostegui) p:05Triage→03Medium [11:00:04] Amir1, Lucas_WMDE, awight, and Urbanecm: May I have your attention please! European mid-day backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210524T1100) [11:00:04] Urbanecm: A patch you scheduled for European mid-day backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [11:00:44] (03CR) 10Urbanecm: [C: 03+2] Deploy Growth features to newcomers on bgwiki, urwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693869 (https://phabricator.wikimedia.org/T280824) (owner: 10Urbanecm) [11:01:20] (03PS2) 10Urbanecm: Remove wgGEMentorshipMigrationStage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693567 (https://phabricator.wikimedia.org/T279853) [11:01:28] (03CR) 10Urbanecm: [C: 03+2] Remove wgGEMentorshipMigrationStage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693567 (https://phabricator.wikimedia.org/T279853) (owner: 10Urbanecm) [11:01:35] (03Merged) 10jenkins-bot: Deploy Growth features to newcomers on bgwiki, urwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693869 (https://phabricator.wikimedia.org/T280824) (owner: 10Urbanecm) [11:03:50] (03Merged) 10jenkins-bot: Remove wgGEMentorshipMigrationStage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693567 (https://phabricator.wikimedia.org/T279853) (owner: 10Urbanecm) [11:05:54] (03CR) 10Urbanecm: [C: 03+2] Revert "Enable Wikidata description override on dewiki at beta" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693536 (https://phabricator.wikimedia.org/T279829) (owner: 10Zabe) [11:05:57] (03PS3) 10Urbanecm: Revert "Enable Wikidata description override on dewiki at beta" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693536 (https://phabricator.wikimedia.org/T279829) (owner: 10Zabe) [11:06:00] (03PS1) 10Hnowlan: profile::envoy: Add argument for building the envoy-future package [puppet] - 10https://gerrit.wikimedia.org/r/693870 [11:06:02] (03CR) 10Urbanecm: [C: 03+2] Revert "Enable Wikidata description override on dewiki at beta" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693536 (https://phabricator.wikimedia.org/T279829) (owner: 10Zabe) [11:07:45] (03PS1) 10Giuseppe Lavagetto: mediawiki: add egress policies to databases [deployment-charts] - 10https://gerrit.wikimedia.org/r/693871 [11:07:47] (03Merged) 10jenkins-bot: Revert "Enable Wikidata description override on dewiki at beta" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693536 (https://phabricator.wikimedia.org/T279829) (owner: 10Zabe) [11:09:38] (03CR) 10jerkins-bot: [V: 04-1] mediawiki: add egress policies to databases [deployment-charts] - 10https://gerrit.wikimedia.org/r/693871 (owner: 10Giuseppe Lavagetto) [11:25:28] (03PS1) 10Jbond: configmaster: add puppetdb-api to disc_desired_state.py [puppet] - 10https://gerrit.wikimedia.org/r/693872 [11:30:38] (03CR) 10Jbond: [C: 03+2] configmaster: add puppetdb-api to disc_desired_state.py [puppet] - 10https://gerrit.wikimedia.org/r/693872 (owner: 10Jbond) [11:39:20] (03PS1) 10Jbond: docekr-report: add python3-build-jessie to filter list [puppet] - 10https://gerrit.wikimedia.org/r/693874 [11:44:09] 10SRE, 10Wikimedia-Mailing-lists, 10Upstream: Mailman 3: Months with no emails are still listed in the archive - https://phabricator.wikimedia.org/T282341 (10Marostegui) p:05Triage→03Medium [11:48:27] (03PS1) 10Effie Mouzeli: (WIP) mwdebug: add helfile configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/693875 [11:52:23] (03PS1) 10Hnowlan: osm: fix import issues in imposm-initia-import [puppet] - 10https://gerrit.wikimedia.org/r/693876 [11:54:23] 10SRE, 10Wikimedia-Mailing-lists: The Great Clean Up of Mailman2 - https://phabricator.wikimedia.org/T282303 (10Ladsgroup) a:03jcrespo > I'll ping you again once we're ready for that, thanks! I think we are ready. Assigning as requested in IRC. [11:55:29] 10SRE, 10Data-Persistence-Backup, 10Wikimedia-Mailing-lists: The Great Clean Up of Mailman2 - https://phabricator.wikimedia.org/T282303 (10jcrespo) [11:56:49] (03PS1) 10Urbanecm: ruwiki: Disable Flow/StructuredDiscussion [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693878 (https://phabricator.wikimedia.org/T282132) [11:57:22] (03CR) 10Urbanecm: [C: 03+2] ruwiki: Disable Flow/StructuredDiscussion [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693878 (https://phabricator.wikimedia.org/T282132) (owner: 10Urbanecm) [11:58:08] (03Merged) 10jenkins-bot: ruwiki: Disable Flow/StructuredDiscussion [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693878 (https://phabricator.wikimedia.org/T282132) (owner: 10Urbanecm) [11:58:17] 10SRE, 10CAS-SSO: Cookbook for centralised logouts and session status queries - https://phabricator.wikimedia.org/T283242 (10Marostegui) p:05Triage→03Medium [11:59:46] (03CR) 10MSantos: [C: 03+1] osm: fix import issues in imposm-initia-import [puppet] - 10https://gerrit.wikimedia.org/r/693876 (owner: 10Hnowlan) [11:59:55] 10SRE, 10Wikimedia-Mailing-lists: Cannot manually add users to wmfreqs list due to regex banning any addresses - https://phabricator.wikimedia.org/T283103 (10Marostegui) p:05Triage→03Medium >>! In T283103#7096798, @Legoktm wrote: >>>! In T283103#7096658, @Legoktm wrote: >> Also it's a mistake that (03CR) 10Arturo Borrero Gonzalez: [C: 04-1] "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/691154 (owner: 10Arturo Borrero Gonzalez) [12:10:52] (03CR) 10Jbond: [C: 03+2] docekr-report: add python3-build-jessie to filter list [puppet] - 10https://gerrit.wikimedia.org/r/693874 (owner: 10Jbond) [12:13:16] (03CR) 10MSantos: (WIP) Add tokens and users for maps-vector-server (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692669 (owner: 10Effie Mouzeli) [12:13:24] 10SRE, 10User-MoritzMuehlenhoff: Sensible updates of java.security properties - https://phabricator.wikimedia.org/T282545 (10Marostegui) p:05Triage→03Medium [12:39:34] 10SRE: x509-bundle as used by envoy::tlsproxy fails on single certificate file - https://phabricator.wikimedia.org/T283001 (10Marostegui) p:05Triage→03Medium [12:43:42] (03CR) 10Effie Mouzeli: (WIP) Add tokens and users for maps-vector-server (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692669 (owner: 10Effie Mouzeli) [12:49:34] 10SRE, 10Wikimedia-Mailing-lists, 10serviceops: Set up memcached for mailman3 - https://phabricator.wikimedia.org/T282931 (10Marostegui) p:05Triage→03Medium [12:59:39] (03PS1) 10Esanders: Enable DiscussionTools on wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693897 (https://phabricator.wikimedia.org/T283119) [13:00:27] (03CR) 10Esanders: Enable DiscussionTools on wikitech (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693897 (https://phabricator.wikimedia.org/T283119) (owner: 10Esanders) [13:03:09] (03CR) 10Ppchelko: "See https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/router_filter#config-http-filters-router-vcluster-stats fo" [deployment-charts] - 10https://gerrit.wikimedia.org/r/693017 (owner: 10Ppchelko) [13:12:08] 10SRE, 10Wikimedia-Mailing-lists: Find list owners for lists without them - https://phabricator.wikimedia.org/T281779 (10Ladsgroup) On mm3: ` MariaDB [mailman3]> select list_id from member where list_id not in (select list_id from member where role = 2 group by list_id) group by list_id limit 50; +------------... [13:13:53] (03PS2) 10Effie Mouzeli: WIP: add notls support for external addresses to memcached [puppet] - 10https://gerrit.wikimedia.org/r/693474 (https://phabricator.wikimedia.org/T271967) [13:17:40] (03PS2) 10Effie Mouzeli: (WIP) mwdebug: add helmfile configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/693875 [13:24:08] (03PS4) 10Herron: add kafka-main[12]00[45] to existing kafka-main egress rules and broker lists [deployment-charts] - 10https://gerrit.wikimedia.org/r/683706 (https://phabricator.wikimedia.org/T225005) [13:29:32] (03CR) 10Herron: [C: 03+2] add kafka-main[12]00[45] to existing kafka-main egress rules and broker lists [deployment-charts] - 10https://gerrit.wikimedia.org/r/683706 (https://phabricator.wikimedia.org/T225005) (owner: 10Herron) [13:31:43] (03Merged) 10jenkins-bot: add kafka-main[12]00[45] to existing kafka-main egress rules and broker lists [deployment-charts] - 10https://gerrit.wikimedia.org/r/683706 (https://phabricator.wikimedia.org/T225005) (owner: 10Herron) [13:32:33] 10SRE, 10Wikimedia-Mailing-lists: Cannot manually add users to wmfreqs list due to regex banning any addresses - https://phabricator.wikimedia.org/T283103 (10Olauro) Thank you @Legoktm. Confirming that Carie successfully followed your steps and was able to add me as an admin. This can be closed now. We appr... [13:32:51] (03CR) 10MSantos: (WIP) Add tokens and users for maps-vector-server (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692669 (owner: 10Effie Mouzeli) [13:35:43] 10SRE, 10Wikimedia-Mailing-lists: Cannot manually add users to wmfreqs list due to regex banning any addresses - https://phabricator.wikimedia.org/T283103 (10Ladsgroup) 05Open→03Resolved a:03Legoktm [13:37:34] (03CR) 10Elukey: [C: 03+1] docker::baseimages: improvements to build-bare-slim [puppet] - 10https://gerrit.wikimedia.org/r/693865 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [13:38:03] 10SRE, 10Wikimedia-Mailing-lists, 10User-Ladsgroup: " %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s" in recent Wikitech-l posts - https://phabricator.wikimedia.org/T282762 (10Ladsgroup) 05Open→03Resolved Can't see it again. [13:42:42] 10SRE, 10Wikimedia-Mailing-lists, 10Wikimedia-Small-Projects-User-Group, 10User-Ladsgroup: Mailing list request for Wikimedia Small Projects User Group - https://phabricator.wikimedia.org/T283310 (10Ladsgroup) 05Open→03Resolved The list has been created: https://lists.wikimedia.org/postorius/lists/wiki... [13:42:55] (03PS1) 10Zabe: Disable Education Program namespaces in hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693903 (https://phabricator.wikimedia.org/T217137) [13:44:16] 10SRE, 10decommission-hardware, 10observability: decommission mwlog2001 - https://phabricator.wikimedia.org/T282576 (10Papaul) [13:47:58] (03CR) 10Giuseppe Lavagetto: [C: 03+2] docker::baseimages: improvements to build-bare-slim [puppet] - 10https://gerrit.wikimedia.org/r/693865 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [13:48:22] (03PS4) 10Herron: kafka-main: deploy kafka::main role to kafka-main[12]00[45] [puppet] - 10https://gerrit.wikimedia.org/r/683044 (https://phabricator.wikimedia.org/T225005) [13:48:24] 10SRE, 10Traffic, 10netops: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) [13:48:27] 10SRE, 10Traffic, 10Patch-For-Review: Offer Wikidough as an anycasted service - https://phabricator.wikimedia.org/T283027 (10ssingh) [13:51:32] (03CR) 10Herron: [C: 03+2] kafka-main: deploy kafka::main role to kafka-main[12]00[45] [puppet] - 10https://gerrit.wikimedia.org/r/683044 (https://phabricator.wikimedia.org/T225005) (owner: 10Herron) [13:52:30] (03CR) 10Jgiannelos: (WIP) Add tokens and users for maps-vector-server (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692669 (owner: 10Effie Mouzeli) [14:16:27] (03PS1) 10Herron: update puppet defaults and docs to libera.chat [puppet] - 10https://gerrit.wikimedia.org/r/693906 (https://phabricator.wikimedia.org/T283213) [14:17:25] 10SRE, 10ops-codfw, 10DC-Ops, 10Patch-For-Review: (Need By: 2021-04-30) rack/setup/install backup200[4-7] - https://phabricator.wikimedia.org/T277323 (10jcrespo) a:05jcrespo→03Papaul These hosts have the same issue that backup2003 had: T274185#6883969 we need to make manually the first SSD bootable for... [14:34:42] (03CR) 10Herron: [C: 03+2] logstash: remove references to hosts logstash102[012] [puppet] - 10https://gerrit.wikimedia.org/r/693436 (https://phabricator.wikimedia.org/T281266) (owner: 10Herron) [14:38:45] (03CR) 10Effie Mouzeli: (WIP) Add tokens and users for maps-vector-server (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692669 (owner: 10Effie Mouzeli) [14:39:52] (03PS3) 10Effie Mouzeli: (WIP) Add tokens and users for tegola-vector-tiles [puppet] - 10https://gerrit.wikimedia.org/r/692669 [14:40:08] (03CR) 10jerkins-bot: [V: 04-1] (WIP) Add tokens and users for tegola-vector-tiles [puppet] - 10https://gerrit.wikimedia.org/r/692669 (owner: 10Effie Mouzeli) [14:40:36] (03PS4) 10Effie Mouzeli: (WIP) Add tokens and users for tegola-vector-tiles [puppet] - 10https://gerrit.wikimedia.org/r/692669 [14:41:08] (03PS1) 10Hnowlan: postgresql::postgis: use latest packages on buster [puppet] - 10https://gerrit.wikimedia.org/r/693907 [14:41:10] 10SRE, 10Traffic, 10netops: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10A189605) This is still very much an issue unfortunately. I have connected my laptop into a layer 2 switch connected to our ISPs CPE and attempted to access en.wikipedia.org. Still s... [14:42:06] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10Ottomata) Approved! Please also make sure that Kay is in the `nda` LDAP group. Thank you! [14:42:15] (03CR) 10Clarakosi: [C: 03+1] "This is pretty cool! I'd be curious to know how this performs. The docs does mention that virtual clusters should not be applied to every " [deployment-charts] - 10https://gerrit.wikimedia.org/r/693017 (owner: 10Ppchelko) [14:43:02] (03PS1) 10Ssingh: test_tls: add a test for checking the certificates [software/knead-wikidough] - 10https://gerrit.wikimedia.org/r/693908 [14:44:11] (03CR) 10Ssingh: [C: 03+2] test_tls: add a test for checking the certificates [software/knead-wikidough] - 10https://gerrit.wikimedia.org/r/693908 (owner: 10Ssingh) [14:44:31] (03CR) 10Hnowlan: [V: 03+1] "PCC SUCCESS (DIFF 1 NOOP 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29658/console" [puppet] - 10https://gerrit.wikimedia.org/r/693907 (owner: 10Hnowlan) [14:46:03] 10SRE, 10observability, 10Patch-For-Review: Decommission old ELK5 Logstash cluster - https://phabricator.wikimedia.org/T281266 (10herron) [14:46:04] (03CR) 10Alexandros Kosiaris: (WIP) Add tokens and users for tegola-vector-tiles (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692669 (owner: 10Effie Mouzeli) [14:48:04] 10SRE, 10observability, 10Patch-For-Review: Decommission old ELK5 Logstash cluster - https://phabricator.wikimedia.org/T281266 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by herron@cumin1001 for hosts: `logstash1020.eqiad.wmnet` - logstash1020.eqiad.wmnet (**PASS**) - Downtimed host on... [14:48:33] (03PS1) 10Filippo Giunchedi: udev: Bullseye compatibility for udevadm [puppet] - 10https://gerrit.wikimedia.org/r/693909 [14:53:28] 10SRE, 10ops-codfw, 10DBA: Degraded RAID on db2107 - https://phabricator.wikimedia.org/T282072 (10Papaul) ` Create Dispatch: Success You have successfully submitted request SR1060544173. [14:55:34] (03CR) 10Clarakosi: [C: 03+1] Allow Api-User-Agent for API request CORS [deployment-charts] - 10https://gerrit.wikimedia.org/r/693506 (https://phabricator.wikimedia.org/T268791) (owner: 10Ppchelko) [14:57:18] (03PS2) 10Hnowlan: postgresql::postgis: use latest packages on buster [puppet] - 10https://gerrit.wikimedia.org/r/693907 (https://phabricator.wikimedia.org/T277064) [15:05:10] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10RobH) [15:05:11] (03PS2) 10Mforns: reportupdater: Rsync logs to HDFS [puppet] - 10https://gerrit.wikimedia.org/r/692909 (https://phabricator.wikimedia.org/T274880) [15:07:04] 10SRE, 10decommission-hardware, 10observability: decommission mwlog2001 - https://phabricator.wikimedia.org/T282576 (10Papaul) [15:07:21] 10SRE, 10observability, 10Patch-For-Review: Migrate mwlog/udp2log servers to Buster - https://phabricator.wikimedia.org/T224565 (10Papaul) [15:07:45] 10SRE, 10decommission-hardware, 10observability: decommission mwlog2001 - https://phabricator.wikimedia.org/T282576 (10Papaul) 05Open→03Resolved This is complete [15:17:25] (03CR) 10Hnowlan: [C: 03+1] Allow Api-User-Agent for API request CORS [deployment-charts] - 10https://gerrit.wikimedia.org/r/693506 (https://phabricator.wikimedia.org/T268791) (owner: 10Ppchelko) [15:20:58] (03CR) 10Ppchelko: [C: 03+2] Allow Api-User-Agent for API request CORS [deployment-charts] - 10https://gerrit.wikimedia.org/r/693506 (https://phabricator.wikimedia.org/T268791) (owner: 10Ppchelko) [15:23:16] (03Merged) 10jenkins-bot: Allow Api-User-Agent for API request CORS [deployment-charts] - 10https://gerrit.wikimedia.org/r/693506 (https://phabricator.wikimedia.org/T268791) (owner: 10Ppchelko) [15:27:15] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10RobH) [15:28:19] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10RobH) [15:30:25] 10SRE, 10ops-codfw, 10netops: Multiple host down alerts from rack C2 - https://phabricator.wikimedia.org/T279457 (10Papaul) 05Open→03Resolved Received the new switch. Resolving this task {F34466158} [15:37:49] (03CR) 10Dzahn: [C: 03+1] index: Point IRC presence to Libera [software/klaxon] - 10https://gerrit.wikimedia.org/r/693540 (owner: 10MarcoAurelio) [15:37:51] (03PS1) 10Ppchelko: api-gateway: fix allow_headers spec [deployment-charts] - 10https://gerrit.wikimedia.org/r/693911 [15:39:17] (03CR) 10Hnowlan: [C: 03+1] api-gateway: fix allow_headers spec [deployment-charts] - 10https://gerrit.wikimedia.org/r/693911 (owner: 10Ppchelko) [15:39:32] (03CR) 10Ppchelko: [C: 03+2] api-gateway: fix allow_headers spec [deployment-charts] - 10https://gerrit.wikimedia.org/r/693911 (owner: 10Ppchelko) [15:41:41] (03Merged) 10jenkins-bot: api-gateway: fix allow_headers spec [deployment-charts] - 10https://gerrit.wikimedia.org/r/693911 (owner: 10Ppchelko) [15:45:15] (03PS1) 10Jdlrobson: Introduce the vector-body class [skins/Vector] (wmf/1.37.0-wmf.6) - 10https://gerrit.wikimedia.org/r/693653 (https://phabricator.wikimedia.org/T283206) [15:54:06] 10SRE, 10netops, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10jbond) > Is simple query success + rtt enough? If so, the built-in functionality of atlas_exporter is sufficient. Definitely for the first inst... [15:56:36] 10SRE, 10Traffic, 10netops: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) Hey Sukhbir, Can you confirm "doh2001" and "doh2002" are the VMs which will Anycast the Wikidough IP? I'm not sure using a static route is the best idea. It's bette... [15:57:31] 10SRE, 10ops-eqiad, 10DC-Ops: Relabel db1125 to be dbstore1006 - https://phabricator.wikimedia.org/T283300 (10wiki_willy) a:03Cmjohnson [15:58:06] 10SRE, 10Traffic, 10netops: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10Marostegui) p:05Triage→03Medium [16:02:44] 10SRE, 10ops-codfw, 10DC-Ops, 10Patch-For-Review: (Need By: 2021-04-30) rack/setup/install backup200[4-7] - https://phabricator.wikimedia.org/T277323 (10Papaul) @jcrespo ack [16:07:30] (03PS3) 10Effie Mouzeli: (WIP) mwdebug: add helfile configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/693875 [16:12:20] 10SRE, 10Privacy Engineering, 10Traffic: Create and document Wikidough's privacy policy - https://phabricator.wikimedia.org/T275409 (10JFishback_WMF) [16:13:10] 10SRE, 10ops-codfw, 10DC-Ops, 10Patch-For-Review: (Need By: 2021-04-30) rack/setup/install backup200[4-7] - https://phabricator.wikimedia.org/T277323 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jynus on cumin2001.codfw.wmnet for hosts: ` backup2004.codfw.wmnet ` The log can be found in `... [16:14:01] (03PS10) 10Ottomata: Initial debianization and 2.1.0-py3.7-1 release [debs/airflow] (debian) - 10https://gerrit.wikimedia.org/r/693222 (https://phabricator.wikimedia.org/T277012) [16:15:41] 10SRE, 10Traffic, 10netops: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) >>! In T283503#7107926, @cmooney wrote: > Hey Sukhbir, > > Can you confirm "doh2001" and "doh2002" are the VMs which will Anycast the Wikidough IP? Sorry, I should hav... [16:16:05] (03PS1) 10Herron: logstash101[012]: enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/693914 [16:20:55] 10SRE, 10ops-codfw, 10DC-Ops, 10Patch-For-Review: (Need By: 2021-04-30) rack/setup/install backup200[4-7] - https://phabricator.wikimedia.org/T277323 (10jcrespo) I've documented the requirement at: https://wikitech.wikimedia.org/wiki/Raid_setup but please let me know if there is a better place/preferred lo... [16:24:39] 10SRE, 10ops-codfw, 10DC-Ops, 10fundraising-tech-ops: (Need By: TBD) rack/setup/install fran2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T282056 (10Papaul) [16:25:56] 10SRE, 10Discovery, 10Discovery-Search, 10Elasticsearch: Setup backups of elasticsearch indices - https://phabricator.wikimedia.org/T91404 (10EBernhardson) Debatable. We have dumps as mentioned above, but dumps != backups. In the face of data loss (there is no primary data, nothing is really "lost", inste... [16:26:02] 10SRE, 10ops-codfw, 10DC-Ops, 10Patch-For-Review: (Need By: 2021-04-30) rack/setup/install backup200[4-7] - https://phabricator.wikimedia.org/T277323 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['backup2004.codfw.wmnet'] ` and were **ALL** successful. [16:31:50] 10SRE, 10Discovery, 10Discovery-Search, 10Elasticsearch: Setup backups of elasticsearch indices - https://phabricator.wikimedia.org/T91404 (10jcrespo) @EBernhardson one of the initiatives we started from #data-persistence-backup (and for its own reasons, I believe also security and/or legal) is documenting... [16:32:07] (03PS2) 10Hnowlan: osm: fix import issues in imposm-initial-import, run imposm as osmupdater [puppet] - 10https://gerrit.wikimedia.org/r/693876 [16:40:40] 10SRE, 10Maps, 10Product-Infrastructure-Team-Backlog, 10Services, and 2 others: New Service Request maps-vector-server - https://phabricator.wikimedia.org/T274390 (10jijiki) [16:41:06] 10SRE, 10Maps, 10Product-Infrastructure-Team-Backlog, 10Services, and 2 others: New Service Request tegola-vector-tiles - https://phabricator.wikimedia.org/T274390 (10jijiki) [16:44:01] (03PS1) 10Effie Mouzeli: Rename maps-vector-server to tegola-vector-tiles [deployment-charts] - 10https://gerrit.wikimedia.org/r/693917 (https://phabricator.wikimedia.org/T283159) [16:45:13] (03PS2) 10Effie Mouzeli: Rename maps-vector-server to tegola-vector-tiles [deployment-charts] - 10https://gerrit.wikimedia.org/r/693917 (https://phabricator.wikimedia.org/T283159) [16:46:19] (03PS5) 10Effie Mouzeli: Add tokens and users for tegola-vector-tiles [puppet] - 10https://gerrit.wikimedia.org/r/692669 (https://phabricator.wikimedia.org/T283159) [16:47:13] (03CR) 10jerkins-bot: [V: 04-1] Rename maps-vector-server to tegola-vector-tiles [deployment-charts] - 10https://gerrit.wikimedia.org/r/693917 (https://phabricator.wikimedia.org/T283159) (owner: 10Effie Mouzeli) [16:48:07] 10SRE, 10Traffic, 10netops: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) I spoke with Arzhel about it and he's pointed me in the right direction. I think your change is good to merge, however we don't want to have a huge time gap between m... [16:48:10] (03PS1) 10Giuseppe Lavagetto: docker::baseimages: remove bootstrap-vz, build bullseye [puppet] - 10https://gerrit.wikimedia.org/r/693918 (https://phabricator.wikimedia.org/T281984) [16:50:20] 10SRE, 10Data-Persistence-Backup, 10Goal, 10Patch-For-Review: Puppetize media backups infrastructure - https://phabricator.wikimedia.org/T276442 (10jcrespo) First pass of puppetization complete, we now have at least one server setup per type with a very basic class, now it is a question of improving the pr... [16:50:31] 10Puppet, 10Release-Engineering-Team, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) [16:51:49] 10Puppet, 10GitLab (Initialization), 10Patch-For-Review, 10Release-Engineering-Team (Radar), 10User-brennen: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) [16:57:32] 10SRE, 10ops-codfw, 10DC-Ops, 10fundraising-tech-ops: (Need By: TBD) rack/setup/install fran2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T282056 (10Papaul) ` Interface Admin Link Description ge-0/0/4 up up fran2001:eth0 ge-1/0/4 up up fran2001:eth1 ` [16:58:19] (03PS1) 10Volans: Netbox: fix path to new CA [puppet] - 10https://gerrit.wikimedia.org/r/693921 [16:58:58] 10SRE, 10ops-codfw, 10DC-Ops, 10fundraising-tech-ops: (Need By: TBD) rack/setup/install fran2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T282056 (10Papaul) [16:59:13] (03PS1) 10Ppchelko: api-gateway: move envoy CORS filter higher in the chain [deployment-charts] - 10https://gerrit.wikimedia.org/r/693922 [16:59:15] (03PS1) 10Cathal Mooney: Added Wikidough VMs to BGP Anycast codfw [homer/public] - 10https://gerrit.wikimedia.org/r/693923 [16:59:40] (03PS1) 10Volans: Revert "reports/puppetdb: use system certificate store for tls verification" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/693654 [16:59:50] (03CR) 10jerkins-bot: [V: 04-1] Added Wikidough VMs to BGP Anycast codfw [homer/public] - 10https://gerrit.wikimedia.org/r/693923 (owner: 10Cathal Mooney) [17:00:00] 10SRE, 10Traffic, 10netops: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10ssingh) >>! In T283503#7108144, @cmooney wrote: > I spoke with Arzhel about it and he's pointed me in the right direction. > > I think your change is good to merge, however we... [17:00:05] ryankemper: Time to snap out of that daydream and deploy Wikidata Query Service weekly deploy. Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210524T1700). [17:01:25] (03PS5) 10Ppchelko: api-gateway: use virtual clusters to split r ans w traffic [deployment-charts] - 10https://gerrit.wikimedia.org/r/693017 [17:03:39] (03PS6) 10Ppchelko: api-gateway: use virtual clusters to split r ans w traffic [deployment-charts] - 10https://gerrit.wikimedia.org/r/693017 [17:03:56] (03PS1) 10Effie Mouzeli: Add tokens and users for tegola-vector-tiles [labs/private] - 10https://gerrit.wikimedia.org/r/693924 (https://phabricator.wikimedia.org/T283159) [17:04:38] (03CR) 10Volans: [C: 03+2] "Compiler looks happy, merging. I'll address any comment tomorrow if needed." [puppet] - 10https://gerrit.wikimedia.org/r/693921 (owner: 10Volans) [17:05:02] (03PS2) 10Effie Mouzeli: Add tokens and users for tegola-vector-tiles [labs/private] - 10https://gerrit.wikimedia.org/r/693924 (https://phabricator.wikimedia.org/T283159) [17:05:31] (03CR) 10Volans: "> Patch Set 1: Code-Review+2" [puppet] - 10https://gerrit.wikimedia.org/r/693921 (owner: 10Volans) [17:05:45] (03CR) 10Volans: [C: 03+2] Revert "reports/puppetdb: use system certificate store for tls verification" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/693654 (owner: 10Volans) [17:06:38] (03Merged) 10jenkins-bot: Revert "reports/puppetdb: use system certificate store for tls verification" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/693654 (owner: 10Volans) [17:06:40] (03CR) 10Ppchelko: [C: 03+2] "PS 5 and 6 are manual rebase, +2 per Clara and Hugh approvals" [deployment-charts] - 10https://gerrit.wikimedia.org/r/693017 (owner: 10Ppchelko) [17:08:50] (03Merged) 10jenkins-bot: api-gateway: use virtual clusters to split r ans w traffic [deployment-charts] - 10https://gerrit.wikimedia.org/r/693017 (owner: 10Ppchelko) [17:10:30] (03PS1) 10Volans: wmf-auto-reimage: update CA validation [puppet] - 10https://gerrit.wikimedia.org/r/693926 [17:12:29] (03PS2) 10Cathal Mooney: Added Wikidough VMs to BGP Anycast codfw [homer/public] - 10https://gerrit.wikimedia.org/r/693923 [17:14:08] (03CR) 10Hnowlan: [C: 03+2] api-gateway: move envoy CORS filter higher in the chain [deployment-charts] - 10https://gerrit.wikimedia.org/r/693922 (owner: 10Ppchelko) [17:14:16] (03CR) 10Hnowlan: [C: 03+1] api-gateway: move envoy CORS filter higher in the chain [deployment-charts] - 10https://gerrit.wikimedia.org/r/693922 (owner: 10Ppchelko) [17:15:14] 10SRE, 10Traffic, 10netops: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) Change created: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/693923 [17:15:58] (03PS1) 10Effie Mouzeli: Add a namespace for tegola-vector-tiles service [deployment-charts] - 10https://gerrit.wikimedia.org/r/693927 (https://phabricator.wikimedia.org/T283159) [17:16:20] 10ops-eqiad, 10DC-Ops: Netbox Errors - https://phabricator.wikimedia.org/T283518 (10wiki_willy) [17:17:48] (03CR) 10jerkins-bot: [V: 04-1] Add a namespace for tegola-vector-tiles service [deployment-charts] - 10https://gerrit.wikimedia.org/r/693927 (https://phabricator.wikimedia.org/T283159) (owner: 10Effie Mouzeli) [17:25:33] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10KFrancis) @Marostegui Hello! Is Kay a contractor with an agreement with the WMF? [17:30:09] 10SRE, 10Data-Persistence-Backup, 10Wikimedia-Mailing-lists: The Great Clean Up of Mailman2 - https://phabricator.wikimedia.org/T282303 (10Aklapper) [17:31:46] 10SRE, 10Wikimedia-Mailing-lists, 10Upstream: In Mailman3, users cannot change their display name from the web - https://phabricator.wikimedia.org/T283128 (10Aklapper) @Marostegui: There's not much point in removing #SRE because there are Herald/Maintenance_bot rules in place. If you want to change them plea... [17:45:17] (03PS1) 10Ppchelko: api-gateway: exclude /heathz from r and w virtual clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/693930 [17:45:24] 10SRE, 10Traffic, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) 05Open→03Resolved [17:45:48] 10SRE, 10ops-eqiad, 10DC-Ops: Netbox Errors - https://phabricator.wikimedia.org/T283518 (10wiki_willy) Fixed shelf location, procurement task #, and procurement date for Apple Airport Express Wifi [17:48:37] (03CR) 10Ahmon Dancy: [C: 03+2] "Looks reasonable to me." [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/693258 (owner: 10Hashar) [17:48:48] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] Remove Gerrit Managers privileges [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/693258 (owner: 10Hashar) [17:51:58] (03CR) 10Ahmon Dancy: [C: 03+1] Download upstream war with Maven [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/693450 (owner: 10Hashar) [17:58:16] (03CR) 10CDanis: [C: 03+2] "Thanks, MarcoAurelio!" [software/klaxon] - 10https://gerrit.wikimedia.org/r/693540 (owner: 10MarcoAurelio) [17:59:29] 10SRE, 10netops, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10cmooney) Was looking at the NSID stuff based on John's suggestion, indeed it is returned by our authdns at the moment: ` root@nyc2:~# for i in {... [18:00:05] RoanKattouw, Niharika, and Urbanecm: Time to snap out of that daydream and deploy Morning backport window. Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210524T1800). [18:00:05] nray: A patch you scheduled for Morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [18:00:16] (03Merged) 10jenkins-bot: index: Point IRC presence to Libera [software/klaxon] - 10https://gerrit.wikimedia.org/r/693540 (owner: 10MarcoAurelio) [18:00:52] (03CR) 10Urbanecm: [C: 03+2] Introduce the vector-body class [skins/Vector] (wmf/1.37.0-wmf.6) - 10https://gerrit.wikimedia.org/r/693653 (https://phabricator.wikimedia.org/T283206) (owner: 10Jdlrobson) [18:09:09] 10SRE, 10netops, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) Thanks very much for all the helpful comments! * I'll definitely enable the NSID bit, it will be very useful especially for mapping the... [18:11:38] 10SRE, 10SRE-Access-Requests: Requesting access to Analytic Cluster for Research Contractor (Kay Wong) - https://phabricator.wikimedia.org/T283486 (10diego) @KFrancis , yes Kay is a contractor. [18:11:57] (03PS1) 10Mforns: Refine VirtualPageView schema using eventlogging_legacy job [puppet] - 10https://gerrit.wikimedia.org/r/693931 (https://phabricator.wikimedia.org/T238138) [18:17:27] (03CR) 10Ottomata: [C: 03+2] Refine VirtualPageView schema using eventlogging_legacy job [puppet] - 10https://gerrit.wikimedia.org/r/693931 (https://phabricator.wikimedia.org/T238138) (owner: 10Mforns) [18:20:03] (03Merged) 10jenkins-bot: Introduce the vector-body class [skins/Vector] (wmf/1.37.0-wmf.6) - 10https://gerrit.wikimedia.org/r/693653 (https://phabricator.wikimedia.org/T283206) (owner: 10Jdlrobson) [18:20:41] (03PS1) 10Ssingh: site: update and consolidate Wikidough hosts [puppet] - 10https://gerrit.wikimedia.org/r/693933 [18:22:07] (03CR) 10Ssingh: "Do not merge! Blocks on https://gerrit.wikimedia.org/r/c/operations/puppet/+/692368." [puppet] - 10https://gerrit.wikimedia.org/r/693933 (owner: 10Ssingh) [18:22:23] (03CR) 10Clarakosi: [C: 03+1] api-gateway: move envoy CORS filter higher in the chain [deployment-charts] - 10https://gerrit.wikimedia.org/r/693922 (owner: 10Ppchelko) [18:23:44] (03CR) 10Urbanecm: [C: 03+2] Disable Education Program namespaces in hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693903 (https://phabricator.wikimedia.org/T217137) (owner: 10Zabe) [18:24:31] (03Merged) 10jenkins-bot: Disable Education Program namespaces in hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693903 (https://phabricator.wikimedia.org/T217137) (owner: 10Zabe) [18:36:17] 10SRE, 10Wikimedia-Mailing-lists: "X-BeenThere" header not included anymore in daily-article-l mailing list emails (superseded by "List-Id"?) - https://phabricator.wikimedia.org/T283520 (10Aklapper) [18:54:53] (03PS1) 10Andrew Bogott: Add an additional public key for Andrew Bogott [puppet] - 10https://gerrit.wikimedia.org/r/693938 [18:57:17] 10Puppet, 10GitLab (Initialization), 10Patch-For-Review, 10Release-Engineering-Team (Radar), 10User-brennen: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) So I think we're at a point where S&F feels pretty confident with the state of the Ansible playbook, and w... [19:01:48] (03PS2) 10Andrew Bogott: Add an additional public key for Andrew Bogott [puppet] - 10https://gerrit.wikimedia.org/r/693938 [19:04:06] (03CR) 10Bstorm: [C: 03+2] Add an additional public key for Andrew Bogott [puppet] - 10https://gerrit.wikimedia.org/r/693938 (owner: 10Andrew Bogott) [19:04:38] (03Abandoned) 10Ppchelko: Move rate_limits configration to the VH level [deployment-charts] - 10https://gerrit.wikimedia.org/r/621044 (https://phabricator.wikimedia.org/T260591) (owner: 10Ppchelko) [19:07:33] (03CR) 10Clarakosi: [C: 03+1] api-gateway: exclude /heathz from r and w virtual clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/693930 (owner: 10Ppchelko) [19:08:48] (03CR) 10Ppchelko: [C: 03+2] api-gateway: exclude /heathz from r and w virtual clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/693930 (owner: 10Ppchelko) [19:11:14] (03Merged) 10jenkins-bot: api-gateway: exclude /heathz from r and w virtual clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/693930 (owner: 10Ppchelko) [19:12:39] 10SRE, 10Wikimedia-Mailing-lists, 10Bengali-Sites: Some Bangla texts/scripts are getting broken in mailman - https://phabricator.wikimedia.org/T283525 (10Aftabuzzaman) > Could you explain this a bit more for someone who unfortunately isn't familiar with Bengali? What should it look like? What makes it "broke... [19:22:25] (03CR) 10Razzi: "One quick question, some of this parameterization might not be needed, otherwise looks good." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692909 (https://phabricator.wikimedia.org/T274880) (owner: 10Mforns) [19:24:04] (03PS2) 10Ppchelko: api-gateway: move envoy CORS filter higher in the chain [deployment-charts] - 10https://gerrit.wikimedia.org/r/693922 [19:26:33] (03PS3) 10Ppchelko: api-gateway: move envoy CORS filter higher in the chain [deployment-charts] - 10https://gerrit.wikimedia.org/r/693922 [19:26:56] (03CR) 10Ppchelko: [C: 03+2] api-gateway: move envoy CORS filter higher in the chain [deployment-charts] - 10https://gerrit.wikimedia.org/r/693922 (owner: 10Ppchelko) [19:30:08] (03Merged) 10jenkins-bot: api-gateway: move envoy CORS filter higher in the chain [deployment-charts] - 10https://gerrit.wikimedia.org/r/693922 (owner: 10Ppchelko) [19:30:17] (03CR) 10Jbond: "thanks <3" [puppet] - 10https://gerrit.wikimedia.org/r/693496 (owner: 10Dzahn) [19:30:35] 10SRE, 10Wikimedia-Mailing-lists, 10Upstream: In Mailman3, users cannot change their display name from the web - https://phabricator.wikimedia.org/T283128 (10Marostegui) Yeah, talked to @ladsgroup earlier today to change the bot. [19:33:13] (03CR) 10Jbond: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/693926 (owner: 10Volans) [19:35:46] 10SRE, 10ops-eqiad, 10DC-Ops: Netbox Errors - https://phabricator.wikimedia.org/T283518 (10wiki_willy) [19:38:37] 10SRE, 10wikimedia-irc-libera: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10razzi) [19:39:05] 10SRE, 10Wikimedia-Hackathon-2021, 10Wikimedia-Mailing-lists, 10Upstream: Add OAuth login to mailman for accessing list memberships/archive viewing - https://phabricator.wikimedia.org/T249678 (10Legoktm) >>! In T249678#7107094, @Tgr wrote: > Do we want some sort of formal security review for this, BTW? I... [19:41:57] (03CR) 10Mforns: reportupdater: Rsync logs to HDFS (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692909 (https://phabricator.wikimedia.org/T274880) (owner: 10Mforns) [19:43:31] 10SRE, 10netops, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10jbond) >>! In T283359#7108488, @CDanis wrote: > Thanks very much for all the helpful comments! > > * I'll definitely enable the NSID bit, it wi... [19:44:36] 10SRE, 10ops-eqiad, 10DC-Ops, 10Dumps-Generation: (Need By: 2021-03-31) rack/setup/install snapshot101[1-5] - https://phabricator.wikimedia.org/T272509 (10Volans) 05Resolved→03Open p:05Medium→03High Re-opening because `snapshot1014` and `snapshot1015` don't have any role applied and Puppet is broke... [19:52:16] (03CR) 10Bartosz Dziewoński: [C: 03+1] Enable DiscussionTools on wikitech (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693897 (https://phabricator.wikimedia.org/T283119) (owner: 10Esanders) [19:57:07] (03CR) 10Herron: "full pcc looking for any unexpected changes, looks ok https://puppet-compiler.wmflabs.org/compiler1001/29657/" [puppet] - 10https://gerrit.wikimedia.org/r/693906 (https://phabricator.wikimedia.org/T283213) (owner: 10Herron) [19:58:10] (03CR) 10Herron: [C: 03+2] logstash101[012]: enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/693914 (owner: 10Herron) [20:00:05] chrisalbon and accraze: How many deployers does it take to do Services – Graphoid / ORES deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210524T2000). [20:04:05] (03CR) 10Razzi: reportupdater: Rsync logs to HDFS (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692909 (https://phabricator.wikimedia.org/T274880) (owner: 10Mforns) [20:10:30] 10SRE, 10LDAP-Access-Requests: Grant Access to Superset/Turnilo for Kgordon - https://phabricator.wikimedia.org/T283057 (10Maryana) @Aklapper Just making sure you didn't miss this request :) Let us know if we need to provide any additional info to complete this – thanks! [20:18:13] (03PS1) 10Eric Gardner: Enable MediaSearch Assessment filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693951 (https://phabricator.wikimedia.org/T276257) [20:19:14] (03CR) 10jerkins-bot: [V: 04-1] Enable MediaSearch Assessment filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693951 (https://phabricator.wikimedia.org/T276257) (owner: 10Eric Gardner) [20:20:13] (03PS2) 10Eric Gardner: Enable MediaSearch Assessment filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693951 (https://phabricator.wikimedia.org/T276257) [20:21:21] (03CR) 10jerkins-bot: [V: 04-1] Enable MediaSearch Assessment filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693951 (https://phabricator.wikimedia.org/T276257) (owner: 10Eric Gardner) [20:23:13] 10SRE, 10ops-eqiad, 10DC-Ops, 10Dumps-Generation: (Need By: 2021-03-31) rack/setup/install snapshot101[1-5] - https://phabricator.wikimedia.org/T272509 (10ArielGlenn) They have been on hold pending the availability of a colleague who would be learning about their setup. I hope to find out more about that p... [20:24:44] (03PS3) 10Eric Gardner: Enable MediaSearch Assessment filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693951 (https://phabricator.wikimedia.org/T276257) [20:25:57] (03CR) 10jerkins-bot: [V: 04-1] Enable MediaSearch Assessment filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693951 (https://phabricator.wikimedia.org/T276257) (owner: 10Eric Gardner) [20:31:13] (03PS4) 10Eric Gardner: Enable MediaSearch Assessment filter [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693951 (https://phabricator.wikimedia.org/T276257) [20:51:32] 10SRE, 10ops-codfw, 10DC-Ops, 10Discovery-Search: hw troubleshooting: failure to power up for elastic2043.codfw.wmnet - https://phabricator.wikimedia.org/T281327 (10Papaul) Hello Papaul, Here is your case information: Case#: 113228160 Tag#: GBJ8CS2 Model#: POWEREDGE R440 [20:56:29] 10SRE, 10Thumbor, 10Wikimedia-SVG-rendering, 10Upstream: Update librsvg to > 2.44.10 - https://phabricator.wikimedia.org/T265549 (10Aklapper) [20:59:19] 10SRE, 10netops, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) Some very initial results running against the anycast NS at https://atlas.ripe.net/frames/measurements/30366093#!probes Thanks again fo... [21:00:05] Reedy and sbassett: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Weekly Security deployment window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210524T2100). [21:01:07] (03PS3) 10Razzi: kerberos: require --email_address for create and reset-password [puppet] - 10https://gerrit.wikimedia.org/r/686766 (https://phabricator.wikimedia.org/T282185) [21:08:08] 10SRE, 10LDAP-Access-Requests: Grant Access to Superset/Turnilo for Kgordon - https://phabricator.wikimedia.org/T283057 (10Aklapper) @Maryana: I cannot do anything here, sorry - this is #SRE territory [21:09:36] (03CR) 10Razzi: kerberos: require --email_address for create and reset-password (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/686766 (https://phabricator.wikimedia.org/T282185) (owner: 10Razzi) [21:29:38] (03PS5) 10Cwhite: Build on the production builder host [software/logstash/plugins] - 10https://gerrit.wikimedia.org/r/688418 [21:30:27] (03PS1) 10Legoktm: Add shellbox.svc.{codfw,eqiad}.wmnet [dns] - 10https://gerrit.wikimedia.org/r/693957 (https://phabricator.wikimedia.org/T281423) [21:33:05] (03CR) 10Dzahn: [C: 03+1] "lgtm, matches netbox, .51 does not appear to be used" [dns] - 10https://gerrit.wikimedia.org/r/693957 (https://phabricator.wikimedia.org/T281423) (owner: 10Legoktm) [21:34:16] (03PS6) 10Cwhite: Build on the production builder host [software/logstash/plugins] - 10https://gerrit.wikimedia.org/r/688418 [21:36:44] (03CR) 10Legoktm: [C: 03+2] Add shellbox.svc.{codfw,eqiad}.wmnet [dns] - 10https://gerrit.wikimedia.org/r/693957 (https://phabricator.wikimedia.org/T281423) (owner: 10Legoktm) [21:39:45] 10SRE, 10Dumps-Generation: snapshot101[45] have no role, break puppet run - https://phabricator.wikimedia.org/T283545 (10RobH) [21:41:22] 10SRE, 10ops-codfw, 10DC-Ops, 10Discovery-Search: hw troubleshooting: failure to power up for elastic2043.codfw.wmnet - https://phabricator.wikimedia.org/T281327 (10Papaul) Dell ask to upgrade the IDRAC and BIOS since the TSR report doesn't show what component is failing, Maybe the new IDRAC and BIOS upgr... [21:42:09] 10SRE, 10ops-eqiad, 10DC-Ops, 10Dumps-Generation: (Need By: 2021-03-31) rack/setup/install snapshot101[1-5] - https://phabricator.wikimedia.org/T272509 (10RobH) 05Open→03Resolved Keeping a racking task in DC ops open for this isn't going to work, as this counts against our SLA metrics as part of our OK... [21:42:34] (03PS1) 10Cwhite: package_builder: add logstash-plugins build hooks [puppet] - 10https://gerrit.wikimedia.org/r/693958 [21:45:53] (03PS7) 10Cwhite: Set up build on production builder host [software/logstash/plugins] - 10https://gerrit.wikimedia.org/r/688418 [21:47:04] (03CR) 10Cwhite: [C: 03+2] Add HTTP request and response headers fields as object[field:keyword] [software/ecs] - 10https://gerrit.wikimedia.org/r/636515 (owner: 10Cwhite) [21:49:06] (03PS1) 10Legoktm: Add shellbox to LVS [puppet] - 10https://gerrit.wikimedia.org/r/693959 (https://phabricator.wikimedia.org/T281423) [21:49:08] (03PS1) 10Legoktm: service: Switch shellbox to lvs_setup [puppet] - 10https://gerrit.wikimedia.org/r/693960 (https://phabricator.wikimedia.org/T281423) [21:49:10] (03PS1) 10Legoktm: service: Switch shellbox to monitoring_setup [puppet] - 10https://gerrit.wikimedia.org/r/693961 (https://phabricator.wikimedia.org/T281423) [21:49:12] (03PS1) 10Legoktm: service: Switch shellbox to production [puppet] - 10https://gerrit.wikimedia.org/r/693962 (https://phabricator.wikimedia.org/T281423) [21:52:32] (03PS1) 10Cwhite: add whitespace [software/ecs] - 10https://gerrit.wikimedia.org/r/693963 [21:53:33] (03CR) 10Cwhite: [C: 03+2] add whitespace [software/ecs] - 10https://gerrit.wikimedia.org/r/693963 (owner: 10Cwhite) [21:54:10] (03Merged) 10jenkins-bot: add whitespace [software/ecs] - 10https://gerrit.wikimedia.org/r/693963 (owner: 10Cwhite) [21:57:49] (03PS1) 10Cwhite: logstash: bump ecs to 1.7.0-3 [puppet] - 10https://gerrit.wikimedia.org/r/693964 [21:58:12] (03PS1) 10Legoktm: Add shellbox to discovery [dns] - 10https://gerrit.wikimedia.org/r/693965 (https://phabricator.wikimedia.org/T281423) [22:01:01] (03PS1) 10Dzahn: add LVS service IPs for miscweb [dns] - 10https://gerrit.wikimedia.org/r/693966 (https://phabricator.wikimedia.org/T281538) [22:02:00] (03CR) 10Legoktm: [C: 03+1] "LGTM, same as what I did" [dns] - 10https://gerrit.wikimedia.org/r/693966 (https://phabricator.wikimedia.org/T281538) (owner: 10Dzahn) [22:02:42] (03CR) 10Dzahn: [C: 03+2] add LVS service IPs for miscweb [dns] - 10https://gerrit.wikimedia.org/r/693966 (https://phabricator.wikimedia.org/T281538) (owner: 10Dzahn) [22:06:56] (03PS1) 10Dzahn: Add discovery DNS for miscweb [dns] - 10https://gerrit.wikimedia.org/r/693968 (https://phabricator.wikimedia.org/T281538) [22:25:40] 10SRE, 10ops-codfw, 10DC-Ops, 10cloud-services-team (Hardware): (Need By: TBD) rack/setup/install cloudnet2004-dev - https://phabricator.wikimedia.org/T275676 (10Papaul) 05Open→03Resolved @aborrero it doesn't affect in any way when installing the server . I also change it to eno2 [22:30:31] (03PS1) 10Zabe: Revert "Use svwiki 20th anniversary logos" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693662 [22:30:52] (03PS2) 10Zabe: Revert "Use svwiki 20th anniversary logos" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693662 (https://phabricator.wikimedia.org/T282389) [22:31:11] (03PS3) 10Zabe: Revert "Use svwiki 20th anniversary logos" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693662 (https://phabricator.wikimedia.org/T282389) [22:34:32] (03PS1) 10Zabe: Revert "Add svwiki 20th anniversary logos" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693663 [22:34:56] (03CR) 10jerkins-bot: [V: 04-1] Revert "Add svwiki 20th anniversary logos" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693663 (owner: 10Zabe) [22:39:04] (03PS2) 10Zabe: Revert "Add svwiki 20th anniversary logos" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693663 [22:40:04] (03PS3) 10Zabe: Revert "Add svwiki 20th anniversary logos" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693663 (https://phabricator.wikimedia.org/T282389) [22:53:21] (03PS1) 10Andrew Bogott: Cloud-vps hiera: add public IPs to the list of bastion IPs [puppet] - 10https://gerrit.wikimedia.org/r/693973 [22:55:45] (03CR) 10Bstorm: Cloud-vps hiera: add public IPs to the list of bastion IPs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/693973 (owner: 10Andrew Bogott) [22:57:35] (03PS2) 10Andrew Bogott: Cloud-vps hiera: update IP of restricted bastion [puppet] - 10https://gerrit.wikimedia.org/r/693973 [22:58:10] (03CR) 10Bstorm: [C: 03+2] Cloud-vps hiera: update IP of restricted bastion [puppet] - 10https://gerrit.wikimedia.org/r/693973 (owner: 10Andrew Bogott) [23:00:04] RoanKattouw, Niharika, and Urbanecm: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Evening backport window . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210524T2300). [23:00:04] zabe: A patch you scheduled for Evening backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [23:17:36] (03CR) 10Bstorm: [C: 03+2] toolforge: Add separate role for Redis Sentinel [puppet] - 10https://gerrit.wikimedia.org/r/690528 (https://phabricator.wikimedia.org/T153810) (owner: 10Majavah) [23:31:44] (03PS1) 10Razzi: superset: remove caching [puppet] - 10https://gerrit.wikimedia.org/r/693981 (https://phabricator.wikimedia.org/T273850) [23:32:43] (03CR) 10Razzi: [V: 03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29660/console" [puppet] - 10https://gerrit.wikimedia.org/r/693981 (https://phabricator.wikimedia.org/T273850) (owner: 10Razzi) [23:47:08] (03CR) 10Bstorm: "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/691154 (owner: 10Arturo Borrero Gonzalez) [23:57:47] (03CR) 10Legoktm: Add helmfile.d for shellbox (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/692736 (https://phabricator.wikimedia.org/T281423) (owner: 10Legoktm) [23:57:50] (03PS3) 10Legoktm: Add helmfile.d for shellbox [deployment-charts] - 10https://gerrit.wikimedia.org/r/692736 (https://phabricator.wikimedia.org/T281423) [23:57:59] (03PS4) 10Legoktm: Add helmfile.d for shellbox [deployment-charts] - 10https://gerrit.wikimedia.org/r/692736 (https://phabricator.wikimedia.org/T281423)