[00:01:06] 10SRE, 10Wikimedia-Hackathon-2021, 10Wikimedia-Mailing-lists, 10Upstream: Add OAuth login to mailman for accessing list memberships/archive viewing - https://phabricator.wikimedia.org/T249678 (10Legoktm) @Chicocvenancio expressed interested on working on this during the hackathon as well :) Should we pick... [00:18:07] 10SRE, 10ops-eqiad, 10DC-Ops: Relabel db1125 to be dbstore1006 - https://phabricator.wikimedia.org/T283300 (10razzi) [01:54:11] 10SRE, 10Analytics-Radar, 10LDAP-Access-Requests, 10SRE-Access-Requests: Account setup issues for jmixter-ctr - https://phabricator.wikimedia.org/T283250 (10Dzahn) When using Phabricator it is an "either or" thing. You can either use an existing SUL (global wiki projects) user or you can use an existing de... [02:04:51] 10SRE, 10wikimedia-irc-freenode: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Dzahn) @bd808 should #-cloud" be on this list, in addition to #-cloud-admin? [02:47:19] 10SRE, 10wikimedia-irc-freenode: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10bd808) >>! In T283230#7102269, @Dzahn wrote: > @bd808 should #-cloud" be on this list, in addition to #-cloud-admin? I suppose they could be, sure. `#wikimedia-cloud`, `#wikimedia-cloud-admi... [02:48:45] 10SRE, 10wikimedia-irc-freenode: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10bd808) [02:50:33] 10SRE, 10wikimedia-irc-freenode: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10bd808) [03:08:53] 10SRE, 10wikimedia-irc-freenode: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Dzahn) [03:25:50] (03CR) 10Thcipriani: [V: 03+2 C: 03+2] Allow TrainBranchBot to create refs in operations/mediawiki-config [mediawiki-config] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/692660 (owner: 10Ahmon Dancy) [04:43:06] (03PS1) 10Marostegui: Revert "db1143: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/693255 [04:44:50] (03CR) 10Marostegui: [C: 03+2] Revert "db1143: Disable notifications" [puppet] - 10https://gerrit.wikimedia.org/r/693255 (owner: 10Marostegui) [04:49:11] (03PS1) 10Marostegui: mariadb: Decommission db1087 [puppet] - 10https://gerrit.wikimedia.org/r/693279 (https://phabricator.wikimedia.org/T282093) [04:56:02] (03PS1) 10Marostegui: db1142: Disable notifications. [puppet] - 10https://gerrit.wikimedia.org/r/693280 [04:56:57] (03CR) 10Marostegui: [C: 03+2] db1142: Disable notifications. [puppet] - 10https://gerrit.wikimedia.org/r/693280 (owner: 10Marostegui) [04:57:06] (03CR) 10Marostegui: [C: 03+2] mariadb: Decommission db1087 [puppet] - 10https://gerrit.wikimedia.org/r/693279 (https://phabricator.wikimedia.org/T282093) (owner: 10Marostegui) [05:00:16] 10ops-eqiad, 10DC-Ops, 10decommission-hardware: decommission db1087.eqiad.wmnet - https://phabricator.wikimedia.org/T282093 (10Marostegui) [05:00:36] 10SRE, 10DBA, 10Patch-For-Review: Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers) - https://phabricator.wikimedia.org/T258361 (10Marostegui) [05:20:30] (03PS4) 10Muehlenhoff: Skip Cumin/Homer/Spicerack on cumin2001 [puppet] - 10https://gerrit.wikimedia.org/r/693130 (https://phabricator.wikimedia.org/T276589) [05:23:58] (03CR) 10Muehlenhoff: Skip Cumin/Homer/Spicerack on cumin2001 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/693130 (https://phabricator.wikimedia.org/T276589) (owner: 10Muehlenhoff) [05:30:50] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/693130 (https://phabricator.wikimedia.org/T276589) (owner: 10Muehlenhoff) [05:32:55] (03PS1) 10Marostegui: install_server: Do not format /srv on dbstore1006 [puppet] - 10https://gerrit.wikimedia.org/r/693296 (https://phabricator.wikimedia.org/T283125) [05:34:39] (03CR) 10Muehlenhoff: (WIP) create a logout.d profile for managing logout scripts (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/693149 (owner: 10Jbond) [06:29:01] 10SRE, 10DBA, 10Platform Engineering Roadmap Decision Making, 10Performance-Team (Radar), 10User-Kormat: Remove groups from db configs - https://phabricator.wikimedia.org/T263127 (10Marostegui) I want to start working on this next week on s6. [06:34:58] 10SRE, 10Wikimedia-Mailing-lists, 10Wikimedia-Small-Projects-User-Group, 10User-Ladsgroup: Mailing list request for Wikimedia Small Projects User Group - https://phabricator.wikimedia.org/T283310 (10Ladsgroup) p:05Triage→03Medium a:03Ladsgroup Changing name of a mailing list after it's created will b... [06:37:44] 10SRE, 10Analytics-Radar, 10LDAP-Access-Requests, 10SRE-Access-Requests: Account setup issues for jmixter-ctr - https://phabricator.wikimedia.org/T283250 (10elukey) After a chat with Moritz I think that I get what's happening. In the email that OIT sends there is a mention of LDAP and how to change its pas... [06:39:06] 10SRE, 10Wikimedia-Mailing-lists, 10Wikimedia-Small-Projects-User-Group, 10User-Ladsgroup: Mailing list request for Wikimedia Small Projects User Group - https://phabricator.wikimedia.org/T283310 (10White-Master) @Ladsgroup I meant that it was subject to change if the person in charge of creating the list... [06:53:07] (03Abandoned) 10Elukey: hadoop: Add hiera setting to symlink hadoop logs to /var/log/hadoop [puppet] - 10https://gerrit.wikimedia.org/r/661391 (https://phabricator.wikimedia.org/T265126) (owner: 10Razzi) [06:55:32] 10SRE, 10ops-eqiad, 10Analytics-Radar: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10elukey) @Cmjohnson Hi! Do you have time next week for the A2 rack? [06:56:16] (03CR) 10Elukey: "ping :)" [deployment-charts] - 10https://gerrit.wikimedia.org/r/683706 (https://phabricator.wikimedia.org/T225005) (owner: 10Herron) [06:58:56] (03PS2) 10Muehlenhoff: Add grant for cumin2002 [puppet] - 10https://gerrit.wikimedia.org/r/692621 (https://phabricator.wikimedia.org/T276589) [06:59:11] (03CR) 10Muehlenhoff: Add grant for cumin2002 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/692621 (https://phabricator.wikimedia.org/T276589) (owner: 10Muehlenhoff) [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210521T0700) [07:09:57] (03CR) 10Muehlenhoff: C:admin: add ability to manage home (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/691131 (https://phabricator.wikimedia.org/T280989) (owner: 10Jbond) [07:12:28] (03CR) 10Elukey: "LGTM! Didn't +1 since there is the question mark about the absent dir/cmd that Alex brought up :)" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/693174 (owner: 10Giuseppe Lavagetto) [07:22:27] (03CR) 10Gilles: [C: 03+1] trafficserver: Remove X-Request-Id from response headers unless debug [puppet] - 10https://gerrit.wikimedia.org/r/676682 (https://phabricator.wikimedia.org/T283291) (owner: 10Krinkle) [07:23:01] (03PS30) 10Jcrespo: mediabackup: Initial setup for the media backup worker hosts [puppet] - 10https://gerrit.wikimedia.org/r/668380 (https://phabricator.wikimedia.org/T276442) [07:25:55] (03CR) 10Elukey: "Left some comments but I am very ignorant about all this, so it may not be super relevant :)" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/693175 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [07:26:00] (03PS31) 10Jcrespo: mediabackup: Initial setup for the media backup worker hosts [puppet] - 10https://gerrit.wikimedia.org/r/668380 (https://phabricator.wikimedia.org/T276442) [07:28:24] (03CR) 10Ema: [C: 03+2] trafficserver: Remove X-Request-Id from response headers unless debug [puppet] - 10https://gerrit.wikimedia.org/r/676682 (https://phabricator.wikimedia.org/T283291) (owner: 10Krinkle) [07:29:17] (03CR) 10Filippo Giunchedi: "I think the filter is correct in the sense that we want to select only indices allocated on ssd to move to hdd. I think the problem could " [puppet] - 10https://gerrit.wikimedia.org/r/693213 (owner: 10Cwhite) [07:29:24] (03CR) 10Filippo Giunchedi: [C: 04-1] logstash: bugfix filter to exclude hdd-allocated indexes [puppet] - 10https://gerrit.wikimedia.org/r/693213 (owner: 10Cwhite) [07:30:20] 10SRE, 10LDAP-Access-Requests: Grant Access to ldap/wmf for JStephenson1980 - https://phabricator.wikimedia.org/T282521 (10JStephenson) I managed! Thank you very much for your support! [07:37:47] (03PS32) 10Jcrespo: mediabackup: Initial setup for the media backup worker hosts [puppet] - 10https://gerrit.wikimedia.org/r/668380 (https://phabricator.wikimedia.org/T276442) [07:41:52] 10SRE, 10User-Elukey: Investigate janitor, maintenance emails parser - https://phabricator.wikimedia.org/T230835 (10ayounsi) Might be better to switch focus to https://github.com/networktocode/circuit-maintenance-parser cf. https://ripe82.ripe.net/archives/video/516/ [07:42:27] (03PS1) 10Alexandros Kosiaris: Allow absenting the helmchartctl systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/693352 (https://phabricator.wikimedia.org/T283147) [07:46:29] (03CR) 10Alexandros Kosiaris: [V: 03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29634/console" [puppet] - 10https://gerrit.wikimedia.org/r/693352 (https://phabricator.wikimedia.org/T283147) (owner: 10Alexandros Kosiaris) [07:49:24] (03CR) 10JMeybohm: [C: 03+1] Allow absenting the helmchartctl systemd timer [puppet] - 10https://gerrit.wikimedia.org/r/693352 (https://phabricator.wikimedia.org/T283147) (owner: 10Alexandros Kosiaris) [07:52:17] (03CR) 10Alexandros Kosiaris: [V: 03+1 C: 03+2] "Thanks! PCC is happy at https://puppet-compiler.wmflabs.org/compiler1002/29634/, merging to verify hypothesis. I 'll upload and merge a ch" [puppet] - 10https://gerrit.wikimedia.org/r/693352 (https://phabricator.wikimedia.org/T283147) (owner: 10Alexandros Kosiaris) [07:53:07] (03PS1) 10Jcrespo: mediabackups: Move the database password to the common namespace [labs/private] - 10https://gerrit.wikimedia.org/r/693355 (https://phabricator.wikimedia.org/T276442) [07:53:23] 10SRE: Cookbooks for Ganeti maintenance tasks - https://phabricator.wikimedia.org/T283319 (10MoritzMuehlenhoff) [07:54:20] 10SRE: Cookbooks for Ganeti maintenance tasks - https://phabricator.wikimedia.org/T283319 (10MoritzMuehlenhoff) [07:54:23] 10SRE: Create cookbook to add a node to a Ganeti cluster - https://phabricator.wikimedia.org/T274527 (10MoritzMuehlenhoff) [07:54:55] 10SRE: Cookbooks for Ganeti maintenance tasks - https://phabricator.wikimedia.org/T283319 (10MoritzMuehlenhoff) [07:54:58] 10SRE, 10SRE-tools, 10User-Joe: Create a spicerack cookbook to empty a ganeti node from VMs - https://phabricator.wikimedia.org/T203964 (10MoritzMuehlenhoff) [07:54:59] (03CR) 10Jcrespo: [V: 03+2 C: 03+2] mediabackups: Move the database password to the common namespace [labs/private] - 10https://gerrit.wikimedia.org/r/693355 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [08:02:27] (03PS19) 10DCausse: rdf-streaming-updater: create helmfile.d structure [deployment-charts] - 10https://gerrit.wikimedia.org/r/671204 (https://phabricator.wikimedia.org/T264006) (owner: 10Mstyles) [08:02:29] (03PS10) 10DCausse: rdf-streaming-updater: enable HA capability [deployment-charts] - 10https://gerrit.wikimedia.org/r/679519 (https://phabricator.wikimedia.org/T273098) (owner: 10Mstyles) [08:02:31] (03PS11) 10DCausse: rdf-streaming-updater: use session mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/681497 (https://phabricator.wikimedia.org/T280166) (owner: 10Mstyles) [08:02:33] (03PS1) 10DCausse: helm_diff: supports new and renamed charts [deployment-charts] - 10https://gerrit.wikimedia.org/r/693359 [08:06:04] 10SRE: Cookbook to failover the Ganeti master - https://phabricator.wikimedia.org/T283320 (10MoritzMuehlenhoff) [08:06:39] (03PS33) 10Jcrespo: mediabackup: Initial setup for the media backup worker hosts [puppet] - 10https://gerrit.wikimedia.org/r/668380 (https://phabricator.wikimedia.org/T276442) [08:23:28] (03CR) 10Volans: [C: 03+1] "LGTM" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/693130 (https://phabricator.wikimedia.org/T276589) (owner: 10Muehlenhoff) [08:25:08] 10SRE, 10vm-requests: codfw: 4 VM request for docker-registry - https://phabricator.wikimedia.org/T283321 (10JMeybohm) [08:30:15] 10SRE, 10Analytics, 10Traffic: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10hashar) That is still hitting us from time to time, specially when the CI cache is not warmed up or has been wiped. [08:32:48] (03CR) 10Jcrespo: [C: 03+2] mediabackup: Initial setup for the media backup worker hosts [puppet] - 10https://gerrit.wikimedia.org/r/668380 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [08:43:54] (03CR) 10Kormat: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/692621 (https://phabricator.wikimedia.org/T276589) (owner: 10Muehlenhoff) [08:49:49] 10SRE, 10Traffic, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10ema) Looking at average p75 over the last two days, all varnish versions in text@esams now perform very similarly: |**host**... [08:51:07] (03CR) 10Muehlenhoff: [C: 03+2] Add grant for cumin2002 [puppet] - 10https://gerrit.wikimedia.org/r/692621 (https://phabricator.wikimedia.org/T276589) (owner: 10Muehlenhoff) [08:58:12] (03PS1) 10ArielGlenn: fix up comment for mediainfo dumps configuration file size setting [puppet] - 10https://gerrit.wikimedia.org/r/693366 (https://phabricator.wikimedia.org/T280624) [08:59:35] (03CR) 10ArielGlenn: [C: 03+2] fix up comment for mediainfo dumps configuration file size setting [puppet] - 10https://gerrit.wikimedia.org/r/693366 (https://phabricator.wikimedia.org/T280624) (owner: 10ArielGlenn) [09:00:25] (03PS1) 10Alexandros Kosiaris: Add an extra latency bucket at 10s [deployment-charts] - 10https://gerrit.wikimedia.org/r/693367 (https://phabricator.wikimedia.org/T283147) [09:03:52] (03Abandoned) 10Alexandros Kosiaris: mathoid: Rework the canary approach [deployment-charts] - 10https://gerrit.wikimedia.org/r/564054 (owner: 10Alexandros Kosiaris) [09:03:59] (03Abandoned) 10Alexandros Kosiaris: mathoid: 2nd test of canary functionality in codfw [deployment-charts] - 10https://gerrit.wikimedia.org/r/564084 (owner: 10Alexandros Kosiaris) [09:04:07] (03CR) 10Alexandros Kosiaris: [C: 03+2] Add an extra latency bucket at 10s [deployment-charts] - 10https://gerrit.wikimedia.org/r/693367 (https://phabricator.wikimedia.org/T283147) (owner: 10Alexandros Kosiaris) [09:04:55] (03PS1) 10ArielGlenn: add new mirror for xml/sql dumps: wikimedia.bringyour.com [puppet] - 10https://gerrit.wikimedia.org/r/693369 [09:06:36] (03Merged) 10jenkins-bot: Add an extra latency bucket at 10s [deployment-charts] - 10https://gerrit.wikimedia.org/r/693367 (https://phabricator.wikimedia.org/T283147) (owner: 10Alexandros Kosiaris) [09:14:13] (03PS11) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [09:21:10] (03PS12) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [09:23:11] (03PS1) 10Jcrespo: mariadb: Fix small typo on comment [puppet] - 10https://gerrit.wikimedia.org/r/693373 [09:24:18] (03CR) 10Jcrespo: "I run into this when working on https://gerrit.wikimedia.org/r/c/operations/puppet/+/681103" [puppet] - 10https://gerrit.wikimedia.org/r/693373 (owner: 10Jcrespo) [09:24:46] (03PS4) 10Gehel: cloudelastic: bump inactive shard alert threshold [puppet] - 10https://gerrit.wikimedia.org/r/693204 (https://phabricator.wikimedia.org/T283269) (owner: 10Ryan Kemper) [09:28:39] (03CR) 10Kormat: [C: 03+1] install_server: Do not format /srv on dbstore1006 [puppet] - 10https://gerrit.wikimedia.org/r/693296 (https://phabricator.wikimedia.org/T283125) (owner: 10Marostegui) [09:31:04] (03PS12) 10DCausse: rdf-streaming-updater: use session mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/681497 (https://phabricator.wikimedia.org/T280166) (owner: 10Mstyles) [09:32:49] (03CR) 10jerkins-bot: [V: 04-1] rdf-streaming-updater: use session mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/681497 (https://phabricator.wikimedia.org/T280166) (owner: 10Mstyles) [09:38:46] (03CR) 10Kormat: "I think you need to add an entry to `profile::mariadb::section_params` for this section. Something like:" [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [09:38:50] (03CR) 10Jcrespo: [C: 03+2] doc: link to wikitech for list of cumin hosts [software/transferpy] - 10https://gerrit.wikimedia.org/r/690355 (owner: 10Volans) [09:39:14] (03CR) 10Kormat: [C: 03+1] "Thanks :)" [puppet] - 10https://gerrit.wikimedia.org/r/693373 (owner: 10Jcrespo) [09:40:16] (03CR) 10Kormat: "Doing a PCC run is strongly recommended, too." [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [09:44:10] (03PS1) 10Muehlenhoff: Add cumin2002 to mysql_root_clients [puppet] - 10https://gerrit.wikimedia.org/r/693376 (https://phabricator.wikimedia.org/T276589) [09:44:30] (03PS11) 10Hashar: [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 [09:45:14] (03CR) 10Jcrespo: "> Patch Set 12:" [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [09:46:16] (03PS1) 10Ema: varnish: add .sh suffix to shell scripts [puppet] - 10https://gerrit.wikimedia.org/r/693377 (https://phabricator.wikimedia.org/T148494) [09:47:40] 10SRE, 10observability, 10CAS-SSO: thanos u/i gives errors if left idle for a few hours - https://phabricator.wikimedia.org/T268233 (10fgiunchedi) > What's the current refresh time for an SSO session before the refresh in the background kicks in? @jbond @Muehlenhoff re: the above, what's the current SSO sess... [09:52:50] (03PS7) 10Hashar: [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 [09:55:26] (03PS1) 10Filippo Giunchedi: pontoon: add swift stack [puppet] - 10https://gerrit.wikimedia.org/r/693381 [09:55:45] 10SRE, 10observability, 10CAS-SSO: thanos u/i gives errors if left idle for a few hours - https://phabricator.wikimedia.org/T268233 (10MoritzMuehlenhoff) >>! In T268233#7102841, @fgiunchedi wrote: >> What's the current refresh time for an SSO session before the refresh in the background kicks in? > @jbond @M... [09:56:37] (03Abandoned) 10Muehlenhoff: Switch time server on dns5001 to Chrony [puppet] - 10https://gerrit.wikimedia.org/r/422387 (https://phabricator.wikimedia.org/T177742) (owner: 10Muehlenhoff) [09:56:46] (03CR) 10Filippo Giunchedi: [C: 03+2] pontoon: add swift stack [puppet] - 10https://gerrit.wikimedia.org/r/693381 (owner: 10Filippo Giunchedi) [09:57:28] (03CR) 10Ema: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/693377 (https://phabricator.wikimedia.org/T148494) (owner: 10Ema) [09:58:25] (03CR) 10jerkins-bot: [V: 04-1] [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 (owner: 10Hashar) [09:58:43] (03CR) 10Muehlenhoff: "Let's revisit now that we're running 6.3 again?" [puppet] - 10https://gerrit.wikimedia.org/r/661713 (https://phabricator.wikimedia.org/T273867) (owner: 10Jbond) [09:59:10] (03CR) 10Kormat: [C: 03+1] Add cumin2002 to mysql_root_clients [puppet] - 10https://gerrit.wikimedia.org/r/693376 (https://phabricator.wikimedia.org/T276589) (owner: 10Muehlenhoff) [09:59:35] (03PS3) 10Jbond: apereo_cas: rename config properties [puppet] - 10https://gerrit.wikimedia.org/r/661713 (https://phabricator.wikimedia.org/T273867) [10:00:31] (03PS12) 10Hashar: [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 [10:00:43] (03PS8) 10Hashar: [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 [10:00:49] (03CR) 10Jbond: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29638/console" [puppet] - 10https://gerrit.wikimedia.org/r/661713 (https://phabricator.wikimedia.org/T273867) (owner: 10Jbond) [10:02:29] (03CR) 10Jcrespo: "After a lot of reading (it was not obvious to me), I found where other sections are defined at:" [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [10:03:38] (03PS1) 10JMeybohm: Add 4 new docker-regisry nodes in codfw [puppet] - 10https://gerrit.wikimedia.org/r/693382 (https://phabricator.wikimedia.org/T283321) [10:03:40] (03PS1) 10JMeybohm: conftool: Add 4 new docker-registry hosts [puppet] - 10https://gerrit.wikimedia.org/r/693383 (https://phabricator.wikimedia.org/T264209) [10:05:03] (03CR) 10JMeybohm: [C: 03+2] Add 4 new docker-regisry nodes in codfw [puppet] - 10https://gerrit.wikimedia.org/r/693382 (https://phabricator.wikimedia.org/T283321) (owner: 10JMeybohm) [10:05:06] 10SRE, 10observability, 10CAS-SSO: thanos u/i gives errors if left idle for a few hours - https://phabricator.wikimedia.org/T268233 (10fgiunchedi) >>! In T268233#7102899, @MoritzMuehlenhoff wrote: >>>! In T268233#7102841, @fgiunchedi wrote: >>> What's the current refresh time for an SSO session before the re... [10:06:50] (03PS13) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [10:11:40] (03CR) 10Marostegui: [C: 03+1] mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [10:12:02] (03PS14) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [10:12:45] (03CR) 10jerkins-bot: [V: 04-1] [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 (owner: 10Hashar) [10:14:31] (03CR) 10Jcrespo: "https://puppet-compiler.wmflabs.org/compiler1001/29640/" [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [10:14:34] (03PS8) 10Jbond: C:admin: add ability to manage home [puppet] - 10https://gerrit.wikimedia.org/r/691131 (https://phabricator.wikimedia.org/T280989) [10:14:52] (03PS15) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [10:15:23] (03PS13) 10Hashar: [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 [10:15:25] (03PS16) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [10:15:33] (03PS9) 10Hashar: [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 [10:15:40] (03CR) 10Jbond: "updated" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/691131 (https://phabricator.wikimedia.org/T280989) (owner: 10Jbond) [10:17:06] 10SRE, 10Patch-For-Review: migrate services from cumin2001 to cumin2002 - https://phabricator.wikimedia.org/T276589 (10Kormat) The grant for cumin2002 should now be fully deployed. [10:17:43] (03PS2) 10Jcrespo: mariadb: Fix small typo on comment [puppet] - 10https://gerrit.wikimedia.org/r/693373 [10:17:45] (03CR) 10jerkins-bot: [V: 04-1] [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 (owner: 10Hashar) [10:18:03] (03PS3) 10Jcrespo: mariadb: Fix small typo on comment [puppet] - 10https://gerrit.wikimedia.org/r/693373 [10:19:52] (03CR) 10Jcrespo: [C: 03+2] mariadb: Fix small typo on comment [puppet] - 10https://gerrit.wikimedia.org/r/693373 (owner: 10Jcrespo) [10:20:03] (03PS10) 10Hashar: [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 [10:20:42] (03PS17) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [10:21:17] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good. I've also tested this on idp-test by applying the new config names locally and everything continues to work fine." [puppet] - 10https://gerrit.wikimedia.org/r/661713 (https://phabricator.wikimedia.org/T273867) (owner: 10Jbond) [10:27:44] 10SRE, 10netops: Lumen 10G Wave (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) Lumen confirmed fix, we have been stable since. Closing task. __________________________________________________________________________________ From: No-Reply via Ops Maintenance [mailto:ops... [10:28:37] (03PS18) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [10:28:40] 10SRE, 10netops: Lumen 10G Wave (cr2-eqiad to cr2-esams) Down - https://phabricator.wikimedia.org/T283227 (10cmooney) 05Open→03Resolved [10:30:14] (03CR) 10Hashar: "Finally I got some build script to build Gerrit with support for our extra plugins (which are added as submodules by https://gerrit.wikime" [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 (owner: 10Hashar) [10:30:49] 10SRE, 10CAS-SSO: Cookbook for centralised logouts and session status queries - https://phabricator.wikimedia.org/T283242 (10jbond) I think it would also be nice to have the ability to show information for all logged in users. logging out all users from a specific service seems less useful but could probably... [10:32:25] (03PS19) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [10:32:28] (03CR) 10Hashar: "The list comes from https://gerrit.wikimedia.org/r/admin/plugins there are some helper to assist, notably "wmf-plugins-updates" which upd" [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 (owner: 10Hashar) [10:39:26] (03CR) 10Kormat: "> Patch Set 12:" [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [10:41:27] (03CR) 10Muehlenhoff: [C: 03+1] "LGTM" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/691131 (https://phabricator.wikimedia.org/T280989) (owner: 10Jbond) [10:41:33] (03CR) 10Kormat: [C: 03+1] "PCC seems happy, so LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [10:42:13] (03PS9) 10Jbond: C:admin: add ability to manage home [puppet] - 10https://gerrit.wikimedia.org/r/691131 (https://phabricator.wikimedia.org/T280989) [10:44:27] 10SRE, 10CAS-SSO: Cookbook for centralised logouts and session status queries - https://phabricator.wikimedia.org/T283242 (10MoritzMuehlenhoff) >>! In T283242#7103013, @jbond wrote: > I think it would also be nice to have the ability to show information for all logged in users. logging out all users from a sp... [10:44:34] (03PS20) 10Jcrespo: mariadb: Setup 2 new host as temporary metadata database for media backups [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) [10:44:37] (03PS8) 10Jcrespo: mediabackup: Setup the storage hosts [puppet] - 10https://gerrit.wikimedia.org/r/681117 (https://phabricator.wikimedia.org/T276442) [10:45:22] (03CR) 10Jcrespo: [C: 03+2] "> Patch Set 19: Code-Review+1" [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [10:45:35] 10SRE, 10CAS-SSO: Cookbook for centralised logouts and session status queries - https://phabricator.wikimedia.org/T283242 (10MoritzMuehlenhoff) [10:47:23] (03CR) 10Jcrespo: [C: 03+2] "> Patch Set 19:" [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [10:48:11] (03CR) 10Jcrespo: [C: 03+2] mariadb: Setup 2 new host as temporary metadata database for media backups (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/681103 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [10:53:10] (03Abandoned) 10Muehlenhoff: Depool poolcounter1005 for reboot [mediawiki-config] - 10https://gerrit.wikimedia.org/r/663584 (owner: 10Muehlenhoff) [10:55:05] 10SRE, 10DBA: wmf-auto-reinstall fails on hosts that run pt-heartbeat - https://phabricator.wikimedia.org/T252528 (10LSobanski) >>! In T252528#7100826, @Kormat wrote: > This does mean that `pt-heartbeat-wikimedia` needs to be started manually after a boot, however. @Kormat is this captured somewhere in docume... [11:05:44] (03PS1) 10Jbond: puppetdb-api: add new discovery service for the puppetdb micros service [puppet] - 10https://gerrit.wikimedia.org/r/693401 (https://phabricator.wikimedia.org/T283185) [11:12:35] (03PS1) 10Jbond: puppetdb-api: add new discovery service for the puppetdb micros service [dns] - 10https://gerrit.wikimedia.org/r/693403 (https://phabricator.wikimedia.org/T283185) [11:12:40] (03PS2) 10Jbond: puppetdb-api: add new discovery service for the puppetdb micros service [puppet] - 10https://gerrit.wikimedia.org/r/693401 (https://phabricator.wikimedia.org/T283185) [11:14:00] (03CR) 10Klausman: Add knative serving and net-istio images (033 comments) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [11:22:43] (03CR) 10Jbond: [C: 03+2] C:admin: add ability to manage home [puppet] - 10https://gerrit.wikimedia.org/r/691131 (https://phabricator.wikimedia.org/T280989) (owner: 10Jbond) [11:34:52] (03CR) 10Jbond: C:admin: add ability to manage home (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/691131 (https://phabricator.wikimedia.org/T280989) (owner: 10Jbond) [11:42:45] (03Abandoned) 10Hashar: Add rename-project plugin @ a880148 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/630548 (https://phabricator.wikimedia.org/T201953) (owner: 10Hashar) [11:43:25] 10SRE, 10ops-codfw, 10DBA, 10DC-Ops: (Need By: TBD) rack/setup/install db21[45-52] - https://phabricator.wikimedia.org/T273568 (10jcrespo) [11:43:27] 10SRE, 10ops-eqiad, 10DBA, 10DC-Ops, 10Patch-For-Review: (Need By: TBD) rack/setup/install db11[76-84] - https://phabricator.wikimedia.org/T273566 (10jcrespo) [11:47:32] (03CR) 10Ema: [C: 03+1] "One comment about firewall, LGTM otherwise." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/693401 (https://phabricator.wikimedia.org/T283185) (owner: 10Jbond) [11:50:00] (03PS3) 10Jbond: puppetdb-api: add new discovery service for the puppetdb microservice [puppet] - 10https://gerrit.wikimedia.org/r/693401 (https://phabricator.wikimedia.org/T283185) [11:50:17] (03PS2) 10Jbond: puppetdb-api: add new discovery service for the puppetdb microservice [dns] - 10https://gerrit.wikimedia.org/r/693403 (https://phabricator.wikimedia.org/T283185) [11:53:01] (03CR) 10Jcrespo: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1002/29643/" [puppet] - 10https://gerrit.wikimedia.org/r/681117 (https://phabricator.wikimedia.org/T276442) (owner: 10Jcrespo) [11:56:04] 10SRE, 10DBA: wmf-auto-reinstall fails on hosts that run pt-heartbeat - https://phabricator.wikimedia.org/T252528 (10Kormat) >>! In T252528#7103046, @LSobanski wrote: >>>! In T252528#7100826, @Kormat wrote: >> This does mean that `pt-heartbeat-wikimedia` needs to be started manually after a boot, however. > >... [11:56:46] (03PS4) 10Jbond: puppetdb-api: add new discovery service for the puppetdb microservice [puppet] - 10https://gerrit.wikimedia.org/r/693401 (https://phabricator.wikimedia.org/T283185) [11:57:31] (03PS1) 10Jcrespo: mariadb: Reenable notifications on db1176, db2151 after setup [puppet] - 10https://gerrit.wikimedia.org/r/693407 (https://phabricator.wikimedia.org/T275633) [11:58:44] (03CR) 10Ayounsi: system::role: add ability to specify role owners (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/692632 (owner: 10Jbond) [11:59:02] (03CR) 10Jcrespo: [C: 03+2] mariadb: Reenable notifications on db1176, db2151 after setup [puppet] - 10https://gerrit.wikimedia.org/r/693407 (https://phabricator.wikimedia.org/T275633) (owner: 10Jcrespo) [12:00:49] 10SRE, 10Traffic, 10Patch-For-Review, 10Performance-Team (Radar): 8-10% response start regression (Varnish 5.1.3-1wm15 -> 6.0.6-1wm1) - https://phabricator.wikimedia.org/T264398 (10Gilles) Thank you very much for the thorough investigation and explanations, I agree that now the issue is well understood and... [12:01:46] (03CR) 10Ema: [C: 03+1] puppetdb-api: add new discovery service for the puppetdb microservice [puppet] - 10https://gerrit.wikimedia.org/r/693401 (https://phabricator.wikimedia.org/T283185) (owner: 10Jbond) [12:05:14] (03CR) 10Ema: [C: 03+1] puppetdb-api: add new discovery service for the puppetdb microservice [dns] - 10https://gerrit.wikimedia.org/r/693403 (https://phabricator.wikimedia.org/T283185) (owner: 10Jbond) [12:11:04] (03PS2) 10JMeybohm: conftool: Add 4 new docker-registry hosts [puppet] - 10https://gerrit.wikimedia.org/r/693383 (https://phabricator.wikimedia.org/T264209) [12:13:58] (03CR) 10Ayounsi: [C: 03+1] "As this is also used on the recdns, servers I'd recommend disabling puppet on them except one, to make sure it doesn't break them." [puppet] - 10https://gerrit.wikimedia.org/r/692367 (https://phabricator.wikimedia.org/T283027) (owner: 10Ssingh) [12:19:01] (03PS13) 10DCausse: rdf-streaming-updater: use session mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/681497 (https://phabricator.wikimedia.org/T280166) (owner: 10Mstyles) [12:19:03] (03PS2) 10DCausse: helm_diff: supports new and renamed charts [deployment-charts] - 10https://gerrit.wikimedia.org/r/693359 [12:19:05] (03PS1) 10DCausse: Rename chart rdf-streaming-updater as flink-session-cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/693411 [12:19:30] (03CR) 10JMeybohm: [C: 03+2] conftool: Add 4 new docker-registry hosts [puppet] - 10https://gerrit.wikimedia.org/r/693383 (https://phabricator.wikimedia.org/T264209) (owner: 10JMeybohm) [12:20:28] (03CR) 10jerkins-bot: [V: 04-1] Rename chart rdf-streaming-updater as flink-session-cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/693411 (owner: 10DCausse) [12:25:39] 10SRE, 10Gerrit, 10LDAP-Access-Requests: Add dancy to `archiva-deployers` LDAP group - https://phabricator.wikimedia.org/T283347 (10hashar) [12:26:10] (03CR) 10Jbond: [C: 03+2] puppetdb-api: add new discovery service for the puppetdb microservice (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/693401 (https://phabricator.wikimedia.org/T283185) (owner: 10Jbond) [12:26:37] 10SRE, 10Gerrit, 10LDAP-Access-Requests: Add dancy to `archiva-deployers` LDAP group - https://phabricator.wikimedia.org/T283347 (10hashar) [12:29:56] (03PS1) 10Kormat: db-equiad.php: Set pc1010 as pc1 primary. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693413 (https://phabricator.wikimedia.org/T282761) [12:30:19] (03CR) 10Kormat: [C: 04-2] "Not to be submitted until tuesday morning." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693413 (https://phabricator.wikimedia.org/T282761) (owner: 10Kormat) [12:34:22] (03CR) 10Jbond: [C: 03+2] puppetdb-api: add new discovery service for the puppetdb microservice [dns] - 10https://gerrit.wikimedia.org/r/693403 (https://phabricator.wikimedia.org/T283185) (owner: 10Jbond) [12:34:33] (03PS1) 10Jcrespo: prometheus-mysql-exporter: Abort early and capture db integrity issue [puppet] - 10https://gerrit.wikimedia.org/r/693414 [12:39:51] 10SRE, 10vm-requests: codfw: 4 VM request for docker-registry - https://phabricator.wikimedia.org/T283321 (10JMeybohm) 05Open→03Resolved a:03JMeybohm [12:43:22] (03Abandoned) 10DCausse: [WIP] rdf-streaming-updater application mode experiment [deployment-charts] - 10https://gerrit.wikimedia.org/r/686550 (owner: 10DCausse) [12:44:11] (03CR) 10Ayounsi: [C: 03+1] wikidough: update role to work towards anycast support [puppet] - 10https://gerrit.wikimedia.org/r/692368 (https://phabricator.wikimedia.org/T283027) (owner: 10Ssingh) [12:46:15] (03PS2) 10DCausse: Rename chart rdf-streaming-updater as flink-session-cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/693411 [12:46:17] (03PS1) 10DCausse: rdf-streaming-updater: use the flink-session-cluster chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/693416 [12:47:50] (03CR) 10jerkins-bot: [V: 04-1] rdf-streaming-updater: use the flink-session-cluster chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/693416 (owner: 10DCausse) [12:52:21] (03PS14) 10DCausse: rdf-streaming-updater: use session mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/681497 (https://phabricator.wikimedia.org/T280166) (owner: 10Mstyles) [12:52:23] (03PS3) 10DCausse: helm_diff: supports new and renamed charts [deployment-charts] - 10https://gerrit.wikimedia.org/r/693359 [12:52:25] (03PS3) 10DCausse: Rename chart rdf-streaming-updater as flink-session-cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/693411 [12:52:27] (03PS2) 10DCausse: rdf-streaming-updater: use the flink-session-cluster chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/693416 [12:53:04] (03CR) 10Elukey: Add knative serving and net-istio images (033 comments) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [12:53:49] (03PS3) 10Elukey: Add knative serving and net-istio images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) [12:53:56] (03CR) 10jerkins-bot: [V: 04-1] rdf-streaming-updater: use the flink-session-cluster chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/693416 (owner: 10DCausse) [12:57:59] (03PS1) 10Reedy: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693417 [12:58:01] (03CR) 10Reedy: [C: 03+2] Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693417 (owner: 10Reedy) [12:58:50] (03Merged) 10jenkins-bot: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693417 (owner: 10Reedy) [13:00:12] (03PS3) 10Reedy: Change HTTP to HTTPS for concept URIs on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/679327 (https://phabricator.wikimedia.org/T258590) (owner: 10Seddon) [13:03:27] (03CR) 10Elukey: Add knative serving and net-istio images (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [13:08:22] (03CR) 10Marostegui: [C: 03+1] db-equiad.php: Set pc1010 as pc1 primary. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693413 (https://phabricator.wikimedia.org/T282761) (owner: 10Kormat) [13:08:53] (03CR) 10Marostegui: [C: 03+1] "The commit message is wrong btw: s/equiad/eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693413 (https://phabricator.wikimedia.org/T282761) (owner: 10Kormat) [13:10:09] (03PS2) 10Kormat: db-eqiad.php: Set pc1010 as pc1 primary. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693413 (https://phabricator.wikimedia.org/T282761) [13:10:27] (03CR) 10Kormat: [C: 04-2] "> Patch Set 1:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693413 (https://phabricator.wikimedia.org/T282761) (owner: 10Kormat) [13:10:59] (03CR) 10JMeybohm: [C: 03+1] Add istio base images build support [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/688211 (https://phabricator.wikimedia.org/T278192) (owner: 10Elukey) [13:22:41] (03CR) 10Klausman: Add knative serving and net-istio images (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [13:24:15] (03CR) 10JMeybohm: [C: 04-1] Add knative serving and net-istio images (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [13:29:21] (03PS1) 10Alexandros Kosiaris: mathoid: Switch to native prometheus exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/693419 (https://phabricator.wikimedia.org/T283147) [13:30:29] (03CR) 10Elukey: Add knative serving and net-istio images (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [13:30:36] (03CR) 10Alexandros Kosiaris: [C: 03+1] Rename chart rdf-streaming-updater as flink-session-cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/693411 (owner: 10DCausse) [13:32:17] (03CR) 10Alexandros Kosiaris: [C: 04-1] Rename chart rdf-streaming-updater as flink-session-cluster (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/693411 (owner: 10DCausse) [13:32:46] (03CR) 10Alexandros Kosiaris: [C: 03+2] mathoid: Switch to native prometheus exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/693419 (https://phabricator.wikimedia.org/T283147) (owner: 10Alexandros Kosiaris) [13:34:13] 10SRE, 10Analytics, 10Traffic: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10Ottomata) Moving back to incoming for re-triage [13:35:22] (03Merged) 10jenkins-bot: mathoid: Switch to native prometheus exporter [deployment-charts] - 10https://gerrit.wikimedia.org/r/693419 (https://phabricator.wikimedia.org/T283147) (owner: 10Alexandros Kosiaris) [13:36:19] (03PS1) 10Jbond: P:puppetdb::microsite: request certificate from pki [puppet] - 10https://gerrit.wikimedia.org/r/693422 [13:37:42] (03CR) 10jerkins-bot: [V: 04-1] P:puppetdb::microsite: request certificate from pki [puppet] - 10https://gerrit.wikimedia.org/r/693422 (owner: 10Jbond) [13:39:43] (03PS1) 10Jbond: (WIP): clean out certs for old hosts [puppet] - 10https://gerrit.wikimedia.org/r/693424 [13:40:32] (03CR) 10jerkins-bot: [V: 04-1] (WIP): clean out certs for old hosts [puppet] - 10https://gerrit.wikimedia.org/r/693424 (owner: 10Jbond) [13:41:27] (03PS2) 10Jbond: P:puppetdb::microsite: request certificate from pki [puppet] - 10https://gerrit.wikimedia.org/r/693422 [13:43:04] (03CR) 10jerkins-bot: [V: 04-1] P:puppetdb::microsite: request certificate from pki [puppet] - 10https://gerrit.wikimedia.org/r/693422 (owner: 10Jbond) [13:45:37] (03PS1) 10Alexandros Kosiaris: mathoid: Pin the chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/693425 [13:45:55] (03PS3) 10Jbond: P:puppetdb::microsite: request certificate from pki [puppet] - 10https://gerrit.wikimedia.org/r/693422 [13:48:31] (03CR) 10Jbond: [C: 03+2] P:puppetdb::microsite: request certificate from pki [puppet] - 10https://gerrit.wikimedia.org/r/693422 (owner: 10Jbond) [13:48:39] (03PS5) 10Ottomata: Initial debianization and 2.1.0_py3.7-1 release [debs/airflow] (debian) - 10https://gerrit.wikimedia.org/r/693222 (https://phabricator.wikimedia.org/T277012) [13:49:32] (03CR) 10Alexandros Kosiaris: [C: 03+2] mathoid: Pin the chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/693425 (owner: 10Alexandros Kosiaris) [13:51:38] (03Merged) 10jenkins-bot: mathoid: Pin the chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/693425 (owner: 10Alexandros Kosiaris) [13:52:38] (03CR) 10Herron: "> Patch Set 3:" [puppet] - 10https://gerrit.wikimedia.org/r/683044 (https://phabricator.wikimedia.org/T225005) (owner: 10Herron) [13:55:43] (03PS1) 10Jbond: P:puppetdb::microsite: fix config file [puppet] - 10https://gerrit.wikimedia.org/r/693427 [13:57:16] (03CR) 10Jbond: [C: 03+2] P:puppetdb::microsite: fix config file [puppet] - 10https://gerrit.wikimedia.org/r/693427 (owner: 10Jbond) [13:57:20] (03PS1) 10Andrew-WMDE: admin: Add YubiKey SSH key for Andrew Kostka [puppet] - 10https://gerrit.wikimedia.org/r/693428 (https://phabricator.wikimedia.org/T283355) [13:59:47] (03CR) 10Elukey: "> Patch Set 3:" [puppet] - 10https://gerrit.wikimedia.org/r/683044 (https://phabricator.wikimedia.org/T225005) (owner: 10Herron) [14:05:49] (03PS1) 10JMeybohm: docker_registry_ha: Ensure Vary header is send [puppet] - 10https://gerrit.wikimedia.org/r/693430 (https://phabricator.wikimedia.org/T256762) [14:08:30] (03CR) 10Giuseppe Lavagetto: [C: 03+1] helm_diff: supports new and renamed charts [deployment-charts] - 10https://gerrit.wikimedia.org/r/693359 (owner: 10DCausse) [14:08:48] (03CR) 10Ssingh: "> Patch Set 2: Code-Review+1" [puppet] - 10https://gerrit.wikimedia.org/r/692367 (https://phabricator.wikimedia.org/T283027) (owner: 10Ssingh) [14:08:57] (03CR) 10Ssingh: [C: 03+2] bird: add Wikidough's /24 to vips_filter (accept) [puppet] - 10https://gerrit.wikimedia.org/r/692367 (https://phabricator.wikimedia.org/T283027) (owner: 10Ssingh) [14:11:30] (03CR) 10Herron: "(discussed on IRC) we'll plan to deploy this on Monday, with I1b983b046ed5c8fb670d75ad9d39f49d0ab728a1 following shortly after" [deployment-charts] - 10https://gerrit.wikimedia.org/r/683706 (https://phabricator.wikimedia.org/T225005) (owner: 10Herron) [14:12:23] (03PS2) 10Ssingh: acme_chief: add certificates for wikimedia-dns.org [puppet] - 10https://gerrit.wikimedia.org/r/693210 (https://phabricator.wikimedia.org/T252132) [14:12:29] (03CR) 10Herron: "> > I'm thinking we should deploy the updated broker lists and egress rules shortly before this one." [puppet] - 10https://gerrit.wikimedia.org/r/683044 (https://phabricator.wikimedia.org/T225005) (owner: 10Herron) [14:12:34] 10SRE, 10Data-Persistence-Backup, 10netops: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) I've been looking into this issue a litte, and propose to do some tests Monday/Tuesday AM (Europ... [14:12:50] 10SRE, 10Traffic, 10Patch-For-Review: Offer Wikidough as an anycasted service - https://phabricator.wikimedia.org/T283027 (10ssingh) [14:14:00] (03CR) 10Ssingh: [V: 03+1] "PCC SUCCESS (DIFF 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29644/console" [puppet] - 10https://gerrit.wikimedia.org/r/693210 (https://phabricator.wikimedia.org/T252132) (owner: 10Ssingh) [14:14:15] (03PS1) 10Jbond: P:puppetdb:croservice: use server profile for ssl certs [puppet] - 10https://gerrit.wikimedia.org/r/693432 [14:16:31] (03CR) 10Jbond: [C: 03+2] P:puppetdb:croservice: use server profile for ssl certs [puppet] - 10https://gerrit.wikimedia.org/r/693432 (owner: 10Jbond) [14:17:58] (03PS2) 10Cwhite: logstash: bugfix filter to select ssd-allocated indexes [puppet] - 10https://gerrit.wikimedia.org/r/693213 [14:18:01] (03CR) 10Ppchelko: "needs a rebase." [deployment-charts] - 10https://gerrit.wikimedia.org/r/692404 (https://phabricator.wikimedia.org/T260591) (owner: 10Clarakosi) [14:19:39] (03CR) 10Cwhite: [V: 03+1] "Validated that the desired indexes are selected with a dry-run." [puppet] - 10https://gerrit.wikimedia.org/r/693213 (owner: 10Cwhite) [14:21:43] (03CR) 10Vgutierrez: [C: 03+1] acme_chief: add certificates for wikimedia-dns.org [puppet] - 10https://gerrit.wikimedia.org/r/693210 (https://phabricator.wikimedia.org/T252132) (owner: 10Ssingh) [14:26:10] (03CR) 10Ssingh: [V: 03+1 C: 03+2] acme_chief: add certificates for wikimedia-dns.org [puppet] - 10https://gerrit.wikimedia.org/r/693210 (https://phabricator.wikimedia.org/T252132) (owner: 10Ssingh) [14:26:41] (03PS3) 10Jbond: (WIP) create a logout.d profile for managing logout scripts [puppet] - 10https://gerrit.wikimedia.org/r/693149 [14:28:05] (03CR) 10jerkins-bot: [V: 04-1] (WIP) create a logout.d profile for managing logout scripts [puppet] - 10https://gerrit.wikimedia.org/r/693149 (owner: 10Jbond) [14:29:26] (03CR) 10Filippo Giunchedi: [C: 03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/693213 (owner: 10Cwhite) [14:30:39] 10SRE, 10SRE-Access-Requests: Access request for Hal Tredman - https://phabricator.wikimedia.org/T283351 (10Aklapper) Hi, please use the template linked from https://phabricator.wikimedia.org/project/profile/956/ and include wikitech username etc. Thanks. [14:33:25] (03PS4) 10Elukey: Add knative serving and net-istio images [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) [14:33:44] 10SRE, 10netops, 10observability: Create RIPE Atlas probes against our public DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) [14:33:59] (03CR) 10Elukey: Add knative serving and net-istio images (032 comments) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [14:34:16] 10SRE, 10netops, 10observability: Create RIPE Atlas measurements against our public DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) [14:34:25] 10SRE, 10netops, 10observability: Create RIPE Atlas measurements against our authoritative DNS servers; alert on them - https://phabricator.wikimedia.org/T283359 (10CDanis) [14:34:34] (03CR) 10Elukey: Add knative serving and net-istio images (031 comment) [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/692899 (https://phabricator.wikimedia.org/T278194) (owner: 10Elukey) [14:37:02] (03CR) 10Cwhite: [V: 03+1 C: 03+2] logstash: bugfix filter to select ssd-allocated indexes [puppet] - 10https://gerrit.wikimedia.org/r/693213 (owner: 10Cwhite) [14:37:08] (03PS5) 10Clarakosi: api-gateway: Implement new ratelimit configurations from envoy 1.16 [deployment-charts] - 10https://gerrit.wikimedia.org/r/692404 (https://phabricator.wikimedia.org/T260591) [14:40:05] (03PS3) 10Clarakosi: api-gateway: Add default_value to dynamic_metadata if JWT is not set [deployment-charts] - 10https://gerrit.wikimedia.org/r/692714 (https://phabricator.wikimedia.org/T261350) [14:43:23] (03PS3) 10Clarakosi: api-gateway: Replace echoapi with http-https-echo [deployment-charts] - 10https://gerrit.wikimedia.org/r/693008 (https://phabricator.wikimedia.org/T261367) [14:47:10] 10SRE, 10Traffic: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10Vgutierrez) As mentioned on the issue description, debian backported the fix for OpenSSL as it can be seen on a current debian jessie container: ` root@69310d82543d:~# cat /etc/debia... [14:47:18] (03PS1) 10Hnowlan: api-gateway: use newer rate limit image [deployment-charts] - 10https://gerrit.wikimedia.org/r/693435 [14:48:48] (03PS1) 10Herron: logstash: remove references to hosts logstash102[012] [puppet] - 10https://gerrit.wikimedia.org/r/693436 (https://phabricator.wikimedia.org/T281266) [14:50:09] (03PS1) 10Hnowlan: api-gateway: use envoy 1.18.1 everywhere. [deployment-charts] - 10https://gerrit.wikimedia.org/r/693437 [14:50:25] (03CR) 10Giuseppe Lavagetto: [V: 03+1] docker::baseimages: drop alpine support (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/693174 (owner: 10Giuseppe Lavagetto) [14:51:03] (03CR) 10Clarakosi: [C: 03+1] api-gateway: use envoy 1.18.1 everywhere. [deployment-charts] - 10https://gerrit.wikimedia.org/r/693437 (owner: 10Hnowlan) [14:51:51] (03CR) 10Clarakosi: [C: 03+1] api-gateway: use newer rate limit image [deployment-charts] - 10https://gerrit.wikimedia.org/r/693435 (owner: 10Hnowlan) [14:51:56] (03CR) 10Giuseppe Lavagetto: docker::baseimages: add script to build debian-slim (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/693175 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [14:51:58] (03CR) 10Ladsgroup: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/691317 (owner: 10Ladsgroup) [14:52:54] (03CR) 10Hnowlan: [C: 03+2] api-gateway: use envoy 1.18.1 everywhere. [deployment-charts] - 10https://gerrit.wikimedia.org/r/693437 (owner: 10Hnowlan) [14:54:07] (03PS2) 10Giuseppe Lavagetto: docker::baseimages: drop alpine support [puppet] - 10https://gerrit.wikimedia.org/r/693174 [14:54:09] (03PS2) 10Giuseppe Lavagetto: docker::baseimages: add script to build debian-slim [puppet] - 10https://gerrit.wikimedia.org/r/693175 (https://phabricator.wikimedia.org/T281596) [14:54:11] (03CR) 10Hnowlan: [C: 03+2] api-gateway: use newer rate limit image [deployment-charts] - 10https://gerrit.wikimedia.org/r/693435 (owner: 10Hnowlan) [14:55:00] (03CR) 10jerkins-bot: [V: 04-1] docker::baseimages: add script to build debian-slim [puppet] - 10https://gerrit.wikimedia.org/r/693175 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [14:55:11] (03Merged) 10jenkins-bot: api-gateway: use envoy 1.18.1 everywhere. [deployment-charts] - 10https://gerrit.wikimedia.org/r/693437 (owner: 10Hnowlan) [14:56:30] (03PS1) 10Ema: Revert "cache: downgrade Varnish on cp3052 to 5.1.3-1wm15" [puppet] - 10https://gerrit.wikimedia.org/r/693440 (https://phabricator.wikimedia.org/T264398) [14:56:32] (03PS1) 10Ema: Revert "cache: downgrade Varnish on cp3054 to 6.0.0-1wm1" [puppet] - 10https://gerrit.wikimedia.org/r/693441 (https://phabricator.wikimedia.org/T264398) [14:58:02] (03PS2) 10Hnowlan: api-gateway: use newer rate limit image [deployment-charts] - 10https://gerrit.wikimedia.org/r/693435 [14:59:01] (03CR) 10Elukey: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29645/console" [puppet] - 10https://gerrit.wikimedia.org/r/693174 (owner: 10Giuseppe Lavagetto) [14:59:52] (03CR) 10Elukey: [C: 03+1] "LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/693174 (owner: 10Giuseppe Lavagetto) [15:01:39] (03CR) 10Elukey: [V: 03+1] "PCC SUCCESS (DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/29646/console" [puppet] - 10https://gerrit.wikimedia.org/r/693175 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [15:01:51] 10SRE, 10Data-Persistence-Backup, 10netops: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) Thank you very much for your comments. > Due to this I would like to test the performance, usin... [15:02:08] (03CR) 10Hnowlan: api-gateway: use newer rate limit image [deployment-charts] - 10https://gerrit.wikimedia.org/r/693435 (owner: 10Hnowlan) [15:02:11] (03CR) 10Hnowlan: [C: 03+2] api-gateway: use newer rate limit image [deployment-charts] - 10https://gerrit.wikimedia.org/r/693435 (owner: 10Hnowlan) [15:04:18] (03Merged) 10jenkins-bot: api-gateway: use newer rate limit image [deployment-charts] - 10https://gerrit.wikimedia.org/r/693435 (owner: 10Hnowlan) [15:05:21] 10SRE, 10decommission-hardware, 10observability, 10Patch-For-Review: decommission mwlog1001 - https://phabricator.wikimedia.org/T282575 (10herron) a:05herron→03Cmjohnson [15:05:30] 10SRE, 10decommission-hardware, 10observability, 10Patch-For-Review: decommission mwlog1001 - https://phabricator.wikimedia.org/T282575 (10herron) [15:06:21] 10SRE, 10Data-Persistence-Backup, 10netops: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) Ok @jcrespo sounds like a plan. And thanks for the extra info, indeed it does seem to rule out... [15:06:41] (03CR) 10Herron: [C: 03+2] remove icinga[12]001 addresses from firewall rules [homer/public] - 10https://gerrit.wikimedia.org/r/683420 (https://phabricator.wikimedia.org/T279601) (owner: 10Herron) [15:12:18] 10SRE, 10Data-Persistence-Backup, 10netops: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10jcrespo) >>! In T274234#7103617, @cmooney wrote: > In terms of the WAN links I wouldn't rule them out 100... [15:17:43] (03CR) 10Hashar: [C: 03+1] "Common culprit. One might even want to pass the whole os.environ to the test command which is the default behavior of subprocess.run()." [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/692995 (owner: 10Ppchelko) [15:25:04] (03CR) 10Hnowlan: [C: 03+1] api-gateway: Implement new ratelimit configurations from envoy 1.16 [deployment-charts] - 10https://gerrit.wikimedia.org/r/692404 (https://phabricator.wikimedia.org/T260591) (owner: 10Clarakosi) [15:33:32] (03PS15) 10DCausse: rdf-streaming-updater: use session mode [deployment-charts] - 10https://gerrit.wikimedia.org/r/681497 (https://phabricator.wikimedia.org/T280166) (owner: 10Mstyles) [15:33:34] (03PS4) 10DCausse: helm_diff: supports new and renamed charts [deployment-charts] - 10https://gerrit.wikimedia.org/r/693359 [15:33:36] (03PS4) 10DCausse: Rename chart rdf-streaming-updater as flink-session-cluster [deployment-charts] - 10https://gerrit.wikimedia.org/r/693411 [15:33:38] (03PS3) 10DCausse: rdf-streaming-updater: use the flink-session-cluster chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/693416 [15:33:41] (03CR) 10DCausse: Rename chart rdf-streaming-updater as flink-session-cluster (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/693411 (owner: 10DCausse) [15:34:46] 10SRE, 10Traffic, 10netops: Unable to load en.wikipedia.org from 84.19.61.192/26 - https://phabricator.wikimedia.org/T279503 (10cmooney) Is this issue still ongoing? A brief test reveals we appear to be able to route back to this network from the datacentre in question: ` cmooney@lvs3005:~$ mtr --address 9... [15:37:46] (03CR) 10Clarakosi: [C: 03+2] api-gateway: Implement new ratelimit configurations from envoy 1.16 [deployment-charts] - 10https://gerrit.wikimedia.org/r/692404 (https://phabricator.wikimedia.org/T260591) (owner: 10Clarakosi) [15:39:13] (03CR) 10Alexandros Kosiaris: [C: 03+1] docker_registry_ha: Ensure Vary header is send [puppet] - 10https://gerrit.wikimedia.org/r/693430 (https://phabricator.wikimedia.org/T256762) (owner: 10JMeybohm) [15:39:24] (03PS4) 10Clarakosi: api-gateway: Add default_value to metadata if JWT is not set [deployment-charts] - 10https://gerrit.wikimedia.org/r/692714 (https://phabricator.wikimedia.org/T261350) [15:39:53] (03Merged) 10jenkins-bot: api-gateway: Implement new ratelimit configurations from envoy 1.16 [deployment-charts] - 10https://gerrit.wikimedia.org/r/692404 (https://phabricator.wikimedia.org/T260591) (owner: 10Clarakosi) [15:40:48] (03PS1) 10BBlack: check_dotls: bugfix for null output redirect [puppet] - 10https://gerrit.wikimedia.org/r/693443 [15:46:09] (03PS5) 10Clarakosi: api-gateway: Add default_value to metadata if JWT is not set [deployment-charts] - 10https://gerrit.wikimedia.org/r/692714 (https://phabricator.wikimedia.org/T261350) [15:47:55] (03CR) 10Ppchelko: [C: 03+1] api-gateway: Add default_value to metadata if JWT is not set [deployment-charts] - 10https://gerrit.wikimedia.org/r/692714 (https://phabricator.wikimedia.org/T261350) (owner: 10Clarakosi) [15:48:51] (03PS3) 10Ppchelko: Use envoy 1.16 nested json feature for access logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/692709 (https://phabricator.wikimedia.org/T260820) [15:49:20] (03CR) 10Clarakosi: [C: 03+2] api-gateway: Add default_value to metadata if JWT is not set [deployment-charts] - 10https://gerrit.wikimedia.org/r/692714 (https://phabricator.wikimedia.org/T261350) (owner: 10Clarakosi) [15:50:10] (03CR) 10Ladsgroup: "PCC on the whole fleet seems that works fine: https://puppet-compiler.wmflabs.org/compiler1002/751/" [puppet] - 10https://gerrit.wikimedia.org/r/691317 (owner: 10Ladsgroup) [15:52:10] (03Merged) 10jenkins-bot: api-gateway: Add default_value to metadata if JWT is not set [deployment-charts] - 10https://gerrit.wikimedia.org/r/692714 (https://phabricator.wikimedia.org/T261350) (owner: 10Clarakosi) [15:54:17] (03PS4) 10Clarakosi: api-gateway: Replace echoapi with http-https-echo [deployment-charts] - 10https://gerrit.wikimedia.org/r/693008 (https://phabricator.wikimedia.org/T261367) [15:54:39] (03PS4) 10Ppchelko: Use envoy 1.16 nested json feature for access logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/692709 (https://phabricator.wikimedia.org/T260820) [15:57:53] (03CR) 10Clarakosi: [C: 03+2] api-gateway: Replace echoapi with http-https-echo [deployment-charts] - 10https://gerrit.wikimedia.org/r/693008 (https://phabricator.wikimedia.org/T261367) (owner: 10Clarakosi) [15:58:16] (03CR) 10Ppchelko: [C: 03+1] api-gateway: Replace echoapi with http-https-echo [deployment-charts] - 10https://gerrit.wikimedia.org/r/693008 (https://phabricator.wikimedia.org/T261367) (owner: 10Clarakosi) [16:00:13] (03Merged) 10jenkins-bot: api-gateway: Replace echoapi with http-https-echo [deployment-charts] - 10https://gerrit.wikimedia.org/r/693008 (https://phabricator.wikimedia.org/T261367) (owner: 10Clarakosi) [16:00:27] (03CR) 10Clarakosi: [C: 03+1] Use envoy 1.16 nested json feature for access logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/692709 (https://phabricator.wikimedia.org/T260820) (owner: 10Ppchelko) [16:04:07] (03PS1) 10BBlack: dotls haproxy: raise per-listener maxconn [puppet] - 10https://gerrit.wikimedia.org/r/693445 [16:06:23] (03CR) 10BBlack: [C: 03+2] check_dotls: bugfix for null output redirect [puppet] - 10https://gerrit.wikimedia.org/r/693443 (owner: 10BBlack) [16:06:35] (03CR) 10BBlack: [C: 03+2] dotls haproxy: raise per-listener maxconn [puppet] - 10https://gerrit.wikimedia.org/r/693445 (owner: 10BBlack) [16:10:21] (03PS5) 10Ppchelko: Use envoy 1.16 nested json feature for access logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/692709 (https://phabricator.wikimedia.org/T260820) [16:10:56] (03PS3) 10Ppchelko: api-gateway: Make use of host_rewrite_path_regex option [deployment-charts] - 10https://gerrit.wikimedia.org/r/692717 [16:12:34] (03CR) 10Ahmon Dancy: [C: 04-1] "Minor nits." (038 comments) [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 (owner: 10Hashar) [16:23:03] (03CR) 10Ahmon Dancy: [WMF] register our plugins as submodules (031 comment) [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 (owner: 10Hashar) [16:28:18] (03PS1) 10Hashar: Download upstream war with Maven [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/693450 [16:32:32] (03CR) 10Hashar: "So this is the new magic. We no more build our own Gerrit war file and instead consume the upstream ones. The links are at https://www.ge" [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/693450 (owner: 10Hashar) [16:32:39] (03PS3) 10Ppchelko: api-gateway: use virtual clusters to split r ans w traffic [deployment-charts] - 10https://gerrit.wikimedia.org/r/693017 [16:32:45] (03CR) 10Elukey: [V: 03+1 C: 03+1] "LGTM modulo Jenkins not happy for the Bug: line :)" [puppet] - 10https://gerrit.wikimedia.org/r/693175 (https://phabricator.wikimedia.org/T281596) (owner: 10Giuseppe Lavagetto) [16:33:15] (03PS1) 10Hnowlan: api-gateway: Set staging anon ratelimit to a very low value [deployment-charts] - 10https://gerrit.wikimedia.org/r/693453 [16:33:20] 10SRE, 10ops-eqiad, 10DC-Ops, 10decommission-hardware: decommission db1087.eqiad.wmnet - https://phabricator.wikimedia.org/T282093 (10wiki_willy) a:05wiki_willy→03Cmjohnson [16:34:09] (03CR) 10Ppchelko: [C: 03+1] api-gateway: Set staging anon ratelimit to a very low value [deployment-charts] - 10https://gerrit.wikimedia.org/r/693453 (owner: 10Hnowlan) [16:34:34] (03CR) 10Clarakosi: [C: 03+1] api-gateway: Set staging anon ratelimit to a very low value [deployment-charts] - 10https://gerrit.wikimedia.org/r/693453 (owner: 10Hnowlan) [16:36:33] (03CR) 10Clarakosi: [C: 03+1] Use envoy 1.16 nested json feature for access logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/692709 (https://phabricator.wikimedia.org/T260820) (owner: 10Ppchelko) [16:37:32] (03CR) 10Hnowlan: [C: 03+2] api-gateway: Set staging anon ratelimit to a very low value [deployment-charts] - 10https://gerrit.wikimedia.org/r/693453 (owner: 10Hnowlan) [16:39:43] (03Merged) 10jenkins-bot: api-gateway: Set staging anon ratelimit to a very low value [deployment-charts] - 10https://gerrit.wikimedia.org/r/693453 (owner: 10Hnowlan) [16:41:00] (03PS1) 10Majavah: toolforge: Assign more resources to kubernetes ingress [puppet] - 10https://gerrit.wikimedia.org/r/693454 [16:45:53] (03CR) 10Bstorm: [C: 03+2] toolforge: Assign more resources to kubernetes ingress [puppet] - 10https://gerrit.wikimedia.org/r/693454 (owner: 10Majavah) [16:53:39] (03PS1) 10Majavah: toolforge: Fix memory requests for kubernetes ingress [puppet] - 10https://gerrit.wikimedia.org/r/693459 [16:54:50] (03CR) 10Bstorm: [C: 03+2] "Didn't even see it last time. Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/693459 (owner: 10Majavah) [17:01:06] (03CR) 10Hashar: "Even after reading the whole script before sending, I still managed to leak some mistakes. Thank you!" (038 comments) [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 (owner: 10Hashar) [17:01:13] (03PS14) 10Hashar: [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 [17:01:16] (03PS2) 10Ppchelko: Api-gateway: enable x-ratelimit-* headers [deployment-charts] - 10https://gerrit.wikimedia.org/r/692946 (https://phabricator.wikimedia.org/T246278) [17:02:48] (03CR) 10Clarakosi: [C: 03+1] Api-gateway: enable x-ratelimit-* headers [deployment-charts] - 10https://gerrit.wikimedia.org/r/692946 (https://phabricator.wikimedia.org/T246278) (owner: 10Ppchelko) [17:04:33] (03CR) 10Hnowlan: [C: 03+2] Api-gateway: enable x-ratelimit-* headers [deployment-charts] - 10https://gerrit.wikimedia.org/r/692946 (https://phabricator.wikimedia.org/T246278) (owner: 10Ppchelko) [17:06:40] (03Merged) 10jenkins-bot: Api-gateway: enable x-ratelimit-* headers [deployment-charts] - 10https://gerrit.wikimedia.org/r/692946 (https://phabricator.wikimedia.org/T246278) (owner: 10Ppchelko) [17:08:41] (03CR) 10Ahmon Dancy: [WMF] script to build our plugins (031 comment) [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 (owner: 10Hashar) [17:14:34] (03PS1) 10TrainBranchBot: Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693468 [17:14:36] (03CR) 10TrainBranchBot: [C: 03+2] Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693468 (owner: 10TrainBranchBot) [17:14:59] (03PS15) 10Hashar: [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 [17:15:12] (03CR) 10Hashar: [WMF] script to build our plugins (031 comment) [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 (owner: 10Hashar) [17:16:04] (03Merged) 10jenkins-bot: Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693468 (owner: 10TrainBranchBot) [17:16:29] (03CR) 10Ahmon Dancy: [C: 03+1] [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684411 (owner: 10Hashar) [17:23:07] (03PS6) 10Ppchelko: Use envoy 1.16 nested json feature for access logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/692709 (https://phabricator.wikimedia.org/T260820) [17:24:22] (03CR) 10Ppchelko: [C: 03+2] Use envoy 1.16 nested json feature for access logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/692709 (https://phabricator.wikimedia.org/T260820) (owner: 10Ppchelko) [17:26:38] (03Merged) 10jenkins-bot: Use envoy 1.16 nested json feature for access logging [deployment-charts] - 10https://gerrit.wikimedia.org/r/692709 (https://phabricator.wikimedia.org/T260820) (owner: 10Ppchelko) [17:26:45] (03PS1) 10Legoktm: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693469 [17:26:47] (03CR) 10Legoktm: [C: 03+2] Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693469 (owner: 10Legoktm) [17:27:29] (03Merged) 10jenkins-bot: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/693469 (owner: 10Legoktm) [17:27:42] (03PS1) 10TrainBranchBot: Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693470 [17:27:44] (03CR) 10TrainBranchBot: [C: 03+2] Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693470 (owner: 10TrainBranchBot) [17:28:24] (03Merged) 10jenkins-bot: Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693470 (owner: 10TrainBranchBot) [17:29:24] (03PS6) 10Ottomata: Initial debianization and 2.1.0_py3.7-1 release [debs/airflow] (debian) - 10https://gerrit.wikimedia.org/r/693222 (https://phabricator.wikimedia.org/T277012) [17:29:36] (03PS7) 10Ottomata: Initial debianization and 2.1.0-py3.7-1 release [debs/airflow] (debian) - 10https://gerrit.wikimedia.org/r/693222 (https://phabricator.wikimedia.org/T277012) [17:34:49] (03PS1) 10TrainBranchBot: Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693473 [17:34:51] (03CR) 10TrainBranchBot: [C: 03+2] Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693473 (owner: 10TrainBranchBot) [17:35:35] (03Merged) 10jenkins-bot: Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693473 (owner: 10TrainBranchBot) [17:41:54] (03PS1) 10Effie Mouzeli: WIP: add notls support for external addresses to memcached [puppet] - 10https://gerrit.wikimedia.org/r/693474 (https://phabricator.wikimedia.org/T271967) [17:42:26] (03CR) 10jerkins-bot: [V: 04-1] WIP: add notls support for external addresses to memcached [puppet] - 10https://gerrit.wikimedia.org/r/693474 (https://phabricator.wikimedia.org/T271967) (owner: 10Effie Mouzeli) [17:42:29] 10SRE, 10Analytics-Radar, 10LDAP-Access-Requests, 10SRE-Access-Requests: Account setup issues for jmixter-ctr - https://phabricator.wikimedia.org/T283250 (10jmixter) @elukey thank you so much for the help and clarification. My WikiTech shell name is jmixter. [17:44:54] 10SRE, 10Cloud-VPS (Debian Jessie Deprecation), 10cloud-services-team (Kanban): Migrate labstore1004/labstore1005 to Stretch/Buster - https://phabricator.wikimedia.org/T224582 (10Bstorm) [17:46:05] 10SRE, 10ops-eqiad, 10Analytics-Radar: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10Cmjohnson) @elukey sure I can move one of them to A2. Rack A7 is still full [17:46:53] 10SRE, 10ops-eqiad, 10Analytics-Radar: Try to move some new analytics worker nodes to different racks - https://phabricator.wikimedia.org/T276239 (10elukey) Yes perfect! Anytime is good, those nodes are not in service. [17:47:54] (03PS4) 10Ppchelko: api-gateway: Make use of host_rewrite_path_regex option [deployment-charts] - 10https://gerrit.wikimedia.org/r/692717 [17:49:12] (03PS5) 10Ppchelko: api-gateway: Make use of host_rewrite_path_regex option [deployment-charts] - 10https://gerrit.wikimedia.org/r/692717 [17:53:48] (03CR) 10Clarakosi: "Minor nit but otherwise lgtm" (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/692717 (owner: 10Ppchelko) [18:00:30] (03PS6) 10Ppchelko: api-gateway: Make use of host_rewrite_path_regex option [deployment-charts] - 10https://gerrit.wikimedia.org/r/692717 [18:01:58] (03CR) 10Clarakosi: [C: 03+1] api-gateway: Make use of host_rewrite_path_regex option [deployment-charts] - 10https://gerrit.wikimedia.org/r/692717 (owner: 10Ppchelko) [18:02:07] (03CR) 10Ppchelko: api-gateway: Make use of host_rewrite_path_regex option (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/692717 (owner: 10Ppchelko) [18:02:16] (03CR) 10Ppchelko: [C: 03+2] api-gateway: Make use of host_rewrite_path_regex option [deployment-charts] - 10https://gerrit.wikimedia.org/r/692717 (owner: 10Ppchelko) [18:04:26] (03Merged) 10jenkins-bot: api-gateway: Make use of host_rewrite_path_regex option [deployment-charts] - 10https://gerrit.wikimedia.org/r/692717 (owner: 10Ppchelko) [18:31:52] (03PS1) 10TrainBranchBot: Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693480 [18:31:54] (03CR) 10TrainBranchBot: [C: 03+2] Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693480 (owner: 10TrainBranchBot) [18:32:22] (03PS4) 10Ppchelko: api-gateway: use virtual clusters to split r ans w traffic [deployment-charts] - 10https://gerrit.wikimedia.org/r/693017 [18:34:00] (03Merged) 10jenkins-bot: Update train-versions.json [mediawiki-config] (sandbox/dancy) - 10https://gerrit.wikimedia.org/r/693480 (owner: 10TrainBranchBot) [18:52:28] 10SRE, 10wikimedia-irc-freenode: Move SRE-related IRC channels to Libera - https://phabricator.wikimedia.org/T283230 (10Dzahn) [19:26:16] 10SRE, 10Wikimedia-Mailing-lists, 10User-Ladsgroup: Upgrade mailing lists from mailman2 to 3 in batches - https://phabricator.wikimedia.org/T280322 (10Ladsgroup) The final list of not migrated lists yet: [] Abstract-Wikipedia [] affcom-members [] ai [] allhands-travel [] ambassadors [] arbcom-ko [] arbcom-l... [19:28:13] (03PS1) 10Bstorm: toolforge bastion: do not run wheelofmisfortune on buster yet [puppet] - 10https://gerrit.wikimedia.org/r/693485 (https://phabricator.wikimedia.org/T282949) [19:29:24] 10SRE, 10Analytics-Radar, 10LDAP-Access-Requests, 10SRE-Access-Requests: Account setup issues for jmixter-ctr - https://phabricator.wikimedia.org/T283250 (10Dzahn) > I tried to submit a Phabricator question but received this notification: > 'This board is protected. Only autoconfirmed users can participate... [19:35:44] (03PS1) 10Bstorm: wheel_of_misfortune: include the hostname in the email [puppet] - 10https://gerrit.wikimedia.org/r/693487 (https://phabricator.wikimedia.org/T282949) [19:40:51] (03PS2) 10Bstorm: wheel_of_misfortune: include the hostname in the email [puppet] - 10https://gerrit.wikimedia.org/r/693487 (https://phabricator.wikimedia.org/T282949) [19:46:39] (03CR) 10Bstorm: "https://puppet-compiler.wmflabs.org/compiler1001/29647/" [puppet] - 10https://gerrit.wikimedia.org/r/693485 (https://phabricator.wikimedia.org/T282949) (owner: 10Bstorm) [19:49:00] (03CR) 10Bstorm: [C: 03+2] "I'm going to merge this so we can verify where I think the bad behavior is." [puppet] - 10https://gerrit.wikimedia.org/r/693487 (https://phabricator.wikimedia.org/T282949) (owner: 10Bstorm) [20:02:08] 10SRE, 10ops-codfw, 10ops-eqiad, 10DC-Ops: Dc-Ops Commands for Cumin - https://phabricator.wikimedia.org/T279721 (10wiki_willy) [20:16:54] (03PS1) 10Bstorm: wikireplicas: raise limits on memory alerts [puppet] - 10https://gerrit.wikimedia.org/r/693492 (https://phabricator.wikimedia.org/T281732) [20:20:44] (03CR) 10Bstorm: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1002/29649/clouddb1019.eqiad.wmnet/index.html That looks good. I'll merge this for now to shut" [puppet] - 10https://gerrit.wikimedia.org/r/693492 (https://phabricator.wikimedia.org/T281732) (owner: 10Bstorm) [20:58:39] (03PS1) 10Hashar: Review access change [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/693258 [21:02:11] (03PS2) 10Hashar: Remove Gerrit Managers privileges [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/693258 [21:04:02] (03CR) 10Hashar: "Gerrit Managers is a wide range of people, that is merely to create repo and manages rights. They don't need https://gerrit.wikimedia.org/" [software/gerrit] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/693258 (owner: 10Hashar) [21:20:22] (03PS1) 10Dzahn: hieradata/common/service: fix syntax error in Icinga config [puppet] - 10https://gerrit.wikimedia.org/r/693496 [21:20:51] (03CR) 10jerkins-bot: [V: 04-1] hieradata/common/service: fix syntax error in Icinga config [puppet] - 10https://gerrit.wikimedia.org/r/693496 (owner: 10Dzahn) [21:23:42] (03PS2) 10Dzahn: hieradata/common/service: fix syntax error in Icinga config [puppet] - 10https://gerrit.wikimedia.org/r/693496 [21:29:19] (03CR) 10Dzahn: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1001/29651/alert1001.wikimedia.org/index.html" [puppet] - 10https://gerrit.wikimedia.org/r/693496 (owner: 10Dzahn) [21:30:35] (03CR) 10Dzahn: "as chris pointed out we didn't notice this for quite some time despite alerting, I think that had to do with the IRC network move" [puppet] - 10https://gerrit.wikimedia.org/r/693496 (owner: 10Dzahn) [21:32:09] (03CR) 10Dzahn: "https://gerrit.wikimedia.org/r/c/operations/puppet/+/693496" [puppet] - 10https://gerrit.wikimedia.org/r/693401 (https://phabricator.wikimedia.org/T283185) (owner: 10Jbond) [21:33:29] (03CR) 10Dzahn: "Total Warnings: 0" [puppet] - 10https://gerrit.wikimedia.org/r/693496 (owner: 10Dzahn) [21:42:00] (03CR) 10Dzahn: "21:39 < icinga-wm> RECOVERY - Check correctness of the icinga configuration on alert1001 is OK: Icinga configuration is correct https://wi" [puppet] - 10https://gerrit.wikimedia.org/r/693496 (owner: 10Dzahn) [22:28:08] (03PS1) 10Legoktm: mailman3: Fix reading the exclusion list [puppet] - 10https://gerrit.wikimedia.org/r/693499 [22:30:15] (03PS4) 10Legoktm: lists: Redirect domain root to Postorius if mailman3 enabled [puppet] - 10https://gerrit.wikimedia.org/r/681247 [22:38:13] (03CR) 10Legoktm: [C: 03+2] mailman3: Fix reading the exclusion list [puppet] - 10https://gerrit.wikimedia.org/r/693499 (owner: 10Legoktm) [22:38:31] (03CR) 10Legoktm: [C: 03+2] lists: Redirect domain root to Postorius if mailman3 enabled [puppet] - 10https://gerrit.wikimedia.org/r/681247 (owner: 10Legoktm) [22:43:43] (03CR) 10Cwhite: [C: 03+1] logstash: remove references to hosts logstash102[012] [puppet] - 10https://gerrit.wikimedia.org/r/693436 (https://phabricator.wikimedia.org/T281266) (owner: 10Herron) [22:49:03] 10SRE, 10Wikimedia-Mailing-lists: Publish statistics about number of held messages per mailing list (Jan 2021) - https://phabricator.wikimedia.org/T270977 (10Legoktm) @Aklapper I don't know why (besides blaming Mailman2) but the numbers I posted above are just wrong. I logged into the mailman@ list just now an... [22:53:29] (03PS1) 10Bstorm: buster: generate a release for buster and add some files [debs/nfsd-ldap] - 10https://gerrit.wikimedia.org/r/693500 (https://phabricator.wikimedia.org/T283385) [22:55:39] (03CR) 10Bstorm: "Note, the package is already live. This is mostly to record the changelog as well as to make this slightly more maintainable in the future" [debs/nfsd-ldap] - 10https://gerrit.wikimedia.org/r/693500 (https://phabricator.wikimedia.org/T283385) (owner: 10Bstorm)