[00:02:40] hmm okay. I guess I can let you know if becomes more rampant. It is definitely ongoing, got another email 2 minutes ago [00:04:03] But to stress, I don't really know beyond the graphs don't obviously flatline [00:04:03] this is still a very small fraction of our traffic, I'm sure (or else I'd gotten a lot more emails), so maybe that's why the graphs aren't showing much. All I know is it isn't normal [00:04:25] and I'm not really a person who knows about this area [00:20:42] (03PS1) 10Brian Wolff: Adjust CSP to allow inline CSS & playing MP4 videos on jenkins [puppet] - 10https://gerrit.wikimedia.org/r/582604 (https://phabricator.wikimedia.org/T245658) [01:00:58] PROBLEM - Old JVM GC check - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is CRITICAL: 125.1 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [01:03:26] PROBLEM - Old JVM GC check - cloudelastic1001-cloudelastic-chi-eqiad on cloudelastic1001 is CRITICAL: 103.7 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [01:25:08] PROBLEM - Old JVM GC check - cloudelastic1003-cloudelastic-chi-eqiad on cloudelastic1003 is CRITICAL: 115.9 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [01:36:32] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp1081 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [01:48:50] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp1081 is OK: HTTP OK: HTTP/1.0 200 OK - 22312 bytes in 0.006 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [01:54:04] PROBLEM - Old JVM GC check - cloudelastic1004-cloudelastic-chi-eqiad on cloudelastic1004 is CRITICAL: 104.7 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [02:21:44] (03PS1) 10Jhedden: shinken: remove puppet agent checks [puppet] - 10https://gerrit.wikimedia.org/r/582612 (https://phabricator.wikimedia.org/T210993) [02:24:39] (03CR) 10Jhedden: "This data was removed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/549241" [puppet] - 10https://gerrit.wikimedia.org/r/582612 (https://phabricator.wikimedia.org/T210993) (owner: 10Jhedden) [02:32:00] (03PS1) 10DannyS712: Don't try to grant `oathauth-enable` to `*` (part 1) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582613 (https://phabricator.wikimedia.org/T248282) [02:32:22] (03PS2) 10DannyS712: Don't try to grant `oathauth-enable` to `*` (part 1) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582613 (https://phabricator.wikimedia.org/T248282) [02:34:50] (03PS1) 10DannyS712: Don't try to grant `oathauth-enable` to `*` (part 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582615 (https://phabricator.wikimedia.org/T248282) [02:35:16] (03PS2) 10DannyS712: Don't try to grant `oathauth-enable` to `*` (part 2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582615 (https://phabricator.wikimedia.org/T248282) [03:47:06] (03PS1) 10Brian Wolff: Add *.wikidata.beta.wmflabs.org to CSP list for beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582633 [03:50:11] (03CR) 10DannyS712: [C: 03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582633 (owner: 10Brian Wolff) [03:54:24] So for me, command line operations to gerrit (e.g. Running git pull or git push) are really super slow [04:00:54] PROBLEM - Old JVM GC check - cloudelastic1004-cloudelastic-chi-eqiad on cloudelastic1004 is CRITICAL: 100.3 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [04:09:12] RECOVERY - Old JVM GC check - cloudelastic1004-cloudelastic-chi-eqiad on cloudelastic1004 is OK: (C)100 gt (W)80 gt 74.24 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [04:19:48] RECOVERY - Old JVM GC check - cloudelastic1003-cloudelastic-chi-eqiad on cloudelastic1003 is OK: (C)100 gt (W)80 gt 74.24 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [04:48:00] RECOVERY - Old JVM GC check - cloudelastic1001-cloudelastic-chi-eqiad on cloudelastic1001 is OK: (C)100 gt (W)80 gt 77.29 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [05:31:18] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp1081 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [05:41:36] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp1081 is OK: HTTP OK: HTTP/1.0 200 OK - 22316 bytes in 0.006 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [06:04:41] (03Abandoned) 10KartikMistry: apertium-apy: New upstream release [debs/contenttranslation/apertium-apy] - 10https://gerrit.wikimedia.org/r/554849 (https://phabricator.wikimedia.org/T236080) (owner: 10KartikMistry) [06:06:57] (03PS1) 10KartikMistry: apertium-is-sv: Fix FTBFS with apertium 3.6 [debs/contenttranslation/apertium-is-sv] - 10https://gerrit.wikimedia.org/r/582640 (https://phabricator.wikimedia.org/T247585) [06:38:43] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/582440 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [06:41:21] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/582072 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [06:46:25] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/582071 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [06:47:23] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/582068 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [06:48:43] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/582066 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [06:50:22] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582064 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [06:51:36] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/582070 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [06:54:57] (03CR) 10Muehlenhoff: bastionhost: replace auth1001 with auth1002 in pam-sshd config (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582147 (https://phabricator.wikimedia.org/T234909) (owner: 10Dzahn) [06:59:22] (03PS1) 10KartikMistry: apertium-afr-nld: Fix FTBFS with apertium 3.6 + 0.3.0 release [debs/contenttranslation/apertium-af-nl] - 10https://gerrit.wikimedia.org/r/582641 (https://phabricator.wikimedia.org/T248293) [07:02:09] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) (owner: 10Jbond) [07:02:54] PROBLEM - Old JVM GC check - cloudelastic1001-cloudelastic-chi-eqiad on cloudelastic1001 is CRITICAL: 101.7 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [07:23:38] RECOVERY - Old JVM GC check - cloudelastic1001-cloudelastic-chi-eqiad on cloudelastic1001 is OK: (C)100 gt (W)80 gt 78.31 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [07:25:37] (03CR) 10Elukey: [C: 03+2] statistics::sites::analytics: remove reference to statistics-web-users [puppet] - 10https://gerrit.wikimedia.org/r/582440 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [07:26:07] (03CR) 10Elukey: [C: 03+2] admin: add more documentation to analytics posix groups (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582064 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [07:26:20] (03CR) 10Elukey: [C: 03+2] admin: deprecate piwik-roots [puppet] - 10https://gerrit.wikimedia.org/r/582066 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [07:26:28] (03PS3) 10Elukey: admin: deprecate piwik-roots [puppet] - 10https://gerrit.wikimedia.org/r/582066 (https://phabricator.wikimedia.org/T246578) [07:30:11] (03PS3) 10Elukey: admin: flag notebook-roots as deprecated [puppet] - 10https://gerrit.wikimedia.org/r/582068 (https://phabricator.wikimedia.org/T246578) [07:33:11] (03CR) 10Elukey: [C: 03+2] admin: flag notebook-roots as deprecated [puppet] - 10https://gerrit.wikimedia.org/r/582068 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [07:33:27] (03PS4) 10Elukey: admin: refactor eventlogging-related groups [puppet] - 10https://gerrit.wikimedia.org/r/582070 (https://phabricator.wikimedia.org/T246578) [07:40:24] (03CR) 10Elukey: [C: 03+2] admin: refactor eventlogging-related groups [puppet] - 10https://gerrit.wikimedia.org/r/582070 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [07:40:40] (03PS4) 10Elukey: admin: use the *analytics_admins_members placeholder when possible [puppet] - 10https://gerrit.wikimedia.org/r/582071 (https://phabricator.wikimedia.org/T246578) [07:44:44] (03CR) 10Elukey: [C: 03+2] admin: use the *analytics_admins_members placeholder when possible [puppet] - 10https://gerrit.wikimedia.org/r/582071 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [07:45:01] (03PS4) 10Elukey: admin: deprecate aqs-users [puppet] - 10https://gerrit.wikimedia.org/r/582072 (https://phabricator.wikimedia.org/T246578) [07:45:58] RECOVERY - Old JVM GC check - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is OK: (C)100 gt (W)80 gt 75.25 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad [07:46:30] !log Stop MySQL on db1077 (non used) for 10.4 upgrade and gtid_domain_id on multisource T149418 [07:46:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:46:37] T149418: Deploy gtid_domain_id flag in our mysql hosts - https://phabricator.wikimedia.org/T149418 [07:50:08] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 271, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [07:50:20] RECOVERY - OSPF status on cr1-codfw is OK: OSPFv2: 6/6 UP : OSPFv3: 6/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [07:50:50] (03PS4) 10Muehlenhoff: New ferm sub profile for public mysql proxies [puppet] - 10https://gerrit.wikimedia.org/r/582020 [07:56:22] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 269, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [07:56:35] PROBLEM - OSPF status on cr1-codfw is CRITICAL: OSPFv2: 5/6 UP : OSPFv3: 5/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [07:56:41] (03PS1) 10Muehlenhoff: Switch dbproxy1019 to public Ferm profile [puppet] - 10https://gerrit.wikimedia.org/r/582643 [07:58:24] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 271, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [07:58:38] RECOVERY - OSPF status on cr1-codfw is OK: OSPFv2: 6/6 UP : OSPFv3: 6/6 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [08:02:22] (03CR) 10Elukey: [C: 03+2] admin: deprecate aqs-users [puppet] - 10https://gerrit.wikimedia.org/r/582072 (https://phabricator.wikimedia.org/T246578) (owner: 10Elukey) [08:02:51] (03CR) 10Muehlenhoff: "I think PS4 should address all earlier comments." [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [08:03:54] !log installing python-cryptography bug fix updates from Stretch point release [08:03:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:04:05] !log Stop m1 and m2 on db1117 to transfer them to db1077 - this will trigger dbproxies IRC alert [08:04:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:04:34] PROBLEM - Check systemd state on sodium is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [08:04:38] (03CR) 10Jcrespo: [C: 03+1] "+1 But deployment should be done with puppet disabled, in a progressive way" [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [08:05:29] (03CR) 10Jcrespo: [C: 03+1] "> Patch Set 4: Code-Review+1" [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [08:07:06] !log Start m1 and m2 on db1117 [08:07:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:08:43] (03CR) 10Marostegui: [C: 03+1] "Thanks for working on this. We can deploy this to dbproxy1019 for instance, and manually test everything (including access to labs hosts)" [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [08:14:58] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [08:17:02] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [08:28:57] (03PS1) 10Elukey: Improve the old_jvm_gc_checks's elasticsearch alarm [puppet] - 10https://gerrit.wikimedia.org/r/582772 [08:41:29] (03PS1) 10Marostegui: install_server: Do not reimage db1107 [puppet] - 10https://gerrit.wikimedia.org/r/582773 [08:43:27] (03CR) 10Marostegui: [C: 03+2] install_server: Do not reimage db1107 [puppet] - 10https://gerrit.wikimedia.org/r/582773 (owner: 10Marostegui) [08:46:18] (03CR) 10Gehel: [C: 03+1] "LGTM, minor comment inline" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582772 (owner: 10Elukey) [08:46:59] (03CR) 10Marostegui: [C: 03+1] "Let's coordinate before merging. Probably tomorrow?" [puppet] - 10https://gerrit.wikimedia.org/r/582643 (owner: 10Muehlenhoff) [09:02:45] (03PS3) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [09:10:39] (03CR) 10Elukey: [C: 03+2] "Had a chat with Gehel, we are going to merge as it is and then we'll review later in case." [puppet] - 10https://gerrit.wikimedia.org/r/582772 (owner: 10Elukey) [09:12:57] (03PS4) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [09:14:41] (03PS5) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [09:22:03] PROBLEM - check_trafficserver_log_fifo_analytics_tls on cp3050 is CRITICAL: CRITICAL: /srv/trafficserver/tls/var/log/analytics.pipe - fifo-log-demux not reading from pipe https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [09:22:53] cp3050 is me, please ignore ^ [09:24:19] RECOVERY - check_trafficserver_log_fifo_analytics_tls on cp3050 is OK: OK: TS_MAIN writing to and fifo-log-demux reading from /srv/trafficserver/tls/var/log/analytics.pipe https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [09:28:05] PROBLEM - check_trafficserver_log_fifo_analytics_tls on cp3050 is CRITICAL: CRITICAL: /srv/trafficserver/tls/var/log/analytics.pipe - fifo-log-demux not reading from pipe https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [09:31:58] (03PS6) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [09:32:01] ok [09:32:42] (03CR) 10jerkins-bot: [V: 04-1] Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) (owner: 10Ema) [09:35:10] (03PS7) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [09:40:35] RECOVERY - check_trafficserver_log_fifo_analytics_tls on cp3050 is OK: OK: TS_MAIN writing to and fifo-log-demux reading from /srv/trafficserver/tls/var/log/analytics.pipe https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [09:41:44] (03PS8) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [09:46:42] 10Operations, 10Commons, 10SRE-swift-storage, 10User-fgiunchedi: Big number of uploads from DPLA bot - https://phabricator.wikimedia.org/T248151 (10fgiunchedi) [09:55:05] PROBLEM - rsyslog in eqiad is failing to deliver messages on icinga1001 is CRITICAL: action=fwd_centrallog1001.eqiad.wmnet:6514 https://wikitech.wikimedia.org/wiki/Rsyslog https://grafana.wikimedia.org/d/000000596/rsyslog?var-datasource=eqiad+prometheus/ops [09:56:25] RECOVERY - rsyslog in eqiad is failing to deliver messages on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Rsyslog https://grafana.wikimedia.org/d/000000596/rsyslog?var-datasource=eqiad+prometheus/ops [09:59:48] (03PS1) 10Elukey: Add wdqs blazegraph's alerts only for non-test clusters [puppet] - 10https://gerrit.wikimedia.org/r/582774 [10:04:39] ACKNOWLEDGEMENT - WDQS high update lag on wdqs1010 is CRITICAL: 1.575e+06 ge 3600 Gehel Catching up on lag after data reimport https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook%23Update_lag https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [10:10:18] (03CR) 10Filippo Giunchedi: [C: 03+2] prometheus: add mediawiki recording rules [puppet] - 10https://gerrit.wikimedia.org/r/580888 (owner: 10Filippo Giunchedi) [10:10:20] (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler1003/21532/" [puppet] - 10https://gerrit.wikimedia.org/r/582774 (owner: 10Elukey) [10:11:31] (03PS1) 10Marostegui: install_server: Reimage db1077 to buster [puppet] - 10https://gerrit.wikimedia.org/r/582775 (https://phabricator.wikimedia.org/T149418) [10:17:54] (03CR) 10Marostegui: [C: 03+2] install_server: Reimage db1077 to buster [puppet] - 10https://gerrit.wikimedia.org/r/582775 (https://phabricator.wikimedia.org/T149418) (owner: 10Marostegui) [10:21:40] 10Operations, 10Analytics, 10Product-Analytics, 10SRE-Access-Requests: Hive access - https://phabricator.wikimedia.org/T248097 (10Volans) p:05Triage→03Medium [10:22:35] (03CR) 10Jbond: New ferm sub profile for public mysql proxies (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [10:24:50] (03PS1) 10Marostegui: install_server: Allow reimage db1077 [puppet] - 10https://gerrit.wikimedia.org/r/582776 (https://phabricator.wikimedia.org/T149418) [10:30:05] jan_drewniak: That opportune time is upon us again. Time for a Wikimedia Portals Update deploy. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200323T1030). [10:32:28] (03PS1) 10Giuseppe Lavagetto: Make configuration of envoy a ConfigMap [deployment-charts] - 10https://gerrit.wikimedia.org/r/582777 (https://phabricator.wikimedia.org/T244843) [10:35:58] (03CR) 10Jbond: "looks good, minor nit" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/582558 (owner: 10Elukey) [10:36:01] (03PS1) 10Filippo Giunchedi: kibana: assign httpd_extra_modules in all cases [puppet] - 10https://gerrit.wikimedia.org/r/582778 [10:36:43] (03PS1) 10Filippo Giunchedi: logstash: don't require full logstash heap on plugin installation [puppet] - 10https://gerrit.wikimedia.org/r/582779 [10:37:00] !log switch idp1001 to tlsproxy::envoy profile [10:37:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:37:49] (03CR) 10Jbond: [C: 03+2] profile::idp: update profile to use tlsproxy::envoy [puppet] - 10https://gerrit.wikimedia.org/r/581996 (https://phabricator.wikimedia.org/T240941) (owner: 10Jbond) [10:39:51] (03CR) 10Muehlenhoff: New ferm sub profile for public mysql proxies (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [10:40:03] (03PS5) 10Muehlenhoff: New ferm sub profile for public mysql proxies [puppet] - 10https://gerrit.wikimedia.org/r/582020 [10:44:37] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=idp site=eqiad https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [10:46:18] (03CR) 10Gehel: Add wdqs blazegraph's alerts only for non-test clusters (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582774 (owner: 10Elukey) [10:46:29] (03PS9) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [10:47:28] (03CR) 10Elukey: systemd::timer::job: add a parameter to change the systemd slice (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/582558 (owner: 10Elukey) [10:48:08] (03CR) 10Elukey: Add wdqs blazegraph's alerts only for non-test clusters (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582774 (owner: 10Elukey) [10:50:26] (03PS2) 10Elukey: Add wdqs blazegraph's alerts only for non-test clusters [puppet] - 10https://gerrit.wikimedia.org/r/582774 [10:50:45] (03PS1) 10Jbond: update port to port 8080 [puppet] - 10https://gerrit.wikimedia.org/r/582783 [10:51:11] (03PS1) 10Gehel: wdqs: added monitoring to data-reload cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 [10:51:56] (03PS2) 10Jbond: role idp: update port to port 8080 [puppet] - 10https://gerrit.wikimedia.org/r/582783 [10:52:41] (03PS2) 10Gehel: wdqs: added monitoring to data-reload cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 [10:53:30] (03CR) 10Jbond: New ferm sub profile for public mysql proxies (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [10:53:32] (03CR) 10Giuseppe Lavagetto: "Apart from john's comments, with I agree with, LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/582558 (owner: 10Elukey) [10:53:39] (03CR) 10Jbond: [C: 03+2] role idp: update port to port 8080 [puppet] - 10https://gerrit.wikimedia.org/r/582783 (owner: 10Jbond) [10:54:20] (03CR) 10jerkins-bot: [V: 04-1] wdqs: added monitoring to data-reload cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 (owner: 10Gehel) [10:57:01] (03CR) 10Giuseppe Lavagetto: [C: 03+1] "LGTM on paper, but this needs to be deployed and tested before we are near a deployment window." [puppet] - 10https://gerrit.wikimedia.org/r/582153 (https://phabricator.wikimedia.org/T247113) (owner: 10Krinkle) [10:57:04] (03CR) 10Elukey: wdqs: added monitoring to data-reload cookbook (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 (owner: 10Gehel) [10:59:36] (03CR) 10Volans: "nits inline" (033 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 (owner: 10Gehel) [11:00:04] Amir1, Lucas_WMDE, awight, and Urbanecm: My dear minions, it's time we take the moon! Just kidding. Time for European Mid-day SWAT(Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200323T1100). [11:00:04] tgr: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [11:00:08] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Please bump the changelog of the other images that depend on run.sh so that we rebuild all of them now." [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/580128 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [11:02:24] (03PS2) 10Elukey: systemd::timer::job: add a parameter to change the systemd slice [puppet] - 10https://gerrit.wikimedia.org/r/582558 [11:02:26] (03PS2) 10Elukey: statistics::discovery: run the timer under the user.slice [puppet] - 10https://gerrit.wikimedia.org/r/582559 [11:03:18] o/ [11:04:41] the -1 is some kind of generic CI breakage, phan dies on some php7.4 testcase included in vendor [11:04:50] good luck! [11:05:00] 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Rename multimedia-team to structured-data-team - https://phabricator.wikimedia.org/T235550 (10Volans) @MBinder_WMF we're in the process of changing the way we schedule clinic duty, the current person on clinic duty is mentioned in the /topic of the... [11:05:45] (03PS10) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [11:08:18] (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler1002/21533/" [puppet] - 10https://gerrit.wikimedia.org/r/582559 (owner: 10Elukey) [11:08:33] (03CR) 10Elukey: "Tested the related patch in https://puppet-compiler.wmflabs.org/compiler1002/21533/ and looks good :)" [puppet] - 10https://gerrit.wikimedia.org/r/582558 (owner: 10Elukey) [11:09:20] (03PS1) 10Filippo Giunchedi: icinga: use fqdn_rand [puppet] - 10https://gerrit.wikimedia.org/r/582785 [11:10:56] (03PS1) 10Jbond: idp: failover IDP to eqiad [dns] - 10https://gerrit.wikimedia.org/r/582786 (https://phabricator.wikimedia.org/T240941) [11:14:59] (03CR) 10Jbond: [C: 03+1] "lgtm thanks" [puppet] - 10https://gerrit.wikimedia.org/r/582558 (owner: 10Elukey) [11:17:42] (03CR) 10Elukey: [C: 03+2] systemd::timer::job: add a parameter to change the systemd slice [puppet] - 10https://gerrit.wikimedia.org/r/582558 (owner: 10Elukey) [11:17:53] (03CR) 10Elukey: [C: 03+2] statistics::discovery: run the timer under the user.slice [puppet] - 10https://gerrit.wikimedia.org/r/582559 (owner: 10Elukey) [11:18:55] I guess that means self-swatting [11:19:24] (03CR) 10Muehlenhoff: [C: 03+1] "LGTM" [dns] - 10https://gerrit.wikimedia.org/r/582786 (https://phabricator.wikimedia.org/T240941) (owner: 10Jbond) [11:22:21] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp1081 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [11:22:31] tgr: lucky you [11:25:29] filed the CI issue as T248306 [11:25:29] T248306: CI error on WMF branches: Cannot use the final modifier on an abstract class in vendor/microsoft/tolerant-php-parser/tests/cases/parser/abstractMethodDeclaration7.php on line 3 - https://phabricator.wikimedia.org/T248306 [11:27:31] !log upload oozie 4.3.0-3 to thirparty/bigtop14 on wikimedia-stretch - T244499 [11:27:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:27:37] T244499: Upgrade the Hadoop test cluster to BigTop - https://phabricator.wikimedia.org/T244499 [11:28:58] (03CR) 10Jbond: [C: 03+2] idp: failover IDP to eqiad [dns] - 10https://gerrit.wikimedia.org/r/582786 (https://phabricator.wikimedia.org/T240941) (owner: 10Jbond) [11:30:38] 10Operations, 10Wikimedia-Mailing-lists: Please decom reading-wmf mailing list - https://phabricator.wikimedia.org/T248126 (10Volans) p:05Triage→03Medium Followed https://wikitech.wikimedia.org/wiki/Mailman#Disable_or_re-enable_a_mailing_list Mailing list disabled and removed previous admins. ` [fermium] ~... [11:32:37] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp1081 is OK: HTTP OK: HTTP/1.0 200 OK - 22325 bytes in 0.005 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [11:32:38] !log cp1081: restart prometheus-trafficserver-tls-exporter.service [11:32:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:33:28] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: Request for a ldap account and be added to nda ldap group for PHPCC - https://phabricator.wikimedia.org/T247731 (10darthmon_wmde) are there any updates here, please? [11:36:53] 10Operations, 10LDAP-Access-Requests: Offboard Sandra Müllrick from WMF systems - https://phabricator.wikimedia.org/T247750 (10Volans) p:05Triage→03Medium [11:41:24] !log tgr@deploy1001 Synchronized php-1.35.0-wmf.24/extensions/OAuth/includes/frontend/specialpages/SpecialMWOAuthManageMyGrants.php: SWAT: [[gerrit:582768|Get consumerKey from consumerId not from acceptanceId (T247531)]] (duration: 01m 01s) [11:41:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:41:30] T247531: SpecialMWOAuthManageMyGrants: Call to a member function getConsumerKey() on boolean - https://phabricator.wikimedia.org/T247531 [11:47:02] (03PS2) 10Volans: add thephp.cc contractors as ldap-only users [puppet] - 10https://gerrit.wikimedia.org/r/581977 (https://phabricator.wikimedia.org/T247731) (owner: 10ArielGlenn) [11:47:14] (03CR) 10Vgutierrez: Reopen Unix socket upon read errors (032 comments) [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) (owner: 10Ema) [11:51:57] (03PS1) 10Filippo Giunchedi: icinga: manage 'metamonitor' user home [puppet] - 10https://gerrit.wikimedia.org/r/582788 [11:51:59] (03PS1) 10Filippo Giunchedi: icinga: add 'patch' to requirements [puppet] - 10https://gerrit.wikimedia.org/r/582789 [11:54:09] (03CR) 10ArielGlenn: [C: 03+1] add thephp.cc contractors as ldap-only users [puppet] - 10https://gerrit.wikimedia.org/r/581977 (https://phabricator.wikimedia.org/T247731) (owner: 10ArielGlenn) [11:59:10] (03PS2) 10Filippo Giunchedi: icinga: use fqdn_rand [puppet] - 10https://gerrit.wikimedia.org/r/582785 [11:59:13] (03PS1) 10Filippo Giunchedi: prometheus: merge mediawiki rule groups [puppet] - 10https://gerrit.wikimedia.org/r/582790 [11:59:24] (03CR) 10Filippo Giunchedi: [C: 03+2] prometheus: merge mediawiki rule groups [puppet] - 10https://gerrit.wikimedia.org/r/582790 (owner: 10Filippo Giunchedi) [11:59:37] (03PS1) 10Jcrespo: mariadb-backups: Skip Saturday snapshot to prevent a retention of 4 copies [puppet] - 10https://gerrit.wikimedia.org/r/582791 (https://phabricator.wikimedia.org/T138562) [11:59:39] (03CR) 10Volans: [C: 04-1] "See inline" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582785 (owner: 10Filippo Giunchedi) [12:00:15] (03CR) 10Jcrespo: [C: 03+1] install_server: Allow reimage db1077 [puppet] - 10https://gerrit.wikimedia.org/r/582776 (https://phabricator.wikimedia.org/T149418) (owner: 10Marostegui) [12:01:10] (03CR) 10Volans: [C: 03+1] "Don't think it's strictly needed, but sure, LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/582788 (owner: 10Filippo Giunchedi) [12:01:18] (03PS1) 10Giuseppe Lavagetto: Add local service proxy to the tls terminator v0.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/582792 (https://phabricator.wikimedia.org/T244843) [12:01:35] (03PS2) 10Filippo Giunchedi: prometheus: merge mediawiki rule groups [puppet] - 10https://gerrit.wikimedia.org/r/582790 [12:01:37] (03CR) 10Volans: [C: 03+2] add thephp.cc contractors as ldap-only users [puppet] - 10https://gerrit.wikimedia.org/r/581977 (https://phabricator.wikimedia.org/T247731) (owner: 10ArielGlenn) [12:02:20] (03CR) 10Jcrespo: "See inline--" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582791 (https://phabricator.wikimedia.org/T138562) (owner: 10Jcrespo) [12:03:29] (03CR) 10Marostegui: [C: 03+2] install_server: Allow reimage db1077 [puppet] - 10https://gerrit.wikimedia.org/r/582776 (https://phabricator.wikimedia.org/T149418) (owner: 10Marostegui) [12:03:47] (03CR) 10Filippo Giunchedi: icinga: use fqdn_rand (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582785 (owner: 10Filippo Giunchedi) [12:04:08] (03CR) 10Hnowlan: [C: 03+1] Make configuration of envoy a ConfigMap (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/582777 (https://phabricator.wikimedia.org/T244843) (owner: 10Giuseppe Lavagetto) [12:05:59] (03CR) 10Marostegui: [C: 03+1] "makes sense" [puppet] - 10https://gerrit.wikimedia.org/r/582791 (https://phabricator.wikimedia.org/T138562) (owner: 10Jcrespo) [12:07:25] (03CR) 10Filippo Giunchedi: [C: 03+2] "> Patch Set 1: Code-Review+1" [puppet] - 10https://gerrit.wikimedia.org/r/582788 (owner: 10Filippo Giunchedi) [12:07:39] (03PS2) 10Filippo Giunchedi: icinga: manage 'metamonitor' user home [puppet] - 10https://gerrit.wikimedia.org/r/582788 [12:08:10] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: Request for a ldap account and be added to nda ldap group for PHPCC - https://phabricator.wikimedia.org/T247731 (10Volans) p:05Triage→03Medium Change merged, added users `theseer`, `spriebsch` and `sebastianbergmann` to the NDA LDAP group. Please... [12:14:07] (03PS2) 10Filippo Giunchedi: icinga: add 'patch' to requirements [puppet] - 10https://gerrit.wikimedia.org/r/582789 [12:14:09] (03PS3) 10Filippo Giunchedi: icinga: use fqdn_rand [puppet] - 10https://gerrit.wikimedia.org/r/582785 [12:17:03] (03CR) 10Arturo Borrero Gonzalez: Horizon: split $version into $horizon_version and $openstack_version (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) (owner: 10Andrew Bogott) [12:20:20] 10Operations, 10Wikimedia-Mailing-lists: Please decom reading-wmf mailing list - https://phabricator.wikimedia.org/T248126 (10dr0ptp4kt) Thanks @Volans. @Quiddity look correct to you? I'll also for good measure attempt login with the admin password, just don't have it at hand right now. [12:21:33] 10Operations, 10Wikimedia-Mailing-lists: Please decom reading-wmf mailing list - https://phabricator.wikimedia.org/T248126 (10dr0ptp4kt) @Aklapper do you think I should file a new task here re: offboarding process for deletion of users in privileged domains? [12:21:57] (03CR) 10Arturo Borrero Gonzalez: realm.pp: trusted facts unavailable when performing a lookup or pcc (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) (owner: 10Jbond) [12:24:01] (03PS4) 10Jbond: realm.pp: trusted facts unavailable when performing a lookup or pcc [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) [12:25:16] !log marostegui@cumin1001 START - Cookbook sre.hosts.downtime [12:25:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:25:25] (03CR) 10Jbond: "thanks see response inline" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) (owner: 10Jbond) [12:27:43] !log marostegui@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [12:27:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:29:40] (03CR) 10Arturo Borrero Gonzalez: realm.pp: trusted facts unavailable when performing a lookup or pcc (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) (owner: 10Jbond) [12:30:28] (03PS5) 10Jbond: realm.pp: trusted facts unavailable when performing a lookup or pcc [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) [12:30:53] (03CR) 10Arturo Borrero Gonzalez: realm.pp: trusted facts unavailable when performing a lookup or pcc (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) (owner: 10Jbond) [12:32:19] (03CR) 10Volans: "reply inline" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582785 (owner: 10Filippo Giunchedi) [12:33:59] (03PS6) 10Jbond: realm.pp: trusted facts unavailable when performing a lookup or pcc [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) [12:35:31] (03CR) 10Jbond: "thanks updated" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) (owner: 10Jbond) [12:49:42] (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] realm.pp: trusted facts unavailable when performing a lookup or pcc [puppet] - 10https://gerrit.wikimedia.org/r/582048 (https://phabricator.wikimedia.org/T248169) (owner: 10Jbond) [13:17:39] (03CR) 10Daniel Kinzler: Add configuration variable $wgEnableRestAPIDevelopmentEndpoints (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) (owner: 10BPirkle) [13:22:06] (03PS6) 10Muehlenhoff: New ferm sub profile for public mysql proxies [puppet] - 10https://gerrit.wikimedia.org/r/582020 [13:22:26] (03CR) 10Muehlenhoff: New ferm sub profile for public mysql proxies (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [13:23:50] (03PS3) 10Gehel: wdqs: added monitoring to data-reload cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 [13:25:28] (03CR) 10jerkins-bot: [V: 04-1] wdqs: added monitoring to data-reload cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 (owner: 10Gehel) [13:26:31] (03PS4) 10Gehel: wdqs: added monitoring to data-reload cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 [13:26:47] (03PS1) 10Jbond: FasterXML: add jackson-module-kotlin [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/582803 (https://phabricator.wikimedia.org/T246010) [13:32:04] (03PS1) 10Ottomata: Temporarily disable client side error logging for a deploy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582804 (https://phabricator.wikimedia.org/T226986) [13:32:35] (03PS1) 10Muehlenhoff: Remove postgres spec tests [puppet] - 10https://gerrit.wikimedia.org/r/582805 [13:32:52] (03CR) 10Muehlenhoff: [C: 03+1] "Worth a shot!" [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/582803 (https://phabricator.wikimedia.org/T246010) (owner: 10Jbond) [13:33:41] !log installing python-cryptography updates from Stretch point release [13:33:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:34:13] (03CR) 10Jbond: [V: 03+2 C: 03+2] FasterXML: add jackson-module-kotlin [software/cas-overlay-template] - 10https://gerrit.wikimedia.org/r/582803 (https://phabricator.wikimedia.org/T246010) (owner: 10Jbond) [13:36:08] (03CR) 10Ottomata: [C: 03+2] Temporarily disable client side error logging for a deploy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582804 (https://phabricator.wikimedia.org/T226986) (owner: 10Ottomata) [13:37:41] (03CR) 10Gehel: wdqs: added monitoring to data-reload cookbook (034 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 (owner: 10Gehel) [13:39:48] (03CR) 10Elukey: [C: 03+2] Add wdqs blazegraph's alerts only for non-test clusters [puppet] - 10https://gerrit.wikimedia.org/r/582774 (owner: 10Elukey) [13:40:22] !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Temporarily disable client side error logging for a deploy - T226986 (duration: 01m 01s) [13:40:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:40:27] T226986: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 [13:54:32] !log hnowlan@deploy1001 helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [13:54:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:56:45] (03PS3) 10BPirkle: Add configuration variable $wgEnableRestAPIDevelopmentEndpoints [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) [13:57:13] 10Operations, 10LDAP-Access-Requests: Request for a ldap account and be added to nda ldap group for PHPCC - https://phabricator.wikimedia.org/T247731 (10darthmon_wmde) >>! In T247731#5991622, @Volans wrote: > Change merged, added users `theseer`, `spriebsch` and `sebastianbergmann` to the NDA LDAP group. > Ple... [13:58:25] (03CR) 10BPirkle: Add configuration variable $wgEnableRestAPIDevelopmentEndpoints (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) (owner: 10BPirkle) [14:01:04] (03PS1) 10Ottomata: eventgate-logging-external - bump image version to 2020-03-23-133653-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/582808 (https://phabricator.wikimedia.org/T226986) [14:06:16] (03CR) 10Filippo Giunchedi: "PCC for PS3 https://puppet-compiler.wmflabs.org/compiler1003/21534/" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582785 (owner: 10Filippo Giunchedi) [14:07:31] (03CR) 10Giuseppe Lavagetto: [C: 03+1] Bump versions for envoy and envoy-tls-local-proxy to 1.13.1. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/581747 (https://phabricator.wikimedia.org/T246868) (owner: 10RLazarus) [14:13:34] !log elukey@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [14:13:34] !log elukey@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [14:13:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:13:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:14:02] (first k8s deployment) [14:15:43] 10Operations, 10Scap, 10serviceops: Make canary wait time configurable - https://phabricator.wikimedia.org/T217924 (10LarsWirzenius) I'm thinking the following now: * make a release with --canary-wait-times -- hopefully this week * only --canary-wait-times and a build-time acceptance test suite as the onl... [14:21:38] (03CR) 10Volans: [C: 03+1] "LGTM, ack to all comments" [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 (owner: 10Gehel) [14:22:23] 10Operations, 10Performance Issue: Investigate CAS performance - https://phabricator.wikimedia.org/T246010 (10CDanis) Currently seeing a 1.1 minute (!) TTFB for https://idp.wikimedia.org/logout followed by a 504 response code with body "upstream request timeout" [14:22:45] (03CR) 10Elukey: [C: 03+1] wdqs: added monitoring to data-reload cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/582784 (owner: 10Gehel) [14:23:04] (03CR) 10Volans: [C: 03+1] "LGTM, maybe check the compiler to see if my quick test for the resulting times was correct. Thanks for the patch." [puppet] - 10https://gerrit.wikimedia.org/r/582785 (owner: 10Filippo Giunchedi) [14:25:10] !log elukey@deploy1001 helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [14:25:10] !log elukey@deploy1001 helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [14:25:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:25:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:28:28] !log elukey@deploy1001 helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'production' . [14:28:28] !log elukey@deploy1001 helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'canary' . [14:28:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:28:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:35:36] !log jynus@cumin1001 dbctl commit (dc=all): 'remove db1107 from special groups', diff saved to https://phabricator.wikimedia.org/P10746 and previous config saved to /var/cache/conftool/dbconfig/20200323-143536-jynus.json [14:35:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:35:49] (03CR) 10Filippo Giunchedi: [C: 03+2] "> Patch Set 3: Code-Review+1" [puppet] - 10https://gerrit.wikimedia.org/r/582785 (owner: 10Filippo Giunchedi) [14:36:01] (03PS4) 10Filippo Giunchedi: icinga: use fqdn_rand [puppet] - 10https://gerrit.wikimedia.org/r/582785 [14:40:05] !log jynus@cumin1001 dbctl commit (dc=all): 'Increase db1107 main s1 traffic a 50%', diff saved to https://phabricator.wikimedia.org/P10747 and previous config saved to /var/cache/conftool/dbconfig/20200323-144005-jynus.json [14:40:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:45:53] 10Operations: Integrate Stretch 9.12 point update - https://phabricator.wikimedia.org/T244695 (10MoritzMuehlenhoff) [14:46:12] !log jynus@cumin1001 dbctl commit (dc=all): 'Finish doubling db1107 main s1 traffic', diff saved to https://phabricator.wikimedia.org/P10748 and previous config saved to /var/cache/conftool/dbconfig/20200323-144612-jynus.json [14:46:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:00:05] !log otto@deploy1001 Synchronized php-1.35.0-wmf.24/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: [[gerrit:578951|clientError: Changes event fields (T226986)]] (duration: 01m 01s) [15:00:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:00:12] T226986: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 [15:00:46] !log jynus@cumin1001 dbctl commit (dc=all): 'Remove db1089 for special groups (rc)', diff saved to https://phabricator.wikimedia.org/P10749 and previous config saved to /var/cache/conftool/dbconfig/20200323-150046-jynus.json [15:00:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:04:08] !log otto@deploy1001 Synchronized php-1.35.0-wmf.24/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: [[gerrit:578951|clientError: Changes event fields (T226986)]] (take 2) (duration: 00m 59s) [15:04:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:10:22] (03PS1) 10Ottomata: Re-enable client side error logging on group0 and hawwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582822 (https://phabricator.wikimedia.org/T226986) [15:13:58] !log installing freetype updates from Stretch point release [15:14:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:14:12] (03CR) 10Clarakosi: Add configuration variable $wgEnableRestAPIDevelopmentEndpoints (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) (owner: 10BPirkle) [15:14:17] (03CR) 10RLazarus: [C: 03+2] Bump versions for envoy and envoy-tls-local-proxy to 1.13.1. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/581747 (https://phabricator.wikimedia.org/T246868) (owner: 10RLazarus) [15:14:38] (03CR) 10Giuseppe Lavagetto: Make configuration of envoy a ConfigMap (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/582777 (https://phabricator.wikimedia.org/T244843) (owner: 10Giuseppe Lavagetto) [15:14:53] (03CR) 10RLazarus: [V: 03+2 C: 03+2] Bump versions for envoy and envoy-tls-local-proxy to 1.13.1. [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/581747 (https://phabricator.wikimedia.org/T246868) (owner: 10RLazarus) [15:18:17] (03PS4) 10BPirkle: Add configuration variable $wgRestAPIAdditionalRouteFiles [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) [15:18:26] (03PS1) 10Jbond: profile::tlsproxy::envoy: use the chained ssl certificate [puppet] - 10https://gerrit.wikimedia.org/r/582823 [15:19:47] (03CR) 10BPirkle: Add configuration variable $wgRestAPIAdditionalRouteFiles (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) (owner: 10BPirkle) [15:23:34] (03CR) 10Clarakosi: [C: 03+1] Add configuration variable $wgRestAPIAdditionalRouteFiles [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) (owner: 10BPirkle) [15:24:51] !log hnowlan@deploy1001 helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [15:24:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:25:41] 10Operations, 10SRE-Access-Requests, 10serviceops-radar, 10Core Platform Team Workboards (Clinic Duty Team): Onboarding Hugh Nowlan - https://phabricator.wikimedia.org/T242309 (10Volans) p:05High→03Low [15:27:28] 10Operations: Integrate Stretch 9.12 point update - https://phabricator.wikimedia.org/T244695 (10MoritzMuehlenhoff) [15:31:19] 10Operations, 10Performance Issue: Investigate CAS performance - https://phabricator.wikimedia.org/T246010 (10jbond) >>! In T246010#5991919, @CDanis wrote: > Currently seeing a 1.1 minute (!) TTFB for https://idp.wikimedia.org/logout followed by a 504 response code with body "upstream request timeout" Do you... [15:31:45] (03CR) 10Jforrester: "Is is * or just m?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582633 (owner: 10Brian Wolff) [15:32:35] !log installing maridb-10.1 updates from Stretch point release (client/tools/libraries as packaged by Debian, different from wmf-mariadb) [15:32:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:33:01] (03CR) 10Jbond: [C: 03+1] "lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [15:33:19] (03CR) 10Ema: [C: 03+1] Add profile::base::no_firewall to load balancers [puppet] - 10https://gerrit.wikimedia.org/r/575022 (owner: 10Muehlenhoff) [15:35:14] 10Operations, 10Performance Issue: Investigate CAS performance - https://phabricator.wikimedia.org/T246010 (10CDanis) >>! In T246010#5992097, @jbond wrote: >>>! In T246010#5991919, @CDanis wrote: >> Currently seeing a 1.1 minute (!) TTFB for https://idp.wikimedia.org/logout followed by a 504 response code with... [15:36:51] 10Operations, 10Performance Issue: Investigate CAS performance - https://phabricator.wikimedia.org/T246010 (10jbond) > Logging out now also worked fine. ack thanks please add a comment here f you see it again and we should be able to validate if it is only caused by the restart [15:40:18] (03CR) 10Daniel Kinzler: [C: 03+1] "If I had +2 here, I'd merge this :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) (owner: 10BPirkle) [15:41:29] (03PS4) 10Hashar: Add an image for python2 app based on Buster [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/580128 (https://phabricator.wikimedia.org/T215458) [15:41:51] (03CR) 10Hashar: "PS2: bumped all changelogs :)" [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/580128 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [15:49:18] (03PS7) 10Muehlenhoff: New ferm sub profile for public mysql proxies [puppet] - 10https://gerrit.wikimedia.org/r/582020 [15:51:00] PROBLEM - mediawiki originals uploads -hourly- for eqiad on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe1005:9112 job=statsd_exporter site=eqiad https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=eqiad [15:51:33] it is not clear to me what to do when --^ fires [15:52:07] it's not really actionable, I think, aside from to investigate if there's any reason why there's more uploads than usual [15:52:20] ask _joe_ to slow down uploading all his family photos [15:52:20] PROBLEM - mediawiki originals uploads -hourly- for codfw on icinga1001 is CRITICAL: account=mw-media class=originals cluster=swift instance=ms-fe2005:9112 job=statsd_exporter site=codfw https://wikitech.wikimedia.org/wiki/Swift/How_To https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?panelId=26&fullscreen&orgId=1&var-DC=codfw [15:52:27] (and go.dog and I talked to someone who was running a new bot last week) [15:53:11] (03CR) 10Ottomata: [C: 03+2] Re-enable client side error logging on group0 and hawwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582822 (https://phabricator.wikimedia.org/T226986) (owner: 10Ottomata) [15:53:36] cdanis: ah okok, maybe we could add a runbook or similar with quick links? [15:53:47] 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Rename multimedia-team to structured-data-team - https://phabricator.wikimedia.org/T235550 (10MBinder_WMF) Thanks! [15:55:05] elukey: would be good yeah :) also not sure offhand how to get good numbers on swift cluster full-ness or capacity planning [15:55:24] even the fullest disks in eqiad are only at 80%, so, not bad for now [15:56:03] 5% increase in 3 months [15:56:26] cdanis: oh yes I mean how to check, for example, if the origin of a sudden increase in uploads is a bot etc.. [15:56:33] yeah, that I'm not sure about :) [15:56:38] I know it seems a very basic question but I've never done it [15:56:40] aside from looking at recentchanges on commons, lol [15:56:51] (there must be some aggregated view somewhere) [15:56:56] !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Re-enablle client side error logging for group0 and hawwike - T226986 (duration: 01m 00s) [15:57:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:57:02] T226986: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 [15:57:38] cdanis: looking to rc for commons is like filtering sand with your eyes, a joyful experience :D [15:58:02] jokes aside, let's see if godog has some suggestions etc.. [15:58:13] lol, good analogy [15:58:18] not urgent, just to give indications to people looking about how to debug further [15:58:32] (brb) [15:58:42] (03PS11) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [15:59:02] !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Re-enablle client side error logging for group0 and hawwike - T226986 (take 2) (duration: 00m 59s) [15:59:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:59:30] (03CR) 10Muehlenhoff: [C: 03+2] New ferm sub profile for public mysql proxies [puppet] - 10https://gerrit.wikimedia.org/r/582020 (owner: 10Muehlenhoff) [15:59:53] (03CR) 10Ema: Reopen Unix socket upon read errors (032 comments) [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) (owner: 10Ema) [16:02:00] (03PS12) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [16:04:32] PROBLEM - Maps - OSM synchronization lag - codfw on icinga1001 is CRITICAL: 5.101e+06 ge 2.592e+05 https://wikitech.wikimedia.org/wiki/Maps/Runbook https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=12&fullscreen&orgId=1 [16:04:52] PROBLEM - Maps - OSM synchronization lag - eqiad on icinga1001 is CRITICAL: 5.101e+06 ge 2.592e+05 https://wikitech.wikimedia.org/wiki/Maps/Runbook https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=11&fullscreen&orgId=1 [16:06:09] (03PS3) 10Muehlenhoff: Add profile::base::no_firewall to load balancers [puppet] - 10https://gerrit.wikimedia.org/r/575022 [16:06:10] 10Operations, 10Patch-For-Review, 10Security: envoyproxy: CVE-2020-8664 CVE-2020-8661 CVE-2020-8660 CVE-2020-8659 - https://phabricator.wikimedia.org/T246868 (10RLazarus) All hosts updated except snapshot1006, held back until later this week per @ArielGlenn. [16:14:19] (03CR) 10Muehlenhoff: [C: 03+2] Add profile::base::no_firewall to load balancers [puppet] - 10https://gerrit.wikimedia.org/r/575022 (owner: 10Muehlenhoff) [16:14:31] (03CR) 10Andrew Bogott: Horizon: split $version into $horizon_version and $openstack_version (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) (owner: 10Andrew Bogott) [16:15:43] (03PS13) 10Ema: Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) [16:16:05] (03Abandoned) 10Herron: icinga: change check/retry interval of mgmt host check to 10/15 [puppet] - 10https://gerrit.wikimedia.org/r/580989 (https://phabricator.wikimedia.org/T247538) (owner: 10Herron) [16:16:34] (03CR) 10Herron: [C: 03+1] icinga: add 'patch' to requirements [puppet] - 10https://gerrit.wikimedia.org/r/582789 (owner: 10Filippo Giunchedi) [16:17:24] (03CR) 10Herron: [C: 03+1] kibana: assign httpd_extra_modules in all cases [puppet] - 10https://gerrit.wikimedia.org/r/582778 (owner: 10Filippo Giunchedi) [16:17:37] (03CR) 10Cwhite: [C: 03+1] icinga: add 'patch' to requirements [puppet] - 10https://gerrit.wikimedia.org/r/582789 (owner: 10Filippo Giunchedi) [16:18:03] (03CR) 10Herron: [C: 03+1] logstash: don't require full logstash heap on plugin installation [puppet] - 10https://gerrit.wikimedia.org/r/582779 (owner: 10Filippo Giunchedi) [16:18:26] (03CR) 10Cwhite: [C: 03+1] kibana: assign httpd_extra_modules in all cases [puppet] - 10https://gerrit.wikimedia.org/r/582778 (owner: 10Filippo Giunchedi) [16:19:01] yeah good questions elukey cdanis, what I do is indeed look at commons' recent files [16:19:31] I'll add that to the wikitech page [16:19:53] (03CR) 10Vgutierrez: [C: 03+1] Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) (owner: 10Ema) [16:20:54] (03CR) 10Cwhite: [C: 03+1] "LGTM if it works as-is" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582779 (owner: 10Filippo Giunchedi) [16:20:56] (03CR) 10Filippo Giunchedi: [C: 03+2] kibana: assign httpd_extra_modules in all cases [puppet] - 10https://gerrit.wikimedia.org/r/582778 (owner: 10Filippo Giunchedi) [16:22:34] (03CR) 10Filippo Giunchedi: [C: 03+2] icinga: add 'patch' to requirements [puppet] - 10https://gerrit.wikimedia.org/r/582789 (owner: 10Filippo Giunchedi) [16:23:39] (03CR) 10Ema: [C: 03+2] Reopen Unix socket upon read errors [software/atskafka] - 10https://gerrit.wikimedia.org/r/582060 (https://phabricator.wikimedia.org/T237993) (owner: 10Ema) [16:24:28] (03PS1) 10Jbond: apereo_cas: fix lag [puppet] - 10https://gerrit.wikimedia.org/r/582835 (https://phabricator.wikimedia.org/T246010) [16:25:30] (03CR) 10Muehlenhoff: [C: 03+1] "Woot!" [puppet] - 10https://gerrit.wikimedia.org/r/582835 (https://phabricator.wikimedia.org/T246010) (owner: 10Jbond) [16:28:52] (03PS2) 10Muehlenhoff: Switch dbproxy1019 to public Ferm profile [puppet] - 10https://gerrit.wikimedia.org/r/582643 [16:31:53] !log upload atskafka 0.5 to buster-wikimedia T237993 [16:31:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:31:59] T237993: Create replacement for Varnishkafka - https://phabricator.wikimedia.org/T237993 [16:32:59] (03CR) 10Filippo Giunchedi: logstash: don't require full logstash heap on plugin installation (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/582779 (owner: 10Filippo Giunchedi) [16:34:01] (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/582823 (owner: 10Jbond) [16:34:56] (03CR) 10Jbond: [C: 03+2] apereo_cas: fix lag [puppet] - 10https://gerrit.wikimedia.org/r/582835 (https://phabricator.wikimedia.org/T246010) (owner: 10Jbond) [16:34:59] (03CR) 10Filippo Giunchedi: [C: 03+2] logstash: don't require full logstash heap on plugin installation [puppet] - 10https://gerrit.wikimedia.org/r/582779 (owner: 10Filippo Giunchedi) [16:35:11] (03CR) 10Jbond: [C: 03+2] profile::tlsproxy::envoy: use the chained ssl certificate [puppet] - 10https://gerrit.wikimedia.org/r/582823 (owner: 10Jbond) [16:37:54] (03PS1) 10Elukey: profile::analytics::client::limits: add slice settings for cron [puppet] - 10https://gerrit.wikimedia.org/r/582840 [16:38:14] (03CR) 10Alexandros Kosiaris: [C: 03+1] "Nice! Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/582090 (https://phabricator.wikimedia.org/T246689) (owner: 10Bstorm) [16:39:01] (03CR) 10Muehlenhoff: "PCC: https://puppet-compiler.wmflabs.org/compiler1001/21539/" [puppet] - 10https://gerrit.wikimedia.org/r/582643 (owner: 10Muehlenhoff) [16:39:16] (03PS2) 10Elukey: profile::analytics::client::limits: add slice settings for cron [puppet] - 10https://gerrit.wikimedia.org/r/582840 [16:40:56] (03PS2) 10Jcrespo: mariadb-backups: Skip Saturday snapshot to prevent a retention of 4 copies [puppet] - 10https://gerrit.wikimedia.org/r/582791 (https://phabricator.wikimedia.org/T138562) [16:42:00] (03PS5) 10Holger Knust: Added new chart for cpjobqueue [deployment-charts] - 10https://gerrit.wikimedia.org/r/575108 (https://phabricator.wikimedia.org/T220399) [16:44:09] (03PS3) 10Elukey: profile::analytics::client::limits: add slice settings for cron [puppet] - 10https://gerrit.wikimedia.org/r/582840 [16:44:37] 10Operations, 10Patch-For-Review, 10Performance Issue: Investigate CAS performance - https://phabricator.wikimedia.org/T246010 (10jbond) ok so i don't want to get too excited but it appears to me that the [[ https://gerrit.wikimedia.org/r/582835 | last patch ]] may have fixed the issue with lag. The issues... [16:45:33] (03PS1) 10Herron: lists: SMTP/HTTPS forward multimedia-team -> structured-data-team [puppet] - 10https://gerrit.wikimedia.org/r/582842 (https://phabricator.wikimedia.org/T235550) [16:45:50] (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler1001/21541/stat1007.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/582840 (owner: 10Elukey) [16:46:16] (03PS3) 10Andrew Bogott: Horizon: split $version into $horizon_version and $openstack_version [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) [16:46:18] (03PS1) 10Andrew Bogott: horizon: use lookup() in a the source_deploy profiles [puppet] - 10https://gerrit.wikimedia.org/r/582843 [16:46:20] (03PS1) 10Andrew Bogott: Horizon: add datatypes to source_deploy profiles and class [puppet] - 10https://gerrit.wikimedia.org/r/582844 [16:46:22] (03PS1) 10Andrew Bogott: Horizon: unify horizon nova policy with actual nova policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582845 [16:46:24] (03PS1) 10Andrew Bogott: Horizon: unify horizon glance policy with actual glance policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582846 (https://phabricator.wikimedia.org/T247575) [16:46:26] (03PS1) 10Andrew Bogott: Horizon: integrate Horizon policy with actual designate policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582847 (https://phabricator.wikimedia.org/T247575) [16:46:28] (03PS1) 10Andrew Bogott: Horizon: integrate neutron policy with the actual Neutron policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582848 (https://phabricator.wikimedia.org/T247575) [16:49:18] (03CR) 10jerkins-bot: [V: 04-1] Horizon: split $version into $horizon_version and $openstack_version [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) (owner: 10Andrew Bogott) [16:49:30] (03PS2) 10Andrew Bogott: horizon: use lookup() in the source_deploy profiles [puppet] - 10https://gerrit.wikimedia.org/r/582843 [16:49:32] (03PS2) 10Andrew Bogott: Horizon: add datatypes to source_deploy profiles and class [puppet] - 10https://gerrit.wikimedia.org/r/582844 [16:49:34] (03PS2) 10Andrew Bogott: Horizon: unify horizon nova policy with actual nova policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582845 [16:49:36] (03PS2) 10Andrew Bogott: Horizon: unify horizon glance policy with actual glance policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582846 (https://phabricator.wikimedia.org/T247575) [16:49:38] (03PS2) 10Andrew Bogott: Horizon: integrate Horizon policy with actual designate policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582847 (https://phabricator.wikimedia.org/T247575) [16:49:40] (03PS2) 10Andrew Bogott: Horizon: integrate neutron policy with the actual Neutron policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582848 (https://phabricator.wikimedia.org/T247575) [16:49:51] (03CR) 10Elukey: "John/Moritz: not sure if this change makes sense, maybe I am getting overrides completely wrong, let me know :)" [puppet] - 10https://gerrit.wikimedia.org/r/582840 (owner: 10Elukey) [16:52:01] (03PS4) 10Elukey: profile::analytics::client::limits: add slice settings for cron [puppet] - 10https://gerrit.wikimedia.org/r/582840 [16:52:27] Code Deployment Office Hour - Happening in 10mins in the #wikimedia-office channel. Come get your deployments during COVID related questions answered. See also: https://wikitech.wikimedia.org/wiki/Deployments/Covid-19 [16:55:08] (03PS4) 10Andrew Bogott: Horizon: split $version into $horizon_version and $openstack_version [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) [16:55:11] (03PS3) 10Andrew Bogott: horizon: use lookup() in the source_deploy profiles [puppet] - 10https://gerrit.wikimedia.org/r/582843 [16:55:13] (03PS3) 10Andrew Bogott: Horizon: add datatypes to source_deploy profiles and class [puppet] - 10https://gerrit.wikimedia.org/r/582844 [16:55:15] (03PS3) 10Andrew Bogott: Horizon: unify horizon nova policy with actual nova policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582845 [16:55:17] (03PS3) 10Andrew Bogott: Horizon: unify horizon glance policy with actual glance policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582846 (https://phabricator.wikimedia.org/T247575) [16:55:19] (03PS3) 10Andrew Bogott: Horizon: integrate Horizon policy with actual designate policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582847 (https://phabricator.wikimedia.org/T247575) [16:55:21] (03PS3) 10Andrew Bogott: Horizon: integrate neutron policy with the actual Neutron policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582848 (https://phabricator.wikimedia.org/T247575) [16:56:25] (03CR) 10Jcrespo: [C: 03+2] mariadb-backups: Skip Saturday snapshot to prevent a retention of 4 copies [puppet] - 10https://gerrit.wikimedia.org/r/582791 (https://phabricator.wikimedia.org/T138562) (owner: 10Jcrespo) [16:56:56] (03CR) 10Volans: [C: 03+1] "LGTM, thanks for looking into it!" [puppet] - 10https://gerrit.wikimedia.org/r/582842 (https://phabricator.wikimedia.org/T235550) (owner: 10Herron) [16:57:45] 10Operations, 10vm-requests: codfw: 1 VM for builder - https://phabricator.wikimedia.org/T248165 (10akosiaris) p:05Triage→03Medium LGTM. Thanks! [16:58:48] (03CR) 10jerkins-bot: [V: 04-1] Horizon: split $version into $horizon_version and $openstack_version [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) (owner: 10Andrew Bogott) [17:00:04] gehel and onimisionipe: Dear deployers, time to do the Wikidata Query Service weekly deploy deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200323T1700). [17:05:12] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:06:29] (03PS6) 10Ppchelko: Added new chart for cpjobqueue [deployment-charts] - 10https://gerrit.wikimedia.org/r/575108 (https://phabricator.wikimedia.org/T220399) (owner: 10Holger Knust) [17:07:45] (03CR) 10Herron: [C: 03+2] lists: SMTP/HTTPS forward multimedia-team -> structured-data-team [puppet] - 10https://gerrit.wikimedia.org/r/582842 (https://phabricator.wikimedia.org/T235550) (owner: 10Herron) [17:27:07] 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Rename multimedia-team to structured-data-team - https://phabricator.wikimedia.org/T235550 (10herron) 05Open→03Resolved a:03herron Hello! The multimedia-team list has been renamed to structured-data-team, and redirects/forwarding have been p... [17:34:25] (03PS9) 10DannyS712: abusefilter.php: Remove settings that duplicate defaults, and clean up [mediawiki-config] - 10https://gerrit.wikimedia.org/r/552610 (https://phabricator.wikimedia.org/T238965) [17:36:07] 10Operations, 10serviceops, 10User-WDoran: Canaries canaries canaries - https://phabricator.wikimedia.org/T210143 (10WDoranWMF) [17:37:23] (03PS1) 10DannyS712: InitialiseSettings - clean up groupOverrides layout / spacing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582859 [17:49:30] (03PS1) 10Mvolz: Update citoid to use buster/node10 [deployment-charts] - 10https://gerrit.wikimedia.org/r/582861 [17:51:54] (03PS1) 10Ottomata: eventstreams - bump image version to 2020-03-23-173308-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/582864 (https://phabricator.wikimedia.org/T238658) [17:53:40] (03CR) 10Ottomata: [C: 03+2] eventstreams - bump image version to 2020-03-23-173308-production [deployment-charts] - 10https://gerrit.wikimedia.org/r/582864 (https://phabricator.wikimedia.org/T238658) (owner: 10Ottomata) [17:54:51] ottomata: btw, I'm getting a js 502 error on most things I try to do with Turnilo. is this a known issue? [17:57:34] !log otto@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [17:57:35] !log otto@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [17:57:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:57:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:57:49] Krinkle: hm, not that I know of [17:57:54] (03PS2) 10DannyS712: InitialiseSettings - clean up groupOverrides layout / spacing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582859 (https://phabricator.wikimedia.org/T231178) [17:58:32] looks ok to me Krinkle [17:59:00] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:59:05] (03PS3) 10DannyS712: InitialiseSettings - clean up groupOverrides layout / spacing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582859 (https://phabricator.wikimedia.org/T231178) [17:59:40] ottomata: Im getting http 500 or 503 for 99/100 attempts [17:59:44] https://usercontent.irccloud-cdn.com/file/LvD1Ewt6/Screenshot%202020-03-23%20at%2017.59.24.png [17:59:52] not been able to correlate what makes it work the 100th time. [18:00:04] RoanKattouw, Niharika, and Urbanecm: It is that lovely time of the day again! You are hereby commanded to deploy Morning SWAT(Max 6 patches). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200323T1800). [18:00:04] bpirkle, DannyS712, and DannyS712: A patch you scheduled for Morning SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [18:00:12] I'm here [18:00:18] here [18:00:29] also, MatmaRex has a patch [18:01:10] hi [18:02:34] (03PS1) 10Ottomata: eventstreams - reduce resource requests now that mem leak is gone [deployment-charts] - 10https://gerrit.wikimedia.org/r/582866 (https://phabricator.wikimedia.org/T238658) [18:04:36] (03CR) 10Ottomata: [C: 03+2] eventstreams - reduce resource requests now that mem leak is gone [deployment-charts] - 10https://gerrit.wikimedia.org/r/582866 (https://phabricator.wikimedia.org/T238658) (owner: 10Ottomata) [18:04:38] I can SWAT if no one else is! [18:05:05] (03CR) 10Jforrester: [C: 03+1] "LGTM." [deployment-charts] - 10https://gerrit.wikimedia.org/r/582861 (owner: 10Mvolz) [18:05:33] !log otto@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [18:05:33] !log otto@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [18:05:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:05:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:05:48] (03PS5) 10Andrew Bogott: Horizon: split $version into $horizon_version and $openstack_version [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) [18:05:50] (03PS4) 10Andrew Bogott: horizon: use lookup() in the source_deploy profiles [puppet] - 10https://gerrit.wikimedia.org/r/582843 [18:05:52] (03PS4) 10Andrew Bogott: Horizon: add datatypes to source_deploy profiles and class [puppet] - 10https://gerrit.wikimedia.org/r/582844 [18:05:54] (03PS4) 10Andrew Bogott: Horizon: unify horizon nova policy with actual nova policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582845 [18:05:56] (03PS4) 10Andrew Bogott: Horizon: unify horizon glance policy with actual glance policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582846 (https://phabricator.wikimedia.org/T247575) [18:05:59] (03PS4) 10Andrew Bogott: Horizon: integrate Horizon policy with actual designate policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582847 (https://phabricator.wikimedia.org/T247575) [18:06:03] (03PS4) 10Andrew Bogott: Horizon: integrate neutron policy with the actual Neutron policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582848 (https://phabricator.wikimedia.org/T247575) [18:06:06] bpirkle: you're first, can you test the commit? [18:06:35] (03PS5) 10Urbanecm: Add configuration variable $wgRestAPIAdditionalRouteFiles [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) (owner: 10BPirkle) [18:06:40] (03CR) 10Urbanecm: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) (owner: 10BPirkle) [18:07:12] Urbanecm: looks good [18:07:48] bpirkle: I mean, once it's available for testing :) [18:08:04] (but i see the core patch was merged today, so it may not be possible) [18:08:24] !log otto@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [18:08:25] I just need to test that stuff keeps working :) [18:08:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:08:36] bpirkle: okay! [18:08:43] MatmaRex: I've +2'ed your backport, will ping you once it's ready [18:08:45] (03CR) 10jerkins-bot: [V: 04-1] Horizon: split $version into $horizon_version and $openstack_version [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) (owner: 10Andrew Bogott) [18:08:53] Urbanecm: thanks [18:09:12] I was looking at https://phabricator.wikimedia.org/T248311 and when nlwiki enabled ve by default James_F just removed it from the dblist and yaml file but then the dblists say don’t edit so I’m not sure what to do. [18:09:28] For the record, I know https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/582859/ is almost 600 lines, but its all whitespace and formatting [18:09:40] RhinosF1: Remove it from the YAML file and rebuild the dblists. [18:09:42] !log otto@deploy1001 helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [18:09:42] !log otto@deploy1001 helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [18:09:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:09:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:10:03] James_F: how do I rebuild a db list? [18:10:18] RhinosF1: `composer test` will do it. [18:10:42] RhinosF1: see also https://groups.google.com/a/wikimedia.org/forum/#!topic/sitereq-l/TUnUgF7df70 for reference [18:11:12] (03Merged) 10jenkins-bot: Add configuration variable $wgRestAPIAdditionalRouteFiles [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581886 (https://phabricator.wikimedia.org/T247997) (owner: 10BPirkle) [18:11:35] 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Rename multimedia-team to structured-data-team - https://phabricator.wikimedia.org/T235550 (10MBinder_WMF) Brilliant, thank you. @Aklapper , per your earlier comment > I have created the Phab mailing list user account [structured-data-team-list]... [18:11:56] James_F: I normally just create changesets from gerrit web so where would I run it [18:12:00] bpirkle: available at mwdebug1001, should arrive to labs within half an hour (probably sooner, but it's outside of my control). Lmk if I can deploy it. [18:12:40] Looks good on mwdebug1001 [18:12:46] thanks [18:12:59] thank you! [18:13:28] syncing [18:14:21] RhinosF1: This is not a change you can make through the Web interface, sorry. [18:14:27] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 5eb70ac: Add configuration variable $wgRestAPIAdditionalRouteFiles (T247997) (duration: 01m 00s) [18:14:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:14:33] T247997: System Adminstrator enables or disables core development endpoints - https://phabricator.wikimedia.org/T247997 [18:15:12] James_F: so I have to clone git repo, change locally, run ‘composer test’, push up to gerrit. [18:15:19] RhinosF1: Yes. [18:15:30] (03PS3) 10Urbanecm: wgCopyUploadsDomains: Add supremecourt.gov [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581891 (https://phabricator.wikimedia.org/T248146) (owner: 10DannyS712) [18:15:30] RhinosF1: Eventually, all changes will go that way. [18:15:35] (03PS1) 10Ottomata: eventstreams - set helm timeout to 3 minutes [deployment-charts] - 10https://gerrit.wikimedia.org/r/582870 [18:15:43] (03CR) 10Urbanecm: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581891 (https://phabricator.wikimedia.org/T248146) (owner: 10DannyS712) [18:15:47] James_F: I can do that tonight then. [18:15:52] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 5eb70ac: Add configuration variable $wgRestAPIAdditionalRouteFiles (T247997; take II) (duration: 00m 59s) [18:15:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:16:02] 10Puppet, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: logspam-watch: fatal.log no longer exists - https://phabricator.wikimedia.org/T248337 (10brennen) [18:16:10] RhinosF1: If you get stuck, shout and I can help. [18:16:30] bpirkle: it's live [18:16:32] (03CR) 10Ottomata: [C: 03+2] eventstreams - set helm timeout to 3 minutes [deployment-charts] - 10https://gerrit.wikimedia.org/r/582870 (owner: 10Ottomata) [18:16:40] (03Merged) 10jenkins-bot: wgCopyUploadsDomains: Add supremecourt.gov [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581891 (https://phabricator.wikimedia.org/T248146) (owner: 10DannyS712) [18:17:26] Thanks James! [18:17:30] DannyS712: syncing this one [18:17:34] (03PS1) 10Brennen Bearnes: logspam-watch: don't use fatal.log; shorten names [puppet] - 10https://gerrit.wikimedia.org/r/582871 (https://phabricator.wikimedia.org/T248337) [18:17:39] let me know when to check it [18:18:19] !log otto@deploy1001 helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [18:18:19] !log otto@deploy1001 helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [18:18:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:18:23] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 0c12fc2: wgCopyUploadsDomains: Add supremecourt.gov (T248146) (duration: 01m 00s) [18:18:23] DannyS712: it's going straight to prod, you can review once the second sync is finished [18:18:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:18:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:18:32] T248146: Add supremecourt.gov to the wgCopyUploadsDomains whitelist of Wikimedia Commons - https://phabricator.wikimedia.org/T248146 [18:19:05] (03CR) 10Urbanecm: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582613 (https://phabricator.wikimedia.org/T248282) (owner: 10DannyS712) [18:19:19] DannyS712: ^^ i assume this is a no-op ^^ [18:19:26] 10Puppet, 10Patch-For-Review, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: logspam-watch: fatal.log no longer exists - https://phabricator.wikimedia.org/T248337 (10brennen) p:05Triage→03Medium [18:19:31] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 0c12fc2: wgCopyUploadsDomains: Add supremecourt.gov (T248146, take II) (duration: 00m 59s) [18:19:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:19:43] DannyS712: whitelist change is live, you can check at prod [18:20:03] (03Merged) 10jenkins-bot: Don't try to grant `oathauth-enable` to `*` (part 1) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582613 (https://phabricator.wikimedia.org/T248282) (owner: 10DannyS712) [18:20:08] MatmaRex: https://integration.wikimedia.org/ci/job/mwext-php72-phan-seccheck-docker/42718/ has a failure, could you have a look? [18:20:14] trying to upload https://www.supremecourt.gov/opinions/19pdf/19-5421_o7jq.pdf on beta commons doesn't work... [18:20:48] Nor does it work on testwiki [18:20:55] DannyS712: ehh... missing *., it's on www subdomain [18:20:59] could you fix that please? [18:21:17] (03PS1) 10DannyS712: wgCopyUploadsDomains: Fix supremecourt.gov [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582872 (https://phabricator.wikimedia.org/T248146) [18:21:32] Urbanecm: i have no idea where that error comes from [18:21:34] "PHP Fatal error: Cannot use the final modifier on an abstract class in vendor/microsoft/tolerant-php-parser/tests/cases/parser/abstractMethodDeclaration7.php on line 3" [18:21:37] (03CR) 10Jforrester: [C: 03+1] logspam-watch: don't use fatal.log; shorten names [puppet] - 10https://gerrit.wikimedia.org/r/582871 (https://phabricator.wikimedia.org/T248337) (owner: 10Brennen Bearnes) [18:21:53] (03PS2) 10DannyS712: wgCopyUploadsDomains: Fix supremecourt.gov [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582872 (https://phabricator.wikimedia.org/T248146) [18:22:22] MatmaRex: me neither, but I hesitate to bypass Jenkins [18:22:35] (03PS6) 10Andrew Bogott: Horizon: split $version into $horizon_version and $openstack_version [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) [18:22:46] (03PS5) 10Andrew Bogott: horizon: use lookup() in the source_deploy profiles [puppet] - 10https://gerrit.wikimedia.org/r/582843 [18:22:58] (03PS5) 10Andrew Bogott: Horizon: add datatypes to source_deploy profiles and class [puppet] - 10https://gerrit.wikimedia.org/r/582844 [18:22:59] Urbanecm: is that a voting job? [18:23:01] MatmaRex: tgr opened a phab task earlier about that [18:23:09] Which job? [18:23:24] James_F: https://integration.wikimedia.org/ci/job/mwext-php72-phan-seccheck-docker/42718/ is a failure at https://gerrit.wikimedia.org/r/582857 [18:23:29] scheduled for deployment by MatmaRex [18:23:35] MatmaRex: https://phabricator.wikimedia.org/T248306 ? [18:24:33] I'm giving one-off permission to bypass. [18:24:39] James_F: thank you, sir [18:24:49] But don't get into a habit of it. :-) Will poke the phan issue. [18:24:55] !log urbanecm@deploy1001 Synchronized wmf-config/CommonSettings.php: SWAT: 212114e: Dont try to grant `oathauth-enable` to `*` (T248282) (duration: 00m 59s) [18:25:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:25:01] T248282: Never try to grant `oathauth-enable` to `*` - https://phabricator.wikimedia.org/T248282 [18:25:03] apparently it is "sporadic", so maybe we can just +2 again and see if it passes. [18:25:11] James_F: could you please remove the -1 vote? For some reason, wmf-deployment doesn't have permission to do that [18:25:25] Done. [18:25:25] (03CR) 10jerkins-bot: [V: 04-1] Horizon: split $version into $horizon_version and $openstack_version [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) (owner: 10Andrew Bogott) [18:25:40] thanks [18:26:18] (03CR) 10Andrew Bogott: [V: 03+2 C: 03+2] Horizon: split $version into $horizon_version and $openstack_version [puppet] - 10https://gerrit.wikimedia.org/r/582594 (https://phabricator.wikimedia.org/T247575) (owner: 10Andrew Bogott) [18:27:48] MatmaRex: ready at mwdebug1001, could you test, please? [18:27:56] (03CR) 10Andrew Bogott: [C: 03+2] horizon: use lookup() in the source_deploy profiles [puppet] - 10https://gerrit.wikimedia.org/r/582843 (owner: 10Andrew Bogott) [18:28:07] (03CR) 10Andrew Bogott: [C: 03+2] Horizon: add datatypes to source_deploy profiles and class [puppet] - 10https://gerrit.wikimedia.org/r/582844 (owner: 10Andrew Bogott) [18:29:06] Urbanecm: yeah, looking [18:29:10] thanks [18:29:37] DannyS712: could you add https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/582872 to the calendar if not done already, please? [18:29:53] (03CR) 10Urbanecm: [C: 03+2] "fix of earlier deployed patch" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582872 (https://phabricator.wikimedia.org/T248146) (owner: 10DannyS712) [18:30:08] Urbanecm: looks fixed at https://test.wikipedia.org/w/index.php?title=User:Matma_Rex/sandbox&action=history [18:30:16] thanks, syncing to prod [18:30:37] {{done}} added to deployment calendar [18:30:45] thanks DannyS712 [18:30:46] (03Merged) 10jenkins-bot: wgCopyUploadsDomains: Fix supremecourt.gov [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582872 (https://phabricator.wikimedia.org/T248146) (owner: 10DannyS712) [18:31:08] (03PS4) 10DannyS712: InitialiseSettings - clean up groupOverrides layout / spacing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582859 (https://phabricator.wikimedia.org/T231178) [18:31:19] (03CR) 10Andrew Bogott: [C: 03+2] Horizon: unify horizon nova policy with actual nova policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582845 (owner: 10Andrew Bogott) [18:31:26] (03PS5) 10Andrew Bogott: Horizon: unify horizon nova policy with actual nova policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582845 [18:32:04] !log urbanecm@deploy1001 Synchronized php-1.35.0-wmf.24/extensions/VisualEditor/includes/ApiVisualEditorEdit.php: SWAT: cbda0e5: ApiVisualEditorEdit: Fix handling of minor parameter (T248257) (duration: 01m 00s) [18:32:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:32:10] T248257: Visual editor (and NWE) don't mark edits as minor - https://phabricator.wikimedia.org/T248257 [18:32:13] MatmaRex: synced [18:32:28] thank you! [18:32:36] (03PS5) 10DannyS712: InitialiseSettings - clean up groupOverrides layout / spacing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582859 (https://phabricator.wikimedia.org/T231178) [18:32:59] happy to help! [18:33:24] DannyS712: it's unhelpful if you update your patch during the window when it is about to be deployed :) [18:33:50] sorry, just a rebase and 3 lines: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/582859/4..5/wmf-config/InitialiseSettings.php won't happen again [18:33:55] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 6ca1593: wgCopyUploadsDomains: Fix supremecourt.gov (T248146) (duration: 01m 00s) [18:33:57] noted [18:34:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:34:01] T248146: Add supremecourt.gov to the wgCopyUploadsDomains whitelist of Wikimedia Commons - https://phabricator.wikimedia.org/T248146 [18:35:01] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 6ca1593: wgCopyUploadsDomains: Fix supremecourt.gov (T248146; take II) (duration: 00m 59s) [18:35:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:35:09] DannyS712: could you try again? [18:35:37] (03CR) 10Urbanecm: [C: 03+2] "SWAT: no-op" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582859 (https://phabricator.wikimedia.org/T231178) (owner: 10DannyS712) [18:35:45] still not working on beta cluster [18:35:56] DannyS712: give beta some time, what about production? [18:36:16] didn't work on test wiki, but just realized why not: only applied to commons [18:36:21] (03PS5) 10Andrew Bogott: Horizon: unify horizon glance policy with actual glance policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582846 (https://phabricator.wikimedia.org/T247575) [18:36:23] (03PS5) 10Andrew Bogott: Horizon: integrate Horizon policy with actual designate policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582847 (https://phabricator.wikimedia.org/T247575) [18:36:25] (03PS5) 10Andrew Bogott: Horizon: integrate neutron policy with the actual Neutron policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582848 (https://phabricator.wikimedia.org/T247575) [18:36:27] (03PS1) 10Andrew Bogott: Horizon: fix path to nova policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582875 (https://phabricator.wikimedia.org/T247575) [18:36:30] (03Merged) 10jenkins-bot: InitialiseSettings - clean up groupOverrides layout / spacing [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582859 (https://phabricator.wikimedia.org/T231178) (owner: 10DannyS712) [18:38:19] confirmed to work - https://commons.wikimedia.org/wiki/File:Davis_v._United_States_(2020).pdf [18:38:22] thanks! [18:38:56] (03CR) 10Andrew Bogott: [C: 03+2] Horizon: fix path to nova policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582875 (https://phabricator.wikimedia.org/T247575) (owner: 10Andrew Bogott) [18:39:12] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 0e535b1: InitialiseSettings - clean up groupOverrides layout / spacing (T231178) (duration: 01m 00s) [18:39:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:39:18] T231178: General cleanup of initialize settings - https://phabricator.wikimedia.org/T231178 [18:39:21] should be all! [18:39:33] just need another sync, right? [18:40:40] in progress :) [18:41:17] (03PS4) 10DannyS712: Consolidate user rights assignments, part 1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562396 (https://phabricator.wikimedia.org/T239771) [18:41:26] (03CR) 10DannyS712: Consolidate user rights assignments, part 1 (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562396 (https://phabricator.wikimedia.org/T239771) (owner: 10DannyS712) [18:41:37] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 0e535b1: InitialiseSettings - clean up groupOverrides layout / spacing (T231178; take II) (duration: 00m 59s) [18:41:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:43:29] (03PS1) 10Urbanecm: Add gwtoolset to available rights to allow granting to global groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582876 [18:44:35] (03CR) 10DannyS712: [C: 04-1] Add gwtoolset to available rights to allow granting to global groups (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582876 (owner: 10Urbanecm) [18:45:53] thanks DannyS712, actually, this would be better in the extension itself [18:45:56] !log Morning SWAT done [18:46:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:46:08] (03PS6) 10Andrew Bogott: Horizon: unify horizon glance policy with actual glance policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582846 (https://phabricator.wikimedia.org/T247575) [18:46:10] (03PS6) 10Andrew Bogott: Horizon: integrate Horizon policy with actual designate policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582847 (https://phabricator.wikimedia.org/T247575) [18:46:12] (03PS6) 10Andrew Bogott: Horizon: integrate neutron policy with the actual Neutron policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582848 (https://phabricator.wikimedia.org/T247575) [18:46:16] The extension already includes it as an available right, its just not installed on metawiki [18:46:26] good note... [18:47:12] Urbanecm: when I do that patch, are you around for either 11pm today, 11am tommorow or whenever late swat is tommorow to deploy [18:47:32] RhinosF1: maybe, and if not, someone else probably will. It's not that frequent SWATs are unattended :-) [18:47:33] 11pm tommorow that is [18:48:01] Urbanecm: heh, just wanted to make sure as one of mine was. My first! [18:48:07] (03PS2) 10Urbanecm: Add gwtoolset to available rights to allow granting to global groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582876 [18:48:23] that's sad :( [18:49:00] That’s my luck! [18:51:18] (03CR) 10DannyS712: [C: 03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582876 (owner: 10Urbanecm) [18:52:12] (03PS1) 10DannyS712: More cleanup of InitialiseSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582879 (https://phabricator.wikimedia.org/T231178) [18:56:52] 10Operations, 10Parsoid, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor was removed from Wikitech because Parsoid/PHP isn't yet compatible with how Wikitech is set up - https://phabricator.wikimedia.org/T241961 (10cscott) [18:57:18] 10Operations, 10Parsoid, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor was removed from Wikitech because Parsoid/PHP isn't yet compatible with how Wikitech is set up - https://phabricator.wikimedia.org/T241961 (10cscott) [18:57:23] (03PS1) 10RLazarus: Bump tls.image_version for envoy update to 1.13.1. [deployment-charts] - 10https://gerrit.wikimedia.org/r/582880 (https://phabricator.wikimedia.org/T246868) [18:58:04] 10Operations, 10Parsoid, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor was removed from Wikitech because Parsoid/PHP isn't yet compatible with how Wikitech is set up - https://phabricator.wikimedia.org/T241961 (10cscott) Added T248343 and T237773 as parent tasks based on the two main options for f... [19:02:30] 10Operations, 10Parsoid, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor was removed from Wikitech because Parsoid/PHP isn't yet compatible with how Wikitech is set up - https://phabricator.wikimedia.org/T241961 (10ssastry) >>! In T241961#5993030, @cscott wrote: > Added T248343 and T237773 as parent... [19:03:25] (03CR) 10Elukey: "Applied the override on stat1004, and tested with a crontab. It seems working so far!" [puppet] - 10https://gerrit.wikimedia.org/r/582840 (owner: 10Elukey) [19:03:28] 10Operations, 10Parsoid, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor was removed from Wikitech because Parsoid/PHP isn't yet compatible with how Wikitech is set up - https://phabricator.wikimedia.org/T241961 (10cscott) [19:03:52] (03PS4) 10VolkerE: Remove unnecessary, overqualified element parts of id selectors [mediawiki-config] - 10https://gerrit.wikimedia.org/r/581879 (https://phabricator.wikimedia.org/T248137) [19:03:58] 10Operations, 10Parsoid, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor was removed from Wikitech because Parsoid/PHP isn't yet compatible with how Wikitech is set up - https://phabricator.wikimedia.org/T241961 (10cscott) [19:04:21] 10Operations, 10Parsoid, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor was removed from Wikitech because Parsoid/PHP isn't yet compatible with how Wikitech is set up - https://phabricator.wikimedia.org/T241961 (10cscott) Fixed, thanks! [19:26:27] (03PS2) 10DannyS712: More cleanup of InitialiseSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582879 (https://phabricator.wikimedia.org/T231178) [19:37:31] (03CR) 10Jhedden: [C: 03+2] techblog.wikimedia.org: Reduce TTL [dns] - 10https://gerrit.wikimedia.org/r/577370 (https://phabricator.wikimedia.org/T246507) (owner: 10BryanDavis) [19:39:27] (03PS2) 10Jhedden: techblog.wikimedia.org: Reduce TTL [dns] - 10https://gerrit.wikimedia.org/r/577370 (https://phabricator.wikimedia.org/T246507) (owner: 10BryanDavis) [19:42:32] (03PS2) 10Hashar: zuul: provision the scap repository [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) [19:42:54] (03CR) 10Hashar: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [19:49:53] (03PS1) 10Hashar: Add fake ssh key pair for zuul scap deployment [labs/private] - 10https://gerrit.wikimedia.org/r/582895 (https://phabricator.wikimedia.org/T215458) [19:50:17] (03CR) 10Hashar: "We will need a ssh key pair to be generated and stored: keyholder/deploy_zuul.pub" [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [19:52:50] (03CR) 10Hashar: [V: 03+2 C: 03+2] Add fake ssh key pair for zuul scap deployment [labs/private] - 10https://gerrit.wikimedia.org/r/582895 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [19:54:30] (03CR) 10Hashar: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [20:00:04] cscott, arlolra, subbu, halfak, and accraze: Dear deployers, time to do the Services – Graphoid / Parsoid / Citoid / ORES deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200323T2000). [20:00:31] huh .. i thought James_F had fixed this services ping ^ [20:02:47] subbu: I did. I guess thcipriani didn't `git pull` before creating the latest schedules. :-) [20:03:07] (03CR) 10Krinkle: "If I recall correctly, there is or was also an Apache level override for these that makes them spepecific to the paths that serve artefact" [puppet] - 10https://gerrit.wikimedia.org/r/582604 (https://phabricator.wikimedia.org/T245658) (owner: 10Brian Wolff) [20:03:20] ...I did not :( [20:04:47] * thcipriani does now [20:06:31] (03PS3) 10Hashar: zuul: provision the scap repository [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) [20:07:07] (03CR) 10Hashar: zuul: provision the scap repository (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [20:07:15] (03CR) 10Hashar: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [20:10:16] (fixed) [20:16:52] thcipriani: Thanks! [20:21:27] (03CR) 10Krinkle: [C: 03+1] logspam-watch: don't use fatal.log; shorten names [puppet] - 10https://gerrit.wikimedia.org/r/582871 (https://phabricator.wikimedia.org/T248337) (owner: 10Brennen Bearnes) [20:23:36] (03PS1) 10Andrew Bogott: Keystone policy: merge keystone policy with horizon identity policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582899 (https://phabricator.wikimedia.org/T247575) [20:26:11] (03PS7) 10Andrew Bogott: Horizon: integrate Horizon policy with actual designate policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582847 (https://phabricator.wikimedia.org/T247575) [20:26:13] (03PS7) 10Andrew Bogott: Horizon: integrate neutron policy with the actual Neutron policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582848 (https://phabricator.wikimedia.org/T247575) [20:26:15] (03PS2) 10Andrew Bogott: Keystone policy: merge keystone policy with horizon identity policy.yaml [puppet] - 10https://gerrit.wikimedia.org/r/582899 (https://phabricator.wikimedia.org/T247575) [20:28:02] (03PS4) 10Hashar: zuul: provision the scap repository [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) [20:29:30] (03CR) 10jerkins-bot: [V: 04-1] zuul: provision the scap repository [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [20:33:35] (03CR) 10Andrew Bogott: [C: 03+1] "This is OK with me. I think we're going to wind up ripping out Shinken entirely pretty soon anyway." [puppet] - 10https://gerrit.wikimedia.org/r/582612 (https://phabricator.wikimedia.org/T210993) (owner: 10Jhedden) [20:33:44] 10Operations, 10LDAP-Access-Requests: Offboard Sandra Müllrick from WMF systems - https://phabricator.wikimedia.org/T247750 (10Volans) @WMDE-leszek at the best of my knowledge this has never been done before for WMDE accounts, see for example this list (thanks to quiddity): https://meta.wikimedia.org/w/index.p... [20:34:01] (03CR) 10Andrew Bogott: [C: 03+1] cloud eqiad1: Remove references to old cloud-puppetmaster stretch hosts [puppet] - 10https://gerrit.wikimedia.org/r/581105 (owner: 10Alex Monk) [20:35:27] 10Operations, 10Wikimedia-Mailing-lists: Please decom reading-wmf mailing list - https://phabricator.wikimedia.org/T248126 (10Aklapper) @dr0ptp4kt: I'd say yes. In the best case it'll get implemented, in the worst case you spent 5min and it gets declined. ;) [20:36:00] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe: Ensure hiera only has profile:: qualified or global hiera keys - https://phabricator.wikimedia.org/T247956 (10Volans) [20:37:17] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe: Ensure hiera only has profile:: qualified or global hiera keys - https://phabricator.wikimedia.org/T247956 (10Volans) I've added the unique list of keys in the task description. I think those with the `role::` prefix should be treated separately. At f... [20:41:59] (03CR) 10Alex Monk: "Yeah, IIRC we struggled to install shinken on stretch, so it runs on jessie and I expect it'll die when we get rid of jessie. Haven't dug " [puppet] - 10https://gerrit.wikimedia.org/r/582612 (https://phabricator.wikimedia.org/T210993) (owner: 10Jhedden) [20:44:37] 10Operations, 10Wikimedia-Mailing-lists: Please decom reading-wmf mailing list - https://phabricator.wikimedia.org/T248126 (10Quiddity) Looks good to me. Thanks all :) [20:45:29] (03PS2) 10Fomafix: Add DEPRECATED_LANGUAGE_CODE_MAPPING to wgInterlanguageLinkCodeMap [mediawiki-config] - 10https://gerrit.wikimedia.org/r/558052 (https://phabricator.wikimedia.org/T248352) [20:46:02] (03CR) 10BryanDavis: [C: 03+1] "Looks like a smart way to get around the horrible behavior of `if` in nginx." [puppet] - 10https://gerrit.wikimedia.org/r/579952 (https://phabricator.wikimedia.org/T234617) (owner: 10Arturo Borrero Gonzalez) [20:52:12] (03PS5) 10Hashar: zuul: provision the scap repository [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) [20:52:31] (03PS1) 10Jhedden: openstack: Configure WSGI to use horizon virtualenv [puppet] - 10https://gerrit.wikimedia.org/r/582907 (https://phabricator.wikimedia.org/T240852) [20:52:33] 10Operations, 10MediaWiki-Authentication-and-authorization, 10Security-Team, 10Traffic, 10Security: Investigate usefulness of SameSite cookies for logged-in accounts - https://phabricator.wikimedia.org/T158604 (10MarcoAurelio) Just to mention this is already happening and as a consequence CentralAuth coo... [20:59:16] (03PS2) 10Ppchelko: Update citoid to use buster/node10 [deployment-charts] - 10https://gerrit.wikimedia.org/r/582861 (owner: 10Mvolz) [20:59:40] (03CR) 10Ppchelko: [C: 03+1] "Feel free to self-merge when deploying" [deployment-charts] - 10https://gerrit.wikimedia.org/r/582861 (owner: 10Mvolz) [21:00:04] Reedy and sbassett: #bothumor My software never has bugs. It just develops random features. Rise for Weekly Security deployment window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200323T2100). [21:08:34] Now that is a lot of my code [21:12:35] (03PS1) 10Dwisehaupt: Add frnetmon1001 as a host to check [puppet] - 10https://gerrit.wikimedia.org/r/582912 (https://phabricator.wikimedia.org/T232137) [21:29:07] (03CR) 10Andrew Bogott: [C: 03+1] openstack: Configure WSGI to use horizon virtualenv [puppet] - 10https://gerrit.wikimedia.org/r/582907 (https://phabricator.wikimedia.org/T240852) (owner: 10Jhedden)