[01:50:12] PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job=netbox_device_statistics site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [01:52:34] RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [03:18:28] (03PS1) 10Andrew Bogott: Horizon: put into maintenance mode for Ussuri upgrade [puppet] - 10https://gerrit.wikimedia.org/r/676847 (https://phabricator.wikimedia.org/T261136) [03:18:29] (03PS1) 10Andrew Bogott: Openstack eqiad1 -> Ussuri [puppet] - 10https://gerrit.wikimedia.org/r/676848 (https://phabricator.wikimedia.org/T261136) [03:18:31] (03PS1) 10Andrew Bogott: Revert "Horizon: put into maintenance mode for Ussuri upgrade" [puppet] - 10https://gerrit.wikimedia.org/r/676849 [03:19:54] PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job=atlas_exporter site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [03:22:16] RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [05:11:00] (03CR) 10Marostegui: [C: 03+2] add growthexperiments_mentor_mentee to $private_tables [puppet] - 10https://gerrit.wikimedia.org/r/675185 (https://phabricator.wikimedia.org/T278573) (owner: 10Urbanecm) [05:12:02] !log Restart all sanitarium hosts to pick up new filters T278573 [05:12:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:12:12] T278573: Create growthexperiments_mentor_mentee database table on extension1 for wikis in growthexperiments.dblist - https://phabricator.wikimedia.org/T278573 [05:30:00] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1121 for schema change', diff saved to https://phabricator.wikimedia.org/P15111 and previous config saved to /var/cache/conftool/dbconfig/20210405-053000-marostegui.json [05:30:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:30:27] !log Deploy schema change on db1121, lag will appear on s4 on wikireplicas [05:30:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:48:02] (03PS1) 10Marostegui: db1181: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/676853 (https://phabricator.wikimedia.org/T275633) [05:48:38] (03CR) 10Marostegui: [C: 03+2] db1181: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/676853 (https://phabricator.wikimedia.org/T275633) (owner: 10Marostegui) [05:49:52] !log marostegui@cumin1001 dbctl commit (dc=all): 'Pool db1181 in s7 with minimal weight T275633', diff saved to https://phabricator.wikimedia.org/P15112 and previous config saved to /var/cache/conftool/dbconfig/20210405-054951-marostegui.json [05:49:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:50:00] T275633: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 [06:47:27] !log marostegui@cumin1001 dbctl commit (dc=all): 'Pool db1181 in s7 with minimal weight T275633', diff saved to https://phabricator.wikimedia.org/P15113 and previous config saved to /var/cache/conftool/dbconfig/20210405-064727-marostegui.json [06:47:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:47:36] T275633: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 [07:08:46] (03PS16) 10Ayounsi: Add Capirca support to Homer [software/homer] - 10https://gerrit.wikimedia.org/r/663536 (https://phabricator.wikimedia.org/T273865) [07:09:45] (03CR) 10Ayounsi: [C: 03+2] Add Capirca support to Homer (031 comment) [software/homer] - 10https://gerrit.wikimedia.org/r/663536 (https://phabricator.wikimedia.org/T273865) (owner: 10Ayounsi) [07:13:54] (03Merged) 10jenkins-bot: Add Capirca support to Homer [software/homer] - 10https://gerrit.wikimedia.org/r/663536 (https://phabricator.wikimedia.org/T273865) (owner: 10Ayounsi) [07:20:12] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1121 (re)pooling @ 25%: Repool after schema change', diff saved to https://phabricator.wikimedia.org/P15114 and previous config saved to /var/cache/conftool/dbconfig/20210405-072012-root.json [07:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:35:16] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1121 (re)pooling @ 50%: Repool after schema change', diff saved to https://phabricator.wikimedia.org/P15115 and previous config saved to /var/cache/conftool/dbconfig/20210405-073515-root.json [07:35:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:50:19] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1121 (re)pooling @ 75%: Repool after schema change', diff saved to https://phabricator.wikimedia.org/P15116 and previous config saved to /var/cache/conftool/dbconfig/20210405-075019-root.json [07:50:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:05:23] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1121 (re)pooling @ 100%: Repool after schema change', diff saved to https://phabricator.wikimedia.org/P15117 and previous config saved to /var/cache/conftool/dbconfig/20210405-080523-root.json [08:05:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:25:21] !log marostegui@cumin1001 dbctl commit (dc=all): 'Give more weight to db1181 T275633', diff saved to https://phabricator.wikimedia.org/P15118 and previous config saved to /var/cache/conftool/dbconfig/20210405-082521-marostegui.json [08:25:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:25:30] T275633: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 [09:10:43] !log marostegui@cumin1001 dbctl commit (dc=all): 'Give more weight to db1181 T275633', diff saved to https://phabricator.wikimedia.org/P15119 and previous config saved to /var/cache/conftool/dbconfig/20210405-091043-marostegui.json [09:10:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:10:52] T275633: Productionize db21[45-52] and db11[76-84] - https://phabricator.wikimedia.org/T275633 [09:43:24] (03CR) 10Jbond: [C: 03+2] P:debmonitor: clean up absented files [puppet] - 10https://gerrit.wikimedia.org/r/676112 (owner: 10Jbond) [09:43:36] (03PS2) 10Jbond: P:debmonitor: clean up absented files [puppet] - 10https://gerrit.wikimedia.org/r/676112 [09:47:44] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1181 (re)pooling @ 25%: Pool in s7', diff saved to https://phabricator.wikimedia.org/P15120 and previous config saved to /var/cache/conftool/dbconfig/20210405-094744-root.json [09:47:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:50:36] !log Deploy schema change on s1 codfw, lag will appear in codfw - T276150 T276156 [09:50:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:50:45] T276150: Schema change to make rc_id unsigned and rc_timestamp BINARY - https://phabricator.wikimedia.org/T276150 [09:50:45] T276156: Drop default of rc_timestamp - https://phabricator.wikimedia.org/T276156 [09:58:56] (03CR) 10Jbond: [C: 03+2] C:ssh::server: refactor (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/675124 (owner: 10Jbond) [09:59:05] (03CR) 10Jbond: [C: 03+2] C:ssh::server: update parameter types [puppet] - 10https://gerrit.wikimedia.org/r/675127 (owner: 10Jbond) [10:02:12] (03CR) 10Jbond: [C: 03+2] C:ssh::server: make authorized_keys_file an Arrauy[Stdlib::Unixpath] (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/675128 (owner: 10Jbond) [10:02:28] (03CR) 10Jbond: [C: 03+2] C:ssh::server: add support for multiple listen addresses [puppet] - 10https://gerrit.wikimedia.org/r/675131 (owner: 10Jbond) [10:02:47] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1099 (s1,s8) for upgrade', diff saved to https://phabricator.wikimedia.org/P15121 and previous config saved to /var/cache/conftool/dbconfig/20210405-100246-marostegui.json [10:02:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:02:54] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1181 (re)pooling @ 50%: Pool in s7', diff saved to https://phabricator.wikimedia.org/P15122 and previous config saved to /var/cache/conftool/dbconfig/20210405-100253-root.json [10:03:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:04:42] (03PS7) 10Jbond: C:ssh::server: make authorized_keys_file an Arrauy[Stdlib::Unixpath] [puppet] - 10https://gerrit.wikimedia.org/r/675128 [10:04:51] (03PS8) 10Jbond: C:ssh::server: add support for multiple listen addresses [puppet] - 10https://gerrit.wikimedia.org/r/675131 [10:04:59] (03PS7) 10Jbond: O:gitlab: restrict gitlab ssh to only listen on the primary ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/675135 [10:08:49] (03CR) 10Jbond: [C: 03+2] O:gitlab: restrict gitlab ssh to only listen on the primary ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/675135 (owner: 10Jbond) [10:12:05] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 25%: Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P15123 and previous config saved to /var/cache/conftool/dbconfig/20210405-101204-root.json [10:12:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:12:13] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 25%: Repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P15124 and previous config saved to /var/cache/conftool/dbconfig/20210405-101213-root.json [10:12:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:17:57] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1181 (re)pooling @ 75%: Pool in s7', diff saved to https://phabricator.wikimedia.org/P15125 and previous config saved to /var/cache/conftool/dbconfig/20210405-101757-root.json [10:18:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:20:12] (03PS1) 10Jbond: ssh::server: Support additional options for PermitRootLogin [puppet] - 10https://gerrit.wikimedia.org/r/676868 [10:23:52] (03PS2) 10Jbond: ssh::server: Support additional options for PermitRootLogin [puppet] - 10https://gerrit.wikimedia.org/r/676868 [10:24:39] (03CR) 10Jbond: [V: 03+1] "PCC SUCCESS (NOOP 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/28900/console" [puppet] - 10https://gerrit.wikimedia.org/r/676868 (owner: 10Jbond) [10:26:05] (03PS1) 10Jgiannelos: Bump helm chart version for chromium-render [deployment-charts] - 10https://gerrit.wikimedia.org/r/676869 [10:27:08] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 50%: Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P15126 and previous config saved to /var/cache/conftool/dbconfig/20210405-102708-root.json [10:27:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:27:24] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 50%: Repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P15127 and previous config saved to /var/cache/conftool/dbconfig/20210405-102724-root.json [10:27:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:28:48] (03PS2) 10Jgiannelos: Bump helm chart version for chromium-render [deployment-charts] - 10https://gerrit.wikimedia.org/r/676869 (https://phabricator.wikimedia.org/T277857) [10:33:01] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1181 (re)pooling @ 100%: Pool in s7', diff saved to https://phabricator.wikimedia.org/P15128 and previous config saved to /var/cache/conftool/dbconfig/20210405-103301-root.json [10:33:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:33:18] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1113 (s5,s6) for upgrade', diff saved to https://phabricator.wikimedia.org/P15129 and previous config saved to /var/cache/conftool/dbconfig/20210405-103318-marostegui.json [10:33:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:33:41] (03CR) 10Jbond: hieradata: allow monitoring.wmflabs.org to use Cloud IDP (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/676284 (owner: 10Filippo Giunchedi) [10:40:11] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1113 (s5,s6) after upgrade', diff saved to https://phabricator.wikimedia.org/P15130 and previous config saved to /var/cache/conftool/dbconfig/20210405-104010-marostegui.json [10:40:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:42:12] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 75%: Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P15131 and previous config saved to /var/cache/conftool/dbconfig/20210405-104211-root.json [10:42:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:42:28] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 75%: Repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P15132 and previous config saved to /var/cache/conftool/dbconfig/20210405-104227-root.json [10:42:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:57:16] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 100%: Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P15133 and previous config saved to /var/cache/conftool/dbconfig/20210405-105715-root.json [10:57:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:57:31] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3318 (re)pooling @ 100%: Repool db1099:3318', diff saved to https://phabricator.wikimedia.org/P15134 and previous config saved to /var/cache/conftool/dbconfig/20210405-105731-root.json [10:57:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:02:38] (03PS3) 10ArielGlenn: preclaim job fragments before claiming them [dumps] - 10https://gerrit.wikimedia.org/r/676632 (https://phabricator.wikimedia.org/T252396) [11:02:50] (03CR) 10jerkins-bot: [V: 04-1] preclaim job fragments before claiming them [dumps] - 10https://gerrit.wikimedia.org/r/676632 (https://phabricator.wikimedia.org/T252396) (owner: 10ArielGlenn) [11:03:23] (03PS1) 10Alexandros Kosiaris: scb: Remove all mentions [puppet] - 10https://gerrit.wikimedia.org/r/676881 (https://phabricator.wikimedia.org/T275759) [11:03:31] (03CR) 10jerkins-bot: [V: 04-1] scb: Remove all mentions [puppet] - 10https://gerrit.wikimedia.org/r/676881 (https://phabricator.wikimedia.org/T275759) (owner: 10Alexandros Kosiaris) [11:04:52] (03PS2) 10Alexandros Kosiaris: scb: Remove all mentions [puppet] - 10https://gerrit.wikimedia.org/r/676881 (https://phabricator.wikimedia.org/T275759) [11:05:01] (03CR) 10jerkins-bot: [V: 04-1] scb: Remove all mentions [puppet] - 10https://gerrit.wikimedia.org/r/676881 (https://phabricator.wikimedia.org/T275759) (owner: 10Alexandros Kosiaris) [11:05:06] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1119 for schema change', diff saved to https://phabricator.wikimedia.org/P15135 and previous config saved to /var/cache/conftool/dbconfig/20210405-110506-marostegui.json [11:05:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:06:11] !log akosiaris@cumin1001 START - Cookbook sre.hosts.decommission for hosts scb[2001-2006].codfw.wmnet [11:06:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:06:22] !log akosiaris@cumin1001 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts scb[2001-2006].codfw.wmnet [11:06:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:08:46] (03CR) 10Alexandros Kosiaris: [C: 03+2] "This looks like CI is malfunctioning. I am gonna override it, it's death of an old cluster anyway" [puppet] - 10https://gerrit.wikimedia.org/r/676881 (https://phabricator.wikimedia.org/T275759) (owner: 10Alexandros Kosiaris) [11:08:53] (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] scb: Remove all mentions [puppet] - 10https://gerrit.wikimedia.org/r/676881 (https://phabricator.wikimedia.org/T275759) (owner: 10Alexandros Kosiaris) [11:09:02] (03PS1) 10Effie Mouzeli: hieradata: remove parsoidJS from production 1 [puppet] - 10https://gerrit.wikimedia.org/r/676883 (https://phabricator.wikimedia.org/T279059) [11:09:11] (03CR) 10jerkins-bot: [V: 04-1] hieradata: remove parsoidJS from production 1 [puppet] - 10https://gerrit.wikimedia.org/r/676883 (https://phabricator.wikimedia.org/T279059) (owner: 10Effie Mouzeli) [11:09:58] !log akosiaris@cumin1001 START - Cookbook sre.hosts.decommission for hosts scb[2001-2006].codfw.wmnet [11:10:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:10:11] jouncebot: now [11:10:11] No deployments scheduled for the forseeable future! [11:10:33] jouncebot: refresh [11:10:35] I refreshed my knowledge about deployments. [11:10:38] jouncebot: now [11:10:38] No deployments scheduled for the forseeable future! [11:11:03] Krinkle: maybe related to the new deploy calendar format? [11:11:10] (which is great btw) [11:11:32] anyway, I'll do the deploy [11:11:36] !log akosiaris@cumin1001 START - Cookbook sre.hosts.decommission for hosts scb[1001-1004].eqiad.wmnet [11:11:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:12:36] (03CR) 10Alexandros Kosiaris: [C: 03+1] "CI doesn't like you today, but +1" [puppet] - 10https://gerrit.wikimedia.org/r/676883 (https://phabricator.wikimedia.org/T279059) (owner: 10Effie Mouzeli) [11:15:06] (03CR) 10Gergő Tisza: [C: 03+2] Fix growthexperiments.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676654 (https://phabricator.wikimedia.org/T275171) (owner: 10Gergő Tisza) [11:15:12] (03CR) 10jerkins-bot: [V: 04-1] Fix growthexperiments.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676654 (https://phabricator.wikimedia.org/T275171) (owner: 10Gergő Tisza) [11:15:55] (03PS1) 10Jbond: hiera cloud: update sso-grafana to match production [puppet] - 10https://gerrit.wikimedia.org/r/676885 [11:16:04] (03CR) 10jerkins-bot: [V: 04-1] hiera cloud: update sso-grafana to match production [puppet] - 10https://gerrit.wikimedia.org/r/676885 (owner: 10Jbond) [11:16:14] (03CR) 10Gergő Tisza: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676654 (https://phabricator.wikimedia.org/T275171) (owner: 10Gergő Tisza) [11:16:21] (03CR) 10jerkins-bot: [V: 04-1] Fix growthexperiments.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676654 (https://phabricator.wikimedia.org/T275171) (owner: 10Gergő Tisza) [11:17:34] (03PS2) 10Gergő Tisza: Fix growthexperiments.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676654 (https://phabricator.wikimedia.org/T275171) [11:18:35] huh. Is mediawiki-config CI broken? [11:19:00] (03CR) 10Gergő Tisza: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676654 (https://phabricator.wikimedia.org/T275171) (owner: 10Gergő Tisza) [11:21:12] tgr_: seems to be a problem with other repos as well :/ [11:22:09] (03PS2) 10Jbond: hiera cloud: update sso-grafana to match production [puppet] - 10https://gerrit.wikimedia.org/r/676885 [11:22:18] (03CR) 10jerkins-bot: [V: 04-1] hiera cloud: update sso-grafana to match production [puppet] - 10https://gerrit.wikimedia.org/r/676885 (owner: 10Jbond) [11:24:08] (03PS1) 10Effie Mouzeli: hieradata: remove parsoidJS from production 2 [puppet] - 10https://gerrit.wikimedia.org/r/676886 (https://phabricator.wikimedia.org/T279059) [11:24:18] (03CR) 10jerkins-bot: [V: 04-1] hieradata: remove parsoidJS from production 2 [puppet] - 10https://gerrit.wikimedia.org/r/676886 (https://phabricator.wikimedia.org/T279059) (owner: 10Effie Mouzeli) [11:28:18] just noticed on dumps repo [11:30:41] (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] reprepro: use a different key for the k8s repo [puppet] - 10https://gerrit.wikimedia.org/r/676342 (https://phabricator.wikimedia.org/T279042) (owner: 10David Caro) [11:43:57] (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] gridengine: cleaner release default [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/676033 (https://phabricator.wikimedia.org/T278748) (owner: 10Arturo Borrero Gonzalez) [11:50:38] (03PS1) 10Effie Mouzeli: admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676892 (https://phabricator.wikimedia.org/T278265) [11:50:48] (03CR) 10jerkins-bot: [V: 04-1] admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676892 (https://phabricator.wikimedia.org/T278265) (owner: 10Effie Mouzeli) [11:51:05] (03Abandoned) 10Effie Mouzeli: (WIP) modules: remove parsoidJS from puppet [puppet] - 10https://gerrit.wikimedia.org/r/676071 (https://phabricator.wikimedia.org/T279059) (owner: 10Effie Mouzeli) [11:52:40] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1119 (re)pooling @ 25%: Repool after schema change', diff saved to https://phabricator.wikimedia.org/P15136 and previous config saved to /var/cache/conftool/dbconfig/20210405-115240-root.json [11:52:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:53:13] (03PS1) 10Effie Mouzeli: admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676893 (https://phabricator.wikimedia.org/T278265) [11:53:17] (03Abandoned) 10Effie Mouzeli: admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676892 (https://phabricator.wikimedia.org/T278265) (owner: 10Effie Mouzeli) [11:53:22] (03CR) 10jerkins-bot: [V: 04-1] admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676893 (https://phabricator.wikimedia.org/T278265) (owner: 10Effie Mouzeli) [12:00:30] RECOVERY - Check systemd state on stat1007 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:03:02] !log akosiaris@cumin1001 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts scb[2001-2006].codfw.wmnet [12:03:03] !log akosiaris@cumin1001 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts scb[1001-1004].eqiad.wmnet [12:03:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:03:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:06:09] (03PS1) 10Alexandros Kosiaris: Remove site.pp scb entries [puppet] - 10https://gerrit.wikimedia.org/r/676906 (https://phabricator.wikimedia.org/T275759) [12:06:25] (03CR) 10jerkins-bot: [V: 04-1] Remove site.pp scb entries [puppet] - 10https://gerrit.wikimedia.org/r/676906 (https://phabricator.wikimedia.org/T275759) (owner: 10Alexandros Kosiaris) [12:07:44] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1119 (re)pooling @ 50%: Repool after schema change', diff saved to https://phabricator.wikimedia.org/P15137 and previous config saved to /var/cache/conftool/dbconfig/20210405-120744-root.json [12:07:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:07:56] PROBLEM - Check systemd state on stat1007 is CRITICAL: CRITICAL - degraded: The following units failed: wmde-toolkit-analyzer-build.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:09:59] (03CR) 10Jbond: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/676885 (owner: 10Jbond) [12:22:48] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1119 (re)pooling @ 75%: Repool after schema change', diff saved to https://phabricator.wikimedia.org/P15138 and previous config saved to /var/cache/conftool/dbconfig/20210405-122247-root.json [12:22:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:23:01] (03CR) 10Abijeet Patro: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676909 (https://phabricator.wikimedia.org/T277965) (owner: 10Abijeet Patro) [12:31:20] RECOVERY - Check systemd state on pki2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:37:51] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1119 (re)pooling @ 100%: Repool after schema change', diff saved to https://phabricator.wikimedia.org/P15139 and previous config saved to /var/cache/conftool/dbconfig/20210405-123751-root.json [12:37:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:38:18] PROBLEM - Check systemd state on pki2001 is CRITICAL: CRITICAL - degraded: The following units failed: cfssl-ocsprefresh-Wikimedia_Internal_Root_CA.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:39:02] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1006-cloudelastic-psi-eqiad on cloudelastic1006 is CRITICAL: 114.9 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-psi-eqiad&var-instance=cloudelastic1006&panelId=37 [12:41:19] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1099:3311 for schema change', diff saved to https://phabricator.wikimedia.org/P15140 and previous config saved to /var/cache/conftool/dbconfig/20210405-124118-marostegui.json [12:41:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:43:42] (03CR) 10ArielGlenn: "recheck" [dumps] - 10https://gerrit.wikimedia.org/r/676632 (https://phabricator.wikimedia.org/T252396) (owner: 10ArielGlenn) [12:46:20] (03CR) 10Jbond: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/676885 (owner: 10Jbond) [12:49:21] (03PS4) 10ArielGlenn: preclaim job fragments before claiming them [dumps] - 10https://gerrit.wikimedia.org/r/676632 (https://phabricator.wikimedia.org/T252396) [13:07:47] ACKNOWLEDGEMENT - haproxy failover on dbproxy2003 is CRITICAL: CRITICAL check_failover servers up 1 down 1 Marostegui known https://wikitech.wikimedia.org/wiki/HAProxy [13:12:21] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1141 for upgrade', diff saved to https://phabricator.wikimedia.org/P15141 and previous config saved to /var/cache/conftool/dbconfig/20210405-131221-marostegui.json [13:12:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:22:41] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1141 (re)pooling @ 25%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P15142 and previous config saved to /var/cache/conftool/dbconfig/20210405-132240-root.json [13:22:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:23:14] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 25%: Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P15143 and previous config saved to /var/cache/conftool/dbconfig/20210405-132314-root.json [13:23:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:27:58] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1006-cloudelastic-psi-eqiad on cloudelastic1006 is OK: (C)100 gt (W)80 gt 13.22 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-psi-eqiad&var-instance=cloudelastic1006&panelId=37 [13:32:12] (03PS1) 10Ottomata: dumps.wikimedia.org - add section in legal specifying CC0 license for analytics [puppet] - 10https://gerrit.wikimedia.org/r/676922 (https://phabricator.wikimedia.org/T278409) [13:37:44] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1141 (re)pooling @ 50%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P15144 and previous config saved to /var/cache/conftool/dbconfig/20210405-133744-root.json [13:37:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:38:18] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 50%: Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P15145 and previous config saved to /var/cache/conftool/dbconfig/20210405-133818-root.json [13:38:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:41:13] (03PS1) 10Amire80: Add default import sources [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676930 (https://phabricator.wikimedia.org/T214139) [13:47:26] (03PS17) 10Jbond: gerrit: Add ed25519 and ecdsa ssh host keys [puppet] - 10https://gerrit.wikimedia.org/r/556270 (https://phabricator.wikimedia.org/T240266) (owner: 10Paladox) [13:52:48] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1141 (re)pooling @ 75%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P15146 and previous config saved to /var/cache/conftool/dbconfig/20210405-135248-root.json [13:52:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:53:22] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 75%: Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P15147 and previous config saved to /var/cache/conftool/dbconfig/20210405-135321-root.json [13:53:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:59:19] tgr|away: yeah, might be that. [14:00:22] https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/bots/jouncebot/+/refs/heads/master/jouncebot/deploypage.py [14:00:33] Looks like it scrapes parser output html [14:00:42] I'll try patching it later [14:00:57] Or might review someone's patch if they get to it first :) [14:07:52] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1141 (re)pooling @ 100%: Repool after upgrade', diff saved to https://phabricator.wikimedia.org/P15148 and previous config saved to /var/cache/conftool/dbconfig/20210405-140751-root.json [14:08:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:08:25] !log marostegui@cumin1001 dbctl commit (dc=all): 'db1099:3311 (re)pooling @ 100%: Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P15149 and previous config saved to /var/cache/conftool/dbconfig/20210405-140825-root.json [14:08:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:09:45] (03PS1) 10Jbond: ssh::server us legacy facts for ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/676933 [14:11:01] (03CR) 10Jbond: [V: 03+1] "PCC SUCCESS (NOOP 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/28901/console" [puppet] - 10https://gerrit.wikimedia.org/r/676933 (owner: 10Jbond) [14:12:05] (03PS2) 10Jbond: ssh::server us legacy facts for ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/676933 (https://phabricator.wikimedia.org/T279283) [14:14:07] (03PS3) 10Jbond: ssh::server us legacy facts for ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/676933 (https://phabricator.wikimedia.org/T279283) [14:14:36] (03PS4) 10Jbond: ssh::server us legacy facts for ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/676933 (https://phabricator.wikimedia.org/T279283) [14:16:35] (03PS5) 10Jbond: ssh::server us legacy facts for ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/676933 [14:17:35] (03CR) 10Abijeet Patro: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676909 (https://phabricator.wikimedia.org/T277965) (owner: 10Abijeet Patro) [14:20:27] (03PS2) 10Alexandros Kosiaris: Segment values.yaml between teams [deployment-charts] - 10https://gerrit.wikimedia.org/r/675558 (https://phabricator.wikimedia.org/T278208) (owner: 10Elukey) [14:20:29] (03PS1) 10Alexandros Kosiaris: admin: Switch values/values.yaml to common.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/676935 [14:20:31] (03PS1) 10Alexandros Kosiaris: admin: Switch usages to internal kubernetes API, with exceptions [deployment-charts] - 10https://gerrit.wikimedia.org/r/676936 [14:20:35] (03CR) 10jerkins-bot: [V: 04-1] Segment values.yaml between teams [deployment-charts] - 10https://gerrit.wikimedia.org/r/675558 (https://phabricator.wikimedia.org/T278208) (owner: 10Elukey) [14:20:37] (03CR) 10jerkins-bot: [V: 04-1] admin: Switch values/values.yaml to common.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/676935 (owner: 10Alexandros Kosiaris) [14:20:39] (03CR) 10jerkins-bot: [V: 04-1] admin: Switch usages to internal kubernetes API, with exceptions [deployment-charts] - 10https://gerrit.wikimedia.org/r/676936 (owner: 10Alexandros Kosiaris) [14:27:14] (03PS6) 10Jbond: ssh::server us legacy facts for ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/676933 [14:34:18] PROBLEM - Postgres Replication Lag on puppetdb2002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 91452360 and 23 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [14:36:50] RECOVERY - Postgres Replication Lag on puppetdb2002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 0 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [14:43:35] !log push pfw policies - T278970 [14:43:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:49:27] (03CR) 10CDanis: [C: 03+1] ssh::server us legacy facts for ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/676933 (owner: 10Jbond) [14:53:21] (03CR) 10Jbond: [C: 03+2] ssh::server us legacy facts for ip addresses [puppet] - 10https://gerrit.wikimedia.org/r/676933 (owner: 10Jbond) [14:58:35] (03Abandoned) 10Alexandros Kosiaris: ml-serve: Switch to stretch, buster isn't yet supported [puppet] - 10https://gerrit.wikimedia.org/r/666684 (owner: 10Alexandros Kosiaris) [14:59:03] (03Abandoned) 10Alexandros Kosiaris: Bump chart versions for netpol bump [deployment-charts] - 10https://gerrit.wikimedia.org/r/602706 (owner: 10Alexandros Kosiaris) [15:10:05] (03PS2) 10Effie Mouzeli: admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676893 (https://phabricator.wikimedia.org/T278265) [15:10:45] (03CR) 10jerkins-bot: [V: 04-1] admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676893 (https://phabricator.wikimedia.org/T278265) (owner: 10Effie Mouzeli) [15:11:25] (03CR) 10Abijeet Patro: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676909 (https://phabricator.wikimedia.org/T277965) (owner: 10Abijeet Patro) [15:11:39] (03CR) 10MSantos: [C: 03+2] Bump helm chart version for chromium-render [deployment-charts] - 10https://gerrit.wikimedia.org/r/676869 (https://phabricator.wikimedia.org/T277857) (owner: 10Jgiannelos) [15:13:32] (03Merged) 10jenkins-bot: Bump helm chart version for chromium-render [deployment-charts] - 10https://gerrit.wikimedia.org/r/676869 (https://phabricator.wikimedia.org/T277857) (owner: 10Jgiannelos) [15:20:32] (03Restored) 10Effie Mouzeli: admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676892 (https://phabricator.wikimedia.org/T278265) (owner: 10Effie Mouzeli) [15:20:43] (03PS2) 10Effie Mouzeli: admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676892 (https://phabricator.wikimedia.org/T278265) [15:21:12] (03CR) 10jerkins-bot: [V: 04-1] admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676892 (https://phabricator.wikimedia.org/T278265) (owner: 10Effie Mouzeli) [15:25:30] (03PS2) 10Effie Mouzeli: hieradata: remove parsoidJS from production 1 [puppet] - 10https://gerrit.wikimedia.org/r/676883 (https://phabricator.wikimedia.org/T279059) [15:39:35] (03CR) 10Umherirrender: "recheck - T279283" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676654 (https://phabricator.wikimedia.org/T275171) (owner: 10Gergő Tisza) [15:39:41] (03CR) 10Umherirrender: "recheck - T279283" [puppet] - 10https://gerrit.wikimedia.org/r/676886 (https://phabricator.wikimedia.org/T279059) (owner: 10Effie Mouzeli) [15:52:05] (03PS2) 10Alexandros Kosiaris: admin: Switch values/values.yaml to common.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/676935 [15:52:07] (03PS2) 10Alexandros Kosiaris: admin: Switch usages to internal kubernetes API, with exceptions [deployment-charts] - 10https://gerrit.wikimedia.org/r/676936 [15:52:09] (03PS3) 10Alexandros Kosiaris: Segment values.yaml between teams [deployment-charts] - 10https://gerrit.wikimedia.org/r/675558 (https://phabricator.wikimedia.org/T278208) (owner: 10Elukey) [16:44:58] !log Start server-side upload of 4 video files (T279204, T279201, T279200, T279198) [16:45:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:45:14] T279204: Server side upload for Sturm - https://phabricator.wikimedia.org/T279204 [16:45:14] T279200: Server side upload for Sturm - https://phabricator.wikimedia.org/T279200 [16:45:14] T279201: Server side upload for Sturm - https://phabricator.wikimedia.org/T279201 [16:45:14] T279198: Server side upload for Sturm - https://phabricator.wikimedia.org/T279198 [16:48:30] (03Abandoned) 10Effie Mouzeli: admin: add bwang and cjming to ldap_only users [puppet] - 10https://gerrit.wikimedia.org/r/676892 (https://phabricator.wikimedia.org/T278265) (owner: 10Effie Mouzeli) [16:50:14] (03CR) 10Sascha: "Thank you! Just a minor nitpick:" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/676922 (https://phabricator.wikimedia.org/T278409) (owner: 10Ottomata) [17:05:48] !log dpifke@deploy1002 Started deploy [performance/navtiming@bc5af87]: Deploy https://gerrit.wikimedia.org/r/c/performance/navtiming/+/676006 [17:05:53] !log dpifke@deploy1002 Finished deploy [performance/navtiming@bc5af87]: Deploy https://gerrit.wikimedia.org/r/c/performance/navtiming/+/676006 (duration: 00m 05s) [17:05:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:06:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:19:26] (03CR) 10Bstorm: toolforge.checker: Update list of etcd nodes (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/676409 (https://phabricator.wikimedia.org/T267082) (owner: 10David Caro) [17:20:36] (03PS1) 10CDanis: Revert "upload-lb: block python-requests UA on AWS" [puppet] - 10https://gerrit.wikimedia.org/r/676947 [17:20:50] (03CR) 10jerkins-bot: [V: 04-1] Revert "upload-lb: block python-requests UA on AWS" [puppet] - 10https://gerrit.wikimedia.org/r/676947 (owner: 10CDanis) [17:36:30] !log pt1979@cumin2001 START - Cookbook sre.dns.netbox [17:36:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:39:47] !log pt1979@cumin2001 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [17:39:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:59:37] jouncebot: next [17:59:37] No deployments scheduled for the forseeable future! [18:00:01] jouncebot: now [18:00:01] No deployments scheduled for the forseeable future! [18:00:07] someone broke jouncebot it seems [18:00:09] wow [18:00:14] it's supposed to be B&C now [18:00:20] or else we're on a very extended vacation.... [18:00:34] i think it was the redesign of [[wikitech:Deployments]] [18:00:38] i guess I'll try to revert it... [18:00:41] jouncebot is more important [18:00:57] awww :-( [18:01:40] tgr|away: you have a patch scheduled, wanna self-service, or should I deploy? [18:01:42] will the edits people made still stick? [18:01:56] apergos: i think it was made somewhere in the backend of how the HTML is generated, but i can't find where [18:03:55] I'll help you hunt i a minute, let me put water for pasta first [18:06:24] Urbanecm: it's a releng tool repo I think [18:06:28] i think i have it [18:06:31] RhinosF1: no, it's on wiki [18:06:34] i found it [18:06:52] Ok [18:08:53] https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=1905616&oldid=1905613 this [18:08:59] oh you found it already [18:12:02] Urbanecm: I can do it, thanks [18:12:08] go for it [18:12:26] let's see if the bot can be fixed? the new format is a lot more convenient [18:13:29] tgr_: maybe, I'm trying to revert first and figure out how to do it with the new format later [18:14:30] but so far i can't figure out a simple way :/ [18:19:02] https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/bots/jouncebot/+/refs/heads/master/jouncebot/deploypage.py is what will need updating I think [18:19:17] yeah [18:19:21] * Urbanecm trying to do it [18:19:28] that class is nasty, but hopefully readable [18:19:32] (03CR) 10Gergő Tisza: [C: 03+2] Fix growthexperiments.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676654 (https://phabricator.wikimedia.org/T275171) (owner: 10Gergő Tisza) [18:20:48] (03Merged) 10jenkins-bot: Fix growthexperiments.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/676654 (https://phabricator.wikimedia.org/T275171) (owner: 10Gergő Tisza) [18:20:50] (03PS1) 10Herron: point wikimania scholarships to mwlog1002 [puppet] - 10https://gerrit.wikimedia.org/r/676995 (https://phabricator.wikimedia.org/T224565) [18:20:52] (03PS1) 10Herron: deploy logster_alarm to mwlog1002 [puppet] - 10https://gerrit.wikimedia.org/r/676996 (https://phabricator.wikimedia.org/T224565) [18:20:54] (03PS1) 10Herron: add mwlog[12]002 to profile::dumps::rsync_internal_clients [puppet] - 10https://gerrit.wikimedia.org/r/676997 (https://phabricator.wikimedia.org/T224565) [18:22:40] instead of trying to parse HTML we should probably just make the template output the event data in some sane machine-readable format [18:22:59] so the bot doesn't have to be updated every time the page is restyled [18:23:12] (which, granted, doesn't happen often, but still) [18:23:43] tgr_: sounds like a good idea [18:24:53] output JSON with display: none [18:25:58] yeah [18:26:25] TestUrbanecm: now [18:26:25] No deployments scheduled for the forseeable future! [18:27:01] !log tgr@deploy1002 Synchronized wmf-config/config/frwiki.yaml: Config: [[gerrit:676654|Fix growthexperiments.dblist (T275171)]] (duration: 00m 59s) [18:27:02] or rdf, or json-ld, or json in the value attribute of a meta tag... [18:27:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:27:10] T275171: Growth: shorten welcome survey retention to 90 days - https://phabricator.wikimedia.org/T275171 [18:28:31] !log tgr@deploy1002 Synchronized dblists/growthexperiments.dblist: Config: [[gerrit:676654|Fix growthexperiments.dblist (T275171)]] (duration: 00m 58s) [18:28:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:28:44] !log Morning deploys done [18:28:49] Reedy: do you _really_ want racktables or do you just want netbox.wikimedia.org access? (https://phabricator.wikimedia.org/T279244#6972902) [18:28:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:29:01] for some value of morning. [18:29:17] Reedy: information in netbox should have replaced anything you could have gotten from racktables [18:29:27] mutante: heh [18:29:35] I had racktables access for years :D [18:29:48] Reedy: the question I have now is.. can you access netbox [18:30:14] I can, yeah [18:30:21] either it's just fixed by that or that ticket should turn into access to netbox [18:30:24] i guess [18:30:37] ah, cool. but does it give you what you wanted from racktables? [18:30:50] It's the usual.. The old service is still up, muscle memory (even if I don't use it regularly) is to go to racktables to look something up, because that's what I did before [18:30:58] ack [18:31:00] And the service is still up, even if in a limited fashion [18:31:07] yea, it's like RT [18:31:13] the other "rt" [18:31:16] right [18:31:30] There's nothing on the banner for racktables going "hey, maybe you want netbox" etc [18:31:44] alright, but _if_ netbox is a working replacement to get you the actual thing [18:31:52] yes, true [18:32:34] if $IP ~ Robh then login as admin; else redirect to netbox [18:34:40] Urbanecm: already working on that? if not, I can give it a shot [18:34:51] currently trying to get JSON to the page [18:38:13] (03PS1) 10Ottomata: Configure Notebook Terminal to not use login shell [puppet] - 10https://gerrit.wikimedia.org/r/676999 (https://phabricator.wikimedia.org/T224658) [18:38:42] (03CR) 10jerkins-bot: [V: 04-1] Configure Notebook Terminal to not use login shell [puppet] - 10https://gerrit.wikimedia.org/r/676999 (https://phabricator.wikimedia.org/T224658) (owner: 10Ottomata) [18:39:00] Scribunto has JSON functions btw, which is why I suggested it [18:39:08] totally didn't abuse this somewhere else >.> [18:39:30] legoktm: with the change, the full deployment item specification doesn't even reach scribunto [18:39:35] so I'm doing that in wikitext [18:39:37] :/ [18:40:02] I think Scribunto can get it anyways? with frame:getParent() or something [18:40:04] mutante: we should probabyl just put a banner at least [18:40:43] legoktm: afaics https://wikitech.wikimedia.org/w/index.php?title=Template:Deployment_calendar_event_card has the logic for building the event card [18:40:45] not scribunto [18:41:20] that still has {{#invoke:Deployment schedule [18:41:30] so in theory Deployment schedule can still access it [18:41:39] yes, but not with all arguments [18:41:47] chaomodus: yea, but banner in racktables itself or banner on the CAS page? [18:41:54] it only calls {{#invoke:Deployment schedule|when, which has start and hours [18:42:13] https://www.mediawiki.org/wiki/Extension:Scribunto/Lua_reference_manual#frame:getParent [18:42:32] as long as {{Deployment calendar event card}} or whatever calls that has the real args, Scribunto can still get to them [18:42:38] interesting [18:45:30] chaomodus: since even Rob said he hasn't used it in 1 year and it's only up in case that netbox is down.. maybe we can just... question of how much longer it should stay in general [18:46:09] and rewrite URL to netbox [18:47:02] (03PS1) 10Herron: replace mwlog1001 with new mwlog[12]002 hosts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/677002 (https://phabricator.wikimedia.org/T224565) [18:47:07] a banner in racktables itself would not have prevented that ticket because that would be after CAS login [18:47:40] mutante: i guess i on the page itself, i dunno about adding stuff on a per-site basis to CAS [18:48:03] mutante: i guess i on the page itself, i dunno about adding stuff on a per-site basis to CAS [18:48:10] woops, silly irc. [18:49:28] My favorite fix would be that the CAS page says "I see you are trying to login at racktables, don't you mean Netbox?" and you can say "Ah, thanks, take me to netbox" or "NO, i know what I'm doing" :) [18:50:10] mutante: what about moving racktables to racktables-secret.wm.o and rewriting racktables.wm.o to netbox? [18:50:11] but maybe "shutdown racktables" better [18:50:20] mutante: hehe i think that would be ideal but idk if that's possible :) [18:50:22] yah [18:50:27] it is surely getting to its eclipse at this point [18:50:32] (03CR) 10Herron: [C: 03+1] logstash: remove logstash output on legacy cluster [puppet] - 10https://gerrit.wikimedia.org/r/676477 (https://phabricator.wikimedia.org/T234854) (owner: 10Cwhite) [18:50:57] Urbanecm: not a bad option! could do that [18:51:11] but obv only if someone actually needs racktables :) [18:51:16] racktables.og.wikimedia.org [18:52:00] in seriousness, racktables.deprecated.wikimedia.org [18:52:10] yea, not sure how good of a reason "if netbox is down" really is (anymore) [18:52:27] it's not a good reason since netbox underlies a lot of automation and such now. [18:52:41] chaomodus: yea, except we cant have a TLS cert for that easily i think [18:52:57] yea, that [18:53:31] (03PS8) 10Sharvaniharan: Add event stream config for android.image_recommendations_interaction [mediawiki-config] - 10https://gerrit.wikimedia.org/r/673005 [18:58:34] Urbanecm: it's not forbidden to use a template parameter multiple times :) just pass them to some lua invocation separately [18:59:15] tgr_: yeah, working on it [18:59:27] that said, JSON is a bit awkwarf because part of the info is in the nested ircnick template [18:59:33] yeah [18:59:35] working on that [18:59:38] I'd probably go for microdata [19:02:32] tgr_: well if you have your own solution on how to fix that, I'd gladly pass that over ;) [19:04:59] (03CR) 10Herron: [C: 03+1] logstash: set cluster name for elasticsearch outputs [puppet] - 10https://gerrit.wikimedia.org/r/676685 (https://phabricator.wikimedia.org/T274394) (owner: 10Cwhite) [19:05:42] for now I'm trying to make the json encode function to actually encode it [19:06:21] !log andrew@deploy1002 Started deploy [horizon/deploy@392708e]: Experimental main deploy of Horizon [19:06:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:07:40] Urbanecm: if you need any rights to work on the jouncebot stuff let me know and I will happily add you to the toolforge project for it or poke at gerrit configs [19:08:10] bd808: getting jouncebot running locally (in ##Urbanecm) was the quickest part of the stuff :) [19:08:11] but thanks [19:08:25] !log andrew@deploy1002 Finished deploy [horizon/deploy@392708e]: Experimental main deploy of Horizon (duration: 02m 04s) [19:08:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:12:17] (03PS2) 10BBlack: Revert "upload-lb: block python-requests UA on AWS" [puppet] - 10https://gerrit.wikimedia.org/r/676947 (owner: 10CDanis) [19:13:26] (03CR) 10BBlack: [C: 03+2] Revert "upload-lb: block python-requests UA on AWS" [puppet] - 10https://gerrit.wikimedia.org/r/676947 (owner: 10CDanis) [19:13:37] ah, thanks for fixing that bblack! [19:14:03] np! [19:14:28] it's a strange case in terms of practice, since the commit history and blame is now slightly less than ideal :) [19:14:39] I blame the tooling for not being smarterer :P [19:14:53] (03CR) 10Herron: "looks like it will work, although I wonder if it could be done with fewer calls to cron and elasticsearch::curator::config, possibly by de" [puppet] - 10https://gerrit.wikimedia.org/r/676631 (https://phabricator.wikimedia.org/T274394) (owner: 10Cwhite) [19:16:10] hmm actually once resolved, the blame looks right (the new additions to the UA-violation clause are attributed to the correct commit) [19:16:23] the tools are smarter than they first appear! [19:20:41] (03CR) 10Herron: "should these still be configured using IPs instead of hostnames?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/677002 (https://phabricator.wikimedia.org/T224565) (owner: 10Herron) [19:21:05] (03PS2) 10Ottomata: dumps.wikimedia.org - add section in legal specifying CC0 license for analytics [puppet] - 10https://gerrit.wikimedia.org/r/676922 (https://phabricator.wikimedia.org/T278409) [19:21:17] (03CR) 10Ottomata: "Thank you!" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/676922 (https://phabricator.wikimedia.org/T278409) (owner: 10Ottomata) [19:29:59] (03CR) 10ArielGlenn: [C: 03+1] "Looks ok to me, if legal likes it." [puppet] - 10https://gerrit.wikimedia.org/r/676922 (https://phabricator.wikimedia.org/T278409) (owner: 10Ottomata) [19:30:01] !log andrew@deploy1002 Started deploy [horizon/deploy@df2b0b4]: Returning cloudweb2001-dev to Horizon/Wallaby [19:30:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:31:43] !log andrew@deploy1002 Finished deploy [horizon/deploy@df2b0b4]: Returning cloudweb2001-dev to Horizon/Wallaby (duration: 01m 41s) [19:31:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:40:55] (03PS1) 10Herron: kafka-logging: migrate broker logstash1010 to kafka-logging1001 [puppet] - 10https://gerrit.wikimedia.org/r/677009 (https://phabricator.wikimedia.org/T279342) [19:41:21] (03CR) 10jerkins-bot: [V: 04-1] kafka-logging: migrate broker logstash1010 to kafka-logging1001 [puppet] - 10https://gerrit.wikimedia.org/r/677009 (https://phabricator.wikimedia.org/T279342) (owner: 10Herron) [19:43:37] (03PS2) 10Cwhite: logstash: use curator cluster config when possible [puppet] - 10https://gerrit.wikimedia.org/r/676631 (https://phabricator.wikimedia.org/T274394) [19:52:55] (03PS1) 10Urbanecm: Attempt to make jouncebot work again [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 [19:53:13] bd808: ^^ attempted to make something that could work [19:53:51] (03CR) 10jerkins-bot: [V: 04-1] Attempt to make jouncebot work again [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 (owner: 10Urbanecm) [19:54:13] Obv i need to jenkinsify the code a bit [19:55:50] (03PS3) 10Cwhite: logstash: use curator cluster config when possible [puppet] - 10https://gerrit.wikimedia.org/r/676631 (https://phabricator.wikimedia.org/T274394) [19:55:59] (03CR) 10Herron: "https://puppet-compiler.wmflabs.org/compiler1001/28906/" [puppet] - 10https://gerrit.wikimedia.org/r/677009 (https://phabricator.wikimedia.org/T279342) (owner: 10Herron) [19:56:30] (03PS4) 10Cwhite: logstash: use curator cluster config when possible [puppet] - 10https://gerrit.wikimedia.org/r/676631 (https://phabricator.wikimedia.org/T274394) [19:57:05] (03PS5) 10Cwhite: logstash: use curator cluster config when possible [puppet] - 10https://gerrit.wikimedia.org/r/676631 (https://phabricator.wikimedia.org/T274394) [19:58:57] (03PS2) 10Urbanecm: Attempt to make jouncebot work again [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 [19:59:40] (03CR) 10jerkins-bot: [V: 04-1] Attempt to make jouncebot work again [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 (owner: 10Urbanecm) [20:01:05] (03PS3) 10Urbanecm: Attempt to make jouncebot work again [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 [20:08:23] (03CR) 10Cwhite: "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/676631 (https://phabricator.wikimedia.org/T274394) (owner: 10Cwhite) [20:09:04] bd808: legoktm: tgr_: CR of https://gerrit.wikimedia.org/r/677017 appreciated :) [20:10:14] is the phabricator bot broken? [20:10:34] cdanis: not sure what you mean by "phabricator bot". jouncebot is broken [20:10:49] cdanis: yes, I restarted it earlier today, but then I got into other stuff and never looked at it again [20:11:14] I forget which irc nick it posts under, but, the bot that posts phab task updates [20:11:46] cdanis: it's wikibugs, I already tried restarting it, but apparently it didn't help [20:12:00] oh, it's still broken? [20:12:13] more broken botz, nice [20:12:16] (03CR) 10Cwhite: [C: 03+1] deploy logster_alarm to mwlog1002 [puppet] - 10https://gerrit.wikimedia.org/r/676996 (https://phabricator.wikimedia.org/T224565) (owner: 10Herron) [20:12:55] legoktm: unfortunately I didn't have time to check that it started working directly after touching it and the forgot :/ [20:13:12] (03CR) 10Cwhite: [C: 03+1] "LGTM. The scholarships app may require a restart to take advantage of the new config value." [puppet] - 10https://gerrit.wikimedia.org/r/676995 (https://phabricator.wikimedia.org/T224565) (owner: 10Herron) [20:13:30] (03CR) 10Cwhite: [C: 03+1] add mwlog[12]002 to profile::dumps::rsync_internal_clients [puppet] - 10https://gerrit.wikimedia.org/r/676997 (https://phabricator.wikimedia.org/T224565) (owner: 10Herron) [20:13:30] s/the/then [20:14:44] * legoktm pokes around [20:15:18] anyway, going to afk briefly. Happy to test the jouncebot patch later when it's deployed. [20:16:17] (03CR) 10Cwhite: "> Patch Set 1:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/677002 (https://phabricator.wikimedia.org/T224565) (owner: 10Herron) [20:18:11] (03CR) 10BryanDavis: Attempt to make jouncebot work again (032 comments) [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 (owner: 10Urbanecm) [20:18:25] fixed [20:18:34] 10SRE, 10SRE-Access-Requests: Need access to noc@wikimedia.org (associated with Analytics' MaxMind account) - https://phabricator.wikimedia.org/T279310 (10Dzahn) Unfortunately I do not see an option to switch from CC to invoicing in the web UI. Contacting MaxMind support to ask how that is done. [20:19:32] (03CR) 10Cwhite: [C: 03+1] kafka-logging: migrate broker logstash1010 to kafka-logging1001 [puppet] - 10https://gerrit.wikimedia.org/r/677009 (https://phabricator.wikimedia.org/T279342) (owner: 10Herron) [20:19:54] PROBLEM - Prometheus jobs reduced availability on alert1001 is CRITICAL: job=netbox_device_statistics site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [20:22:26] RECOVERY - Prometheus jobs reduced availability on alert1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [20:25:55] 10SRE, 10SRE-Access-Requests: Need access to noc@wikimedia.org (associated with Analytics' MaxMind account) - https://phabricator.wikimedia.org/T279310 (10Dzahn) @JLaytonWMF Alright, so I talked to MaxMind support (after being logged in) and asked them specifically about changing to invoicing and the answer wa... [20:26:51] 10SRE, 10SRE-Access-Requests: Need access to noc@wikimedia.org (associated with Analytics' MaxMind account) - https://phabricator.wikimedia.org/T279310 (10Dzahn) a:03JLaytonWMF Let us know if this works and you don't need the web access anymore. [20:30:55] (03CR) 10Cwhite: "https://puppet-compiler.wmflabs.org/compiler1001/28908/" [puppet] - 10https://gerrit.wikimedia.org/r/676631 (https://phabricator.wikimedia.org/T274394) (owner: 10Cwhite) [20:33:29] (03PS4) 10Urbanecm: Make jouncebot work again after [[wikitech:Deployments]] redesign [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 [20:34:01] (03CR) 10Urbanecm: Make jouncebot work again after [[wikitech:Deployments]] redesign (032 comments) [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 (owner: 10Urbanecm) [20:34:08] bd808: fixed :) [20:36:21] (03PS9) 10Mholloway: Add event stream config for android.image_recommendations_interaction [mediawiki-config] - 10https://gerrit.wikimedia.org/r/673005 (owner: 10Sharvaniharan) [20:39:09] (03CR) 10Mholloway: [C: 03+2] Add event stream config for android.image_recommendations_interaction [mediawiki-config] - 10https://gerrit.wikimedia.org/r/673005 (owner: 10Sharvaniharan) [20:40:20] (03CR) 10Nskaggs: [C: 03+1] "Just one question on the arg being required..." (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/676655 (https://phabricator.wikimedia.org/T279185) (owner: 10Bstorm) [20:40:23] (03Merged) 10jenkins-bot: Add event stream config for android.image_recommendations_interaction [mediawiki-config] - 10https://gerrit.wikimedia.org/r/673005 (owner: 10Sharvaniharan) [20:43:37] !log mholloway-shell@deploy1002 Synchronized wmf-config/InitialiseSettings.php: Add event stream config for android.image_recommendation_interaction (duration: 00m 59s) [20:43:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:45:01] Urbanecm: the code looks ok. Does it work? I'm too lazy to spin up a test instance at the moment [20:45:17] Hey all - I need to re-deploy 2 security patches that were accidentally dropped a while ago. Let me know if I shouldn't. See also: https://phabricator.wikimedia.org/T279274 [20:45:29] bd808: it works locally. Can't be sure whether it works on toolforge too :D [20:46:00] Urbanecm: lets find out! :) [20:46:07] yup! [20:46:52] (03CR) 10Bstorm: wikireplicas: add a skip-dns option (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/676655 (https://phabricator.wikimedia.org/T279185) (owner: 10Bstorm) [20:47:10] sbassett: probably ok, unless mholloway (who deployed minutes ago) intends to do sth more [20:47:28] nope, all done [20:47:32] Cool, thanks [20:49:53] jouncebot: now [20:49:53] For the next 0 hour(s) and 10 minute(s): [[mw:Services|Services]] – [[mw:Extension:Graph|Graphoid]] / [[ORES]] (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210405T2000) [20:49:57] jouncebot: next [20:49:57] In 0 hour(s) and 10 minute(s): Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210405T2100) [20:50:02] !log re-deploy security patch for T270988 to wmf.37 [20:50:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:50:11] this sounds like it does what it is supposed to bd808 :) [20:50:16] Urbanecm: looks like it works. :) I'll merge the patch [20:50:23] (03CR) 10BryanDavis: [C: 03+2] Make jouncebot work again after [[wikitech:Deployments]] redesign [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 (owner: 10Urbanecm) [20:51:41] (03Merged) 10jenkins-bot: Make jouncebot work again after [[wikitech:Deployments]] redesign [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/677017 (owner: 10Urbanecm) [20:54:02] jouncebot: next [20:54:02] In 0 hour(s) and 5 minute(s): Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210405T2100) [20:54:11] thanks Urbanecm! [20:54:25] happy to help :) [20:58:26] !log re-deploy security patch for T270453 to wmf.37 [20:58:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:00:04] Reedy and sbassett: (Dis)respected human, time to deploy Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210405T2100). Please do the needful. [21:05:30] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [21:15:06] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [21:20:51] (03CR) 10Bstorm: [C: 03+2] wikireplicas: add a skip-dns option [cookbooks] - 10https://gerrit.wikimedia.org/r/676655 (https://phabricator.wikimedia.org/T279185) (owner: 10Bstorm) [21:23:42] (03Merged) 10jenkins-bot: wikireplicas: add a skip-dns option [cookbooks] - 10https://gerrit.wikimedia.org/r/676655 (https://phabricator.wikimedia.org/T279185) (owner: 10Bstorm) [21:31:49] (03PS1) 10ArielGlenn: small fixups [dumps] - 10https://gerrit.wikimedia.org/r/677031 (https://phabricator.wikimedia.org/T252396) [21:33:13] (03PS1) 10ArielGlenn: pylint fixups for batches-related code [dumps] - 10https://gerrit.wikimedia.org/r/677032 (https://phabricator.wikimedia.org/T252396) [21:37:06] 10SRE, 10Traffic, 10Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: Wikipedia iOS apps sending harmful bursts of traffic synchronized to the top of the hour, especially at 22:00 UTC - https://phabricator.wikimedia.org/T264881 (10JMinor) [21:37:25] 10SRE, 10GitLab (Initialization), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/675135 and the entire chain below limited ssh access... [21:39:15] 10SRE, 10GitLab (Initialization), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Dzahn) > This ticket is to discuss proposed solutions. @thcipriani I think this happened and we know the soluti... [21:39:46] 10SRE, 10GitLab (Initialization), 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Dzahn) 05Stalled→03Open [21:39:52] 10SRE, 10Traffic, 10GitLab (Initialization), 10Patch-For-Review, and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) [23:00:04] RoanKattouw, Niharika, and Urbanecm: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for [[Backport windows|Evening backport window]]
'''''' . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20210405T2300). [23:00:04] No GERRIT patches in the queue for this window AFAICS. [23:06:59] Urbanecm: ^ it looks like the old code was doing some data cleaning that isn't happening now [23:07:16] "[[Backport windows|Evening backport window]]
'''''' " [23:09:56] bd808: yeah. It's a known (to me) bug that is caused by intentionally nowiki'ing the processed JSON blob (because not using nowiki made it an invalid JSON blob). If jouncebot bugs are Phab tracked, could you please fill it as a task and assign it to me? I'll try to look on how to fix that. I'm off-laptop now. [23:17:27] !log Running importMissingLocalNames.php on mwmaint1002 in a screen [23:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log