[00:07:54] (03PS5) 10Mstyles: kibana: refactor kibana profile into two profiles [puppet] - 10https://gerrit.wikimedia.org/r/583414 (https://phabricator.wikimedia.org/T246961) [00:08:56] (03CR) 10Mstyles: "> Patch Set 4:" [puppet] - 10https://gerrit.wikimedia.org/r/583414 (https://phabricator.wikimedia.org/T246961) (owner: 10Mstyles) [00:10:45] (03CR) 10Mstyles: kibana: refactor kibana profile into two profiles (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583414 (https://phabricator.wikimedia.org/T246961) (owner: 10Mstyles) [00:16:32] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad on cloudelastic1004 is CRITICAL: 103.7 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1004&panelId=37 [00:18:46] PROBLEM - Check systemd state on netbox1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:40:23] (03PS2) 10CDanis: phased rollout of sensible flow-table-sizes [homer/public] - 10https://gerrit.wikimedia.org/r/583740 (https://phabricator.wikimedia.org/T248394) [00:45:58] RECOVERY - Check systemd state on netbox1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:48:52] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [01:46:24] PROBLEM - Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad on cloudelastic1004 is CRITICAL: 103.7 gt 100 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1004&panelId=37 [06:23:28] (03PS3) 10KartikMistry: apertium-ind-zlm: Fix FTBFS with apertium 3.6 + 0.1.2 release [debs/contenttranslation/apertium-id-ms] - 10https://gerrit.wikimedia.org/r/582581 (https://phabricator.wikimedia.org/T248653) [06:24:49] (03PS1) 10Marostegui: db1111: Remove force to 10.4 package [puppet] - 10https://gerrit.wikimedia.org/r/583835 [06:26:56] (03CR) 10Marostegui: "Noop as expected: https://puppet-compiler.wmflabs.org/compiler1003/21592/" [puppet] - 10https://gerrit.wikimedia.org/r/583835 (owner: 10Marostegui) [06:26:58] (03CR) 10Marostegui: [C: 03+2] db1111: Remove force to 10.4 package [puppet] - 10https://gerrit.wikimedia.org/r/583835 (owner: 10Marostegui) [06:27:43] (03PS4) 10KartikMistry: apertium-cat-ita: Fix FTBFS with apertium 3.6 + 0.2.1 release [debs/contenttranslation/apertium-ca-it] - 10https://gerrit.wikimedia.org/r/579509 (https://phabricator.wikimedia.org/T248654) [06:30:42] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1082 for schema change', diff saved to https://phabricator.wikimedia.org/P10793 and previous config saved to /var/cache/conftool/dbconfig/20200327-063042-marostegui.json [06:30:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:31:43] !log Deploy schema change on db1082, this will generate lag on s5 labs [06:31:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:00:14] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1082 after schema change', diff saved to https://phabricator.wikimedia.org/P10794 and previous config saved to /var/cache/conftool/dbconfig/20200327-070014-marostegui.json [07:00:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:02:25] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1130 for schema change', diff saved to https://phabricator.wikimedia.org/P10795 and previous config saved to /var/cache/conftool/dbconfig/20200327-070224-marostegui.json [07:02:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:10:52] (03PS2) 10Muehlenhoff: Remove postgres spec tests [puppet] - 10https://gerrit.wikimedia.org/r/582805 [07:14:23] (03CR) 10Muehlenhoff: [C: 03+2] Remove postgres spec tests [puppet] - 10https://gerrit.wikimedia.org/r/582805 (owner: 10Muehlenhoff) [07:15:47] (03Abandoned) 10Muehlenhoff: Enable ferm for dbproxy1019 [puppet] - 10https://gerrit.wikimedia.org/r/531670 (owner: 10Muehlenhoff) [07:22:48] (03CR) 10Muehlenhoff: admin: update jpita account (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583720 (https://phabricator.wikimedia.org/T247722) (owner: 10Volans) [07:23:35] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1130 after schema change', diff saved to https://phabricator.wikimedia.org/P10796 and previous config saved to /var/cache/conftool/dbconfig/20200327-072334-marostegui.json [07:23:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:24:19] (03CR) 10Muehlenhoff: [C: 03+1] "Looks great" [puppet] - 10https://gerrit.wikimedia.org/r/583341 (https://phabricator.wikimedia.org/T233945) (owner: 10Jbond) [07:31:45] (03CR) 10Elukey: kibana: refactor kibana profile into two profiles (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583414 (https://phabricator.wikimedia.org/T246961) (owner: 10Mstyles) [07:32:09] !log installing grub2 updates from Stretch point release [07:32:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:36:40] !log execute 'rm /etc/logrotate.d/ceph-common' on cloudvirt[1,2]* and cloudcontrol* to stop daily cronspam (file not in the puppet catalog anymore) [07:36:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:36:44] Cc: arturo: --^ [07:58:28] (03PS1) 10Elukey: Exclude failed disk from analytics1057 [puppet] - 10https://gerrit.wikimedia.org/r/583889 [07:58:56] !log Deploy schema change on s2 codfw - this will generate lag on s2 codfw - T248333 [07:59:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:59:01] T248333: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 [07:59:45] (03CR) 10Elukey: [C: 03+2] Exclude failed disk from analytics1057 [puppet] - 10https://gerrit.wikimedia.org/r/583889 (owner: 10Elukey) [08:19:20] (03PS1) 10Muehlenhoff: Add Cumin alias for cloudstore [puppet] - 10https://gerrit.wikimedia.org/r/583892 [08:22:18] (03CR) 10Muehlenhoff: [C: 03+2] Add Cumin alias for cloudstore [puppet] - 10https://gerrit.wikimedia.org/r/583892 (owner: 10Muehlenhoff) [08:25:26] (03CR) 10Alexandros Kosiaris: [C: 03+1] "Awesome! Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/583619 (owner: 10Muehlenhoff) [08:38:52] 10Operations, 10SRE-Access-Requests: Request Netbox access for user "dubosv10" - https://phabricator.wikimedia.org/T248445 (10Aklapper) 05Open→03Declined Reflecting task status per last comment [08:44:58] PROBLEM - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3056 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [08:49:07] (03PS2) 10Dzahn: add webserver_misc_apps role to miscweb1002 [puppet] - 10https://gerrit.wikimedia.org/r/583667 (https://phabricator.wikimedia.org/T247887) [08:50:32] (03CR) 10Dzahn: [C: 03+2] add webserver_misc_apps role to miscweb1002 [puppet] - 10https://gerrit.wikimedia.org/r/583667 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [08:56:17] (03PS2) 10Dzahn: misc_apps/httpd: add support for PHP on buster [puppet] - 10https://gerrit.wikimedia.org/r/583675 (https://phabricator.wikimedia.org/T247887) [08:57:19] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Some of those relaxations I am not ok with, comments inline (aka I share chris' worry). The rest seem fine to me." (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/580985 (https://phabricator.wikimedia.org/T247538) (owner: 10Filippo Giunchedi) [08:57:30] RECOVERY - Ensure traffic_exporter binds on port 9322 and responds to HTTP requests on cp3056 is OK: HTTP OK: HTTP/1.0 200 OK - 22389 bytes in 0.257 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [08:58:20] (03CR) 10Muehlenhoff: misc_apps/httpd: add support for PHP on buster (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583675 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [08:59:59] 10Operations: Integrate Stretch 9.10/9.11 point updates - https://phabricator.wikimedia.org/T232308 (10MoritzMuehlenhoff) [09:00:13] 10Operations: Integrate Stretch 9.10/9.11 point updates - https://phabricator.wikimedia.org/T232308 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff This is done [09:00:16] (03CR) 10Ema: [C: 03+1] Release 8.0.6-1wm4 [debs/trafficserver] - 10https://gerrit.wikimedia.org/r/583715 (https://phabricator.wikimedia.org/T245616) (owner: 10Vgutierrez) [09:00:22] (03PS2) 10Muehlenhoff: Make the docker package name configurable and use docker.io on deneb [puppet] - 10https://gerrit.wikimedia.org/r/583619 [09:00:45] (03CR) 10Vgutierrez: [C: 03+2] Release 8.0.6-1wm4 [debs/trafficserver] - 10https://gerrit.wikimedia.org/r/583715 (https://phabricator.wikimedia.org/T245616) (owner: 10Vgutierrez) [09:01:20] (03PS3) 10Dzahn: misc_apps/httpd: add support for PHP on buster [puppet] - 10https://gerrit.wikimedia.org/r/583675 (https://phabricator.wikimedia.org/T247887) [09:01:22] (03CR) 10Dzahn: misc_apps/httpd: add support for PHP on buster (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583675 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [09:01:26] (03CR) 10Dzahn: "https://puppet-compiler.wmflabs.org/compiler1003/21593/miscweb1001.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/583675 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [09:02:27] (03CR) 10Muehlenhoff: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/583675 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [09:02:39] (03PS3) 10Muehlenhoff: postgres: Remove support for jessie [puppet] - 10https://gerrit.wikimedia.org/r/581985 [09:02:42] ACKNOWLEDGEMENT - Rate of JVM GC Old generation-s runs - cloudelastic1001-cloudelastic-chi-eqiad on cloudelastic1001 is CRITICAL: 489.4 gt 100 Gehel cluster overloaded during reindex - https://phabricator.wikimedia.org/T231517 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic [09:02:42] stance=cloudelastic1001&panelId=37 [09:02:42] ACKNOWLEDGEMENT - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is CRITICAL: 126.1 gt 100 Gehel cluster overloaded during reindex - https://phabricator.wikimedia.org/T231517 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic [09:02:42] stance=cloudelastic1002&panelId=37 [09:02:42] ACKNOWLEDGEMENT - Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad on cloudelastic1003 is CRITICAL: 458.3 gt 100 Gehel cluster overloaded during reindex - https://phabricator.wikimedia.org/T231517 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic [09:02:42] stance=cloudelastic1003&panelId=37 [09:02:42] ACKNOWLEDGEMENT - Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad on cloudelastic1004 is CRITICAL: 164.7 gt 100 Gehel cluster overloaded during reindex - https://phabricator.wikimedia.org/T231517 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic [09:02:43] stance=cloudelastic1004&panelId=37 [09:02:46] (03CR) 10Dzahn: [C: 03+2] misc_apps/httpd: add support for PHP on buster [puppet] - 10https://gerrit.wikimedia.org/r/583675 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [09:05:50] (03CR) 10Dzahn: "miscweb1001: Notice: /Stage[main]/Packages::Libapache2_mod_php/Package[libapache2-mod-php]/ensure: created" [puppet] - 10https://gerrit.wikimedia.org/r/583675 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [09:06:15] (03PS4) 10WMDE-leszek: Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569206 (https://phabricator.wikimedia.org/T242087) [09:10:21] (03PS1) 10Dzahn: wikimania_scholarships: fix php-mysql installation on buster [puppet] - 10https://gerrit.wikimedia.org/r/583895 (https://phabricator.wikimedia.org/T247887) [09:14:15] (03PS1) 10Alexandros Kosiaris: mobielapps: Package and publish chart 0.0.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/583896 [09:14:53] (03CR) 10Alexandros Kosiaris: [C: 03+2] mobielapps: Package and publish chart 0.0.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/583896 (owner: 10Alexandros Kosiaris) [09:15:30] (03PS1) 10Dzahn: iegreview: if on buster, use mariadb-client instead of mysql-client [puppet] - 10https://gerrit.wikimedia.org/r/583898 (https://phabricator.wikimedia.org/T247887) [09:15:36] (03Merged) 10jenkins-bot: mobielapps: Package and publish chart 0.0.3 [deployment-charts] - 10https://gerrit.wikimedia.org/r/583896 (owner: 10Alexandros Kosiaris) [09:15:43] (03CR) 10Muehlenhoff: [C: 03+2] Make the docker package name configurable and use docker.io on deneb [puppet] - 10https://gerrit.wikimedia.org/r/583619 (owner: 10Muehlenhoff) [09:19:29] (03PS1) 10Dzahn: racktables: fix php.ini path for buster support [puppet] - 10https://gerrit.wikimedia.org/r/583899 (https://phabricator.wikimedia.org/T247887) [09:21:38] (03CR) 10Muehlenhoff: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/583899 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [09:23:48] (03PS1) 10Dzahn: add IPv6 for miscweb1002.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/583900 (https://phabricator.wikimedia.org/T247887) [09:24:12] (03CR) 10jerkins-bot: [V: 04-1] add IPv6 for miscweb1002.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/583900 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [09:24:38] (03PS2) 10Dzahn: racktables: fix php.ini path for buster support [puppet] - 10https://gerrit.wikimedia.org/r/583899 (https://phabricator.wikimedia.org/T247646) [09:25:04] (03CR) 10Dzahn: [C: 03+2] racktables: fix php.ini path for buster support [puppet] - 10https://gerrit.wikimedia.org/r/583899 (https://phabricator.wikimedia.org/T247646) (owner: 10Dzahn) [09:26:04] (03PS2) 10Dzahn: add IPv6 for miscweb1002.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/583900 (https://phabricator.wikimedia.org/T247887) [09:26:10] (03PS3) 10Dzahn: racktables: fix php.ini path for buster support [puppet] - 10https://gerrit.wikimedia.org/r/583899 (https://phabricator.wikimedia.org/T247646) [09:26:57] (03PS2) 10Dzahn: iegreview: if on buster, use mariadb-client instead of mysql-client [puppet] - 10https://gerrit.wikimedia.org/r/583898 (https://phabricator.wikimedia.org/T247648) [09:27:16] (03PS2) 10Dzahn: wikimania_scholarships: fix php-mysql installation on buster [puppet] - 10https://gerrit.wikimedia.org/r/583895 (https://phabricator.wikimedia.org/T247648) [09:27:21] (03CR) 10Muehlenhoff: "There's no need for a conditional; mysql-client in Stretch is already a transitional package to mariadb, simply include mariadb-client in " [puppet] - 10https://gerrit.wikimedia.org/r/583898 (https://phabricator.wikimedia.org/T247648) (owner: 10Dzahn) [09:28:07] (03CR) 10Muehlenhoff: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/583895 (https://phabricator.wikimedia.org/T247648) (owner: 10Dzahn) [09:29:01] 10Operations: [ftpsync@sodium] ERROR: rsync errors - https://phabricator.wikimedia.org/T248660 (10ayounsi) p:05Triage→03Medium [09:29:48] (03PS3) 10Dzahn: iegreview: use mariadb-client instead of mysql-client [puppet] - 10https://gerrit.wikimedia.org/r/583898 (https://phabricator.wikimedia.org/T247648) [09:29:55] XioNoX: I was chatting about this on the side :) [09:30:07] 10Operations: [ftpsync@sodium] ERROR: rsync errors - https://phabricator.wikimedia.org/T248660 (10ayounsi) [09:30:30] volans: which this? ftpsync? [09:30:34] yes [09:31:07] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1098:3316 for schema change', diff saved to https://phabricator.wikimedia.org/P10798 and previous config saved to /var/cache/conftool/dbconfig/20200327-093106-marostegui.json [09:31:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:31:24] volans: any idea who manages it? [09:31:27] 10Operations: [ftpsync@sodium] ERROR: rsync errors - https://phabricator.wikimedia.org/T248660 (10Dzahn) a:03Dzahn [09:32:15] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1098:3316', diff saved to https://phabricator.wikimedia.org/P10799 and previous config saved to /var/cache/conftool/dbconfig/20200327-093214-marostegui.json [09:32:16] (03PS5) 10WMDE-leszek: Beta commons: Remove custom wmgWikibaseRepoForeignRepositories setting [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569207 (https://phabricator.wikimedia.org/T242087) [09:32:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:32:19] 10Operations: [ftpsync@sodium] ERROR: rsync errors - https://phabricator.wikimedia.org/T248660 (10Volans) For context: - exit code 23 on rsync is `Partial transfer due to error` - there are root owned files and symlinks in `/srv/mirrors/debian/dists/sid/main` - the crontab that runs the rsync (AFACIT) is run as... [09:32:30] 10Operations: [ftpsync@sodium] ERROR: rsync errors - https://phabricator.wikimedia.org/T248660 (10Dzahn) Yesterday Icinga was alerting about failed ftpsync on sodium and to debug i ran it manually which then made the original alert recover. But doing that it looks like i messed up permissions causing the seconda... [09:34:05] (03PS5) 10WMDE-leszek: Beta cluster: remove custom wmgWikibaseClientRepositories settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569208 (https://phabricator.wikimedia.org/T242087) [09:35:55] (03CR) 10Jcrespo: [C: 03+1] CuminExecution: Capture Exception cumin.transports.WorkerError [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/578623 (https://phabricator.wikimedia.org/T218189) (owner: 10Guozr.im) [09:36:15] !log sodium fixing root owned files in /srv/mirrors/debian to be owned by mirror:mirror (T248660) [09:36:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:36:20] T248660: [ftpsync@sodium] ERROR: rsync errors - https://phabricator.wikimedia.org/T248660 [09:36:22] (03PS5) 10WMDE-leszek: Test wikidata: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569209 (https://phabricator.wikimedia.org/T242087) [09:36:56] (03CR) 10WMDE-leszek: [C: 04-1] "config is incorrect, to be fixed in the next patchset" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569209 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [09:37:36] (03CR) 10Volans: [C: 03+1] "LGTM, thanks for your contribution!" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/578623 (https://phabricator.wikimedia.org/T218189) (owner: 10Guozr.im) [09:37:56] !log sodium - running ftpsync as user mirror (T248660) [09:38:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:39:01] (03CR) 10Dzahn: [C: 03+2] wikimania_scholarships: fix php-mysql installation on buster [puppet] - 10https://gerrit.wikimedia.org/r/583895 (https://phabricator.wikimedia.org/T247648) (owner: 10Dzahn) [09:39:55] (03CR) 10Muehlenhoff: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/583898 (https://phabricator.wikimedia.org/T247648) (owner: 10Dzahn) [09:41:00] (03PS4) 10Dzahn: iegreview: use mariadb-client instead of mysql-client [puppet] - 10https://gerrit.wikimedia.org/r/583898 (https://phabricator.wikimedia.org/T247648) [09:41:17] tin will never die [09:41:20] fatal: unable to access 'http://tin.eqiad.wmnet/iegreview/iegreview/.git/': Could not resolve host: tin.eqiad.wmnet [09:41:24] still :) [09:41:34] (03PS1) 10Vgutierrez: nagios_common: Reduce the OCSP warning threshold to 46 hours [puppet] - 10https://gerrit.wikimedia.org/r/583903 [09:41:40] every time you have a host with scap-deployed stuff that is new [09:43:36] (03PS1) 10Jcrespo: transfer.py: Upgrade codebase to latest version on HEAD [puppet] - 10https://gerrit.wikimedia.org/r/583904 [09:44:16] (03CR) 10Jcrespo: [C: 03+2] "Thanks a lot, this was not an easy contribution to do, but you went though! Congrats." [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/578623 (https://phabricator.wikimedia.org/T218189) (owner: 10Guozr.im) [09:44:42] 10Operations: [ftpsync@sodium] ERROR: rsync errors - https://phabricator.wikimedia.org/T248660 (10Dzahn) I ran ` find /srv/mirrors/debian/ -user root -exec chown mirror:mirror {} \; ` and afterwards ` sudo -u mirror ftpsync ` and saw no errors. [09:45:02] (03PS2) 10Jcrespo: transfer.py: Upgrade codebase to latest version on HEAD [puppet] - 10https://gerrit.wikimedia.org/r/583904 [09:47:43] (03CR) 10Ayounsi: [C: 03+1] nagios_common: Reduce the OCSP warning threshold to 46 hours [puppet] - 10https://gerrit.wikimedia.org/r/583903 (owner: 10Vgutierrez) [09:49:55] (03PS6) 10WMDE-leszek: Test wikidata: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569209 (https://phabricator.wikimedia.org/T242087) [09:50:18] (03CR) 10Vgutierrez: [C: 03+2] nagios_common: Reduce the OCSP warning threshold to 46 hours [puppet] - 10https://gerrit.wikimedia.org/r/583903 (owner: 10Vgutierrez) [09:50:21] PROBLEM - Check systemd state on deneb is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [09:51:19] (03CR) 10Dzahn: [C: 03+2] iegreview: use mariadb-client instead of mysql-client [puppet] - 10https://gerrit.wikimedia.org/r/583898 (https://phabricator.wikimedia.org/T247648) (owner: 10Dzahn) [09:54:04] 10Operations, 10Analytics, 10Product-Analytics, 10SRE-Access-Requests: Hive access for Sam Patton - https://phabricator.wikimedia.org/T248097 (10Dzahn) [09:57:29] (03CR) 10Jcrespo: [C: 03+2] transfer.py: Upgrade codebase to latest version on HEAD [puppet] - 10https://gerrit.wikimedia.org/r/583904 (owner: 10Jcrespo) [09:58:43] RECOVERY - Check systemd state on deneb is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [10:00:34] (03CR) 10Elukey: [C: 03+1] "As far as I know all good from the Analytics side, but I have also added members of my team to triple check" [puppet] - 10https://gerrit.wikimedia.org/r/583570 (https://phabricator.wikimedia.org/T210484) (owner: 10Ema) [10:01:20] !log Alter db1096:3315 enwikivoyage.page to set page_restrictions to default NULL - T248333 [10:01:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:01:26] T248333: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 [10:01:49] (03PS1) 10Jbond: people - sso: add protected_uri for people site [puppet] - 10https://gerrit.wikimedia.org/r/583905 [10:02:01] !log Alter db2084:3315 enwikivoyage.page to set page_restrictions to default NULL - T248333 [10:02:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:02:15] 10Operations, 10LDAP-Access-Requests: Add Huei Tan to `wmf` LDAF group - https://phabricator.wikimedia.org/T248605 (10Volans) Verified via hangout that it's indeed Huei's account. @hueitan could you add here also what's your Wikitech account please? That's the one to be added to the `wmf` group. Thanks [10:03:19] (03CR) 10Muehlenhoff: "Looks good, one comment inline" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583905 (owner: 10Jbond) [10:03:21] (03PS1) 10Giuseppe Lavagetto: check_opcache: Use the number of scripts to determine threshold [puppet] - 10https://gerrit.wikimedia.org/r/583906 [10:03:39] !log sodium - find /srv/mirrors/debian/ -user root -exec chown -h mirror:mirror {} \; (-h to also fix symbolic links); sudo -u mirror ftpsync (T248660) [10:03:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:03:44] T248660: [ftpsync@sodium] ERROR: rsync errors - https://phabricator.wikimedia.org/T248660 [10:04:23] (03PS2) 10Jbond: people - sso: add protected_uri for people site [puppet] - 10https://gerrit.wikimedia.org/r/583905 [10:04:26] (03CR) 10Jbond: "thanks updated" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583905 (owner: 10Jbond) [10:04:31] !log upload trafficserver 8.0.6-1wm4 to apt.wm.o (buster) - T245616 T170567 [10:04:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:04:36] T170567: Support TLSv1.3 - https://phabricator.wikimedia.org/T170567 [10:04:37] T245616: Provide a simple and automated SSL Ticket key generation system for ATS - https://phabricator.wikimedia.org/T245616 [10:05:05] (03PS1) 10Muehlenhoff: Adapt deb src to Buster being the primary build host [puppet] - 10https://gerrit.wikimedia.org/r/583907 [10:06:23] 10Operations, 10LDAP-Access-Requests: Add Huei Tan to `wmf` LDAF group - https://phabricator.wikimedia.org/T248605 (10hueitan) Thanks @Volans `Huei Tan` Here's the user page of my account on Wikitech https://wikitech.wikimedia.org/wiki/User:Huei_Tan [10:06:37] (03CR) 10Filippo Giunchedi: "Thanks for review! See next PS" (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/580985 (https://phabricator.wikimedia.org/T247538) (owner: 10Filippo Giunchedi) [10:06:44] (03PS5) 10Filippo Giunchedi: icinga: relax check interval for selected checks [puppet] - 10https://gerrit.wikimedia.org/r/580985 (https://phabricator.wikimedia.org/T247538) [10:10:49] 10Operations: [ftpsync@sodium] ERROR: rsync errors - https://phabricator.wikimedia.org/T248660 (10Dzahn) 05Open→03Resolved After the last run of ftpsync there was no more error. [10:11:00] (03PS4) 10Ema: ATS: remove debug HTTP headers if X-Wikimedia-Debug is absent [puppet] - 10https://gerrit.wikimedia.org/r/583570 (https://phabricator.wikimedia.org/T210484) [10:11:32] 10Operations, 10LDAP-Access-Requests: Add Scardenasmolinar to WMF LDAP group - https://phabricator.wikimedia.org/T248521 (10Aklapper) >>! In T248521#6001420, @Volans wrote: > @Scardenasmolinar: could you also please link your Phabricator account to your official WMF meta account on wiki? @Scardenasmolinar: Se... [10:12:48] !log miscweb1002 - sed -i 's/tin.eqiad/deployment.eqiad/g' /srv/deployment/iegreview/iegreview-cache/.config T247648 [10:13:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:13:01] T247648: miscweb1001/2001 - upgrade to buster or decom - https://phabricator.wikimedia.org/T247648 [10:17:31] mutante: ftpsync errors keep coming [10:17:33] FYI [10:18:09] volans: the last one was 23 minutes ago. that was before i closed it as resolved. [10:18:13] right [10:18:36] (03CR) 10Jbond: [C: 03+2] people - sso: add protected_uri for people site [puppet] - 10https://gerrit.wikimedia.org/r/583905 (owner: 10Jbond) [10:18:47] there was no more error on my last run (now) [10:19:11] RECOVERY - Ensure hosts are not performing a change on every puppet run on puppetdb1002 is OK: OK: all nodes running as expected https://wikitech.wikimedia.org/wiki/Puppet%23check_puppet_run_changes [10:22:55] 10Operations, 10serviceops: miscweb1001/2001 - upgrade to buster or decom - https://phabricator.wikimedia.org/T247648 (10Dzahn) [10:23:15] mutante: ack, thx [10:24:08] 10Operations, 10serviceops: miscweb1001/2001 - upgrade to buster or decom - https://phabricator.wikimedia.org/T247648 (10Dzahn) [10:28:46] !log Alter db2125 s2 to set page_restrictions to default NULL - T248333 [10:28:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:28:52] T248333: Schema change: Make page.page_restrictions column NULL - https://phabricator.wikimedia.org/T248333 [10:32:19] (03PS1) 10Volans: admin: add Huei Tan to wmf ldap only users [puppet] - 10https://gerrit.wikimedia.org/r/583917 (https://phabricator.wikimedia.org/T248605) [10:33:23] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: Add Huei Tan to `wmf` LDAF group - https://phabricator.wikimedia.org/T248605 (10Volans) @hueitan in the meanwhile that the patch gets reviewed could you also link your LDAP account (the wikitech one) with this Phabricator account? See https://phabrica... [10:35:44] (03PS7) 10WMDE-leszek: Test wikidata: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569209 (https://phabricator.wikimedia.org/T248664) [10:35:46] (03PS4) 10WMDE-leszek: Test wikibase clients: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569256 (https://phabricator.wikimedia.org/T248664) [10:40:08] (03CR) 10Dzahn: [C: 03+1] "looks consistent on corp-LDAP and wikitech-LDAP" [puppet] - 10https://gerrit.wikimedia.org/r/583917 (https://phabricator.wikimedia.org/T248605) (owner: 10Volans) [10:42:44] (03CR) 10Dzahn: [C: 03+2] add IPv6 for miscweb1002.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/583900 (https://phabricator.wikimedia.org/T247887) (owner: 10Dzahn) [10:42:49] (03PS3) 10Dzahn: add IPv6 for miscweb1002.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/583900 (https://phabricator.wikimedia.org/T247887) [10:44:47] 10Operations, 10Analytics, 10Performance-Team, 10Traffic, 10Patch-For-Review: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10ema) @Gilles: I see that `X-Varnish` is used by [[https://gerrit.wikimedia.org/g/mediawiki/extensions/MultimediaView... [10:45:47] !log miscweb1002 - upload and unpack RackTables-0.21.4 (T247646 T247648) [10:45:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:45:54] T247648: miscweb1001/2001 - upgrade to buster or decom - https://phabricator.wikimedia.org/T247648 [10:45:54] T247646: migrate racktables to a buster VM (was: decom racktables?) - https://phabricator.wikimedia.org/T247646 [10:50:09] (03PS1) 10Dzahn: ATS: switch racktables to backend miscweb1002 [puppet] - 10https://gerrit.wikimedia.org/r/583920 [10:50:27] (03CR) 10Volans: [C: 03+2] admin: add Huei Tan to wmf ldap only users [puppet] - 10https://gerrit.wikimedia.org/r/583917 (https://phabricator.wikimedia.org/T248605) (owner: 10Volans) [10:51:59] (03CR) 10Elukey: WIP - Introduce profile::mariadb::misc::analytics (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/553742 (https://phabricator.wikimedia.org/T234826) (owner: 10Elukey) [10:52:26] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: Add Huei Tan to `wmf` LDAF group - https://phabricator.wikimedia.org/T248605 (10Volans) 05Open→03Resolved a:03Volans Change merged, LDAP updated, resolving. Feel free to re-open if you have any issue with accessing resources that requires the `w... [10:54:55] (03CR) 10Muehlenhoff: [C: 03+2] Adapt deb src to Buster being the primary build host [puppet] - 10https://gerrit.wikimedia.org/r/583907 (owner: 10Muehlenhoff) [10:55:55] !log revoke puppet cert webserver-misc-apps.discovery.wmnet and recreate with additional SANs for new VMs [10:55:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:56:49] (03CR) 10Ayounsi: "Thanks, a mix of nit, thoughts, suggestions, etc..." (035 comments) [homer/public] - 10https://gerrit.wikimedia.org/r/583740 (https://phabricator.wikimedia.org/T248394) (owner: 10CDanis) [10:58:09] (03PS4) 10WMDE-leszek: Test commons: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569257 (https://phabricator.wikimedia.org/T248664) [11:02:28] (03CR) 10Ema: "o/" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/583366 (owner: 10Jbond) [11:04:56] RECOVERY - people.wikimedia.org requires authentication on people1001 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 586 bytes in 1.009 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration [11:09:18] (03PS1) 10Dzahn: update certificate for webserver-misc-apps.discovery.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/583921 (https://phabricator.wikimedia.org/T247648) [11:13:44] 10Operations, 10cloud-services-team (Kanban): ceph cron-spam emails - https://phabricator.wikimedia.org/T248671 (10Volans) [11:14:20] (03CR) 10Dzahn: [C: 03+2] update certificate for webserver-misc-apps.discovery.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/583921 (https://phabricator.wikimedia.org/T247648) (owner: 10Dzahn) [11:14:31] (03PS2) 10Dzahn: update certificate for webserver-misc-apps.discovery.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/583921 (https://phabricator.wikimedia.org/T247648) [11:18:34] (03PS1) 10Muehlenhoff: Don't include component/thirdparty-k8s on Buster [puppet] - 10https://gerrit.wikimedia.org/r/583923 [11:19:16] PROBLEM - Check systemd state on ores1003 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:23:24] RECOVERY - Check systemd state on ores1003 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:23:49] (03CR) 10Jbond: phased rollout of sensible flow-table-sizes (031 comment) [homer/public] - 10https://gerrit.wikimedia.org/r/583740 (https://phabricator.wikimedia.org/T248394) (owner: 10CDanis) [11:31:04] (03CR) 10Jbond: "thanks see inline also not the PCC this should hopefully indicate that the retry and timeout values haven't changed" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/583366 (owner: 10Jbond) [11:32:30] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: LDAP access to the wmf group for Pita - https://phabricator.wikimedia.org/T247722 (10Volans) I've quickly chatted with @Jpita, here's the summary of my understanding of the current situation. It seems that you have two different accounts on [[ https:... [11:44:19] !log hnowlan@puppetmaster1001 conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2021.codfw.wmnet [11:44:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:39] (03CR) 10Jbond: "LGTM one comment" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/581985 (owner: 10Muehlenhoff) [11:44:49] !log oblivian@puppetmaster1001 conftool action : edit; selector: dc=codfw,cluster=restbase,service=restbase-ssl,name=restbase202[1].codfw.wmnet [11:44:51] 10Operations, 10Analytics, 10Performance-Team, 10Traffic, 10Patch-For-Review: Only serve debug HTTP headers when x-wikimedia-debug is present - https://phabricator.wikimedia.org/T210484 (10Gilles) No, that's some ancient performance logging that dates back to when Media Viewer was launched and we needed... [11:44:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:56] !log hnowlan@puppetmaster1001 conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2022.codfw.wmnet [11:45:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:45:07] !log hnowlan@puppetmaster1001 conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase,name=restbase2023.codfw.wmnet [11:45:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:51:45] !log hnowlan@puppetmaster1001 conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase-ssl,name=restbase202[123].codfw.wmnet [11:51:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:52:38] (03PS2) 10Ladsgroup: Beta wikidata: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572928 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [11:52:49] (03CR) 10Ladsgroup: [C: 03+2] Beta wikidata: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572928 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [11:53:44] (03CR) 10Muehlenhoff: postgres: Remove support for jessie (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/581985 (owner: 10Muehlenhoff) [11:53:49] (03Merged) 10jenkins-bot: Beta wikidata: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/572928 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [11:54:04] !log hnowlan@puppetmaster1001 conftool action : set/pooled=yes:weight=10; selector: dc=codfw,cluster=restbase,service=restbase-backend,name=restbase202[123].codfw.wmnet [11:54:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:01:29] (03PS4) 10Muehlenhoff: postgres: Remove support for jessie [puppet] - 10https://gerrit.wikimedia.org/r/581985 [12:02:35] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1103:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P10800 and previous config saved to /var/cache/conftool/dbconfig/20200327-120234-marostegui.json [12:02:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:02:39] (03CR) 10Jbond: [C: 03+1] postgres: Remove support for jessie [puppet] - 10https://gerrit.wikimedia.org/r/581985 (owner: 10Muehlenhoff) [12:05:16] (03PS1) 10Muehlenhoff: nagios: Remove support for jessie [puppet] - 10https://gerrit.wikimedia.org/r/583931 [12:12:55] (03PS6) 10Ladsgroup: Beta commons: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569205 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:13:44] (03CR) 10Ladsgroup: [C: 03+2] Beta commons: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569205 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:14:36] (03Merged) 10jenkins-bot: Beta commons: Define entity sources configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569205 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:20:59] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1103:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P10801 and previous config saved to /var/cache/conftool/dbconfig/20200327-122058-marostegui.json [12:21:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:21:44] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1105:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P10802 and previous config saved to /var/cache/conftool/dbconfig/20200327-122144-marostegui.json [12:21:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:46:54] (03PS5) 10Ladsgroup: Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569206 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [12:49:15] !log ladsgroup@mwmaint1002:~$ mwscript createAndPromote.php --wiki=labswiki --force "Ladsgroup" --interface-admin [12:49:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:59:56] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1004-cloudelastic-chi-eqiad on cloudelastic1004 is OK: (C)100 gt (W)80 gt 72.2 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1004&panelId=37 [13:04:46] (03PS1) 10Andrew Bogott: Keystone: remove member_role_name from keystone.conf [puppet] - 10https://gerrit.wikimedia.org/r/583940 (https://phabricator.wikimedia.org/T248635) [13:05:43] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1105:3312 after schema change', diff saved to https://phabricator.wikimedia.org/P10803 and previous config saved to /var/cache/conftool/dbconfig/20200327-130542-marostegui.json [13:05:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:07:06] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1076 for schema change', diff saved to https://phabricator.wikimedia.org/P10804 and previous config saved to /var/cache/conftool/dbconfig/20200327-130706-marostegui.json [13:07:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:07:53] (03CR) 10Ladsgroup: [C: 03+2] Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569206 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [13:09:10] (03Merged) 10jenkins-bot: Beta cluster: use entity source Wikibase setting for all wikibase-enabled wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/569206 (https://phabricator.wikimedia.org/T242087) (owner: 10WMDE-leszek) [13:10:31] (03PS1) 10Ema: cache: stop sending X-Varnish [puppet] - 10https://gerrit.wikimedia.org/r/583942 (https://phabricator.wikimedia.org/T210484) [13:11:28] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1002-cloudelastic-chi-eqiad on cloudelastic1002 is OK: (C)100 gt (W)80 gt 79.32 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1002&panelId=37 [13:23:13] (03CR) 10Ema: [C: 03+1] "LGTM except one nit on wording." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583366 (owner: 10Jbond) [13:23:33] (03PS1) 10Subramanya Sastry: update_parsoid.sh: Add phpfm restart after new code is pulled [puppet] - 10https://gerrit.wikimedia.org/r/583945 [13:24:54] 10Operations, 10cloud-services-team (Kanban): ceph cron-spam emails - https://phabricator.wikimedia.org/T248671 (10JHedden) [13:26:31] (03PS8) 10Jbond: envoy: introduce use_remote_address parameter [puppet] - 10https://gerrit.wikimedia.org/r/583366 [13:26:43] 10Operations, 10netops: IRR updates needed - https://phabricator.wikimedia.org/T235886 (10ayounsi) v4 and v6 objects created on the ARIN IRR. Will delete them from RIPE next week. From https://github.com/job/irrexplorer/issues/52#issuecomment-371497341 and to make everything clean, I'm considering replacing t... [13:27:06] 10Operations, 10cloud-services-team (Kanban): ceph cron-spam emails - https://phabricator.wikimedia.org/T248671 (10JHedden) [13:27:11] (03CR) 10Jbond: "thanks updated ill hold off until monday to deploy this now" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/583366 (owner: 10Jbond) [13:28:15] godog: XioNoX: [13:28:22] ignore that miss type [13:30:22] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1076 after schema change', diff saved to https://phabricator.wikimedia.org/P10805 and previous config saved to /var/cache/conftool/dbconfig/20200327-133022-marostegui.json [13:30:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:31:52] (03PS8) 10Hashar: zuul: provision the scap repository [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) [13:35:57] (03CR) 10Ottomata: "If we feel good about TLS only eventgate, we can merge this too ya?" [puppet] - 10https://gerrit.wikimedia.org/r/562810 (https://phabricator.wikimedia.org/T241073) (owner: 10Alexandros Kosiaris) [13:35:58] (03PS1) 10Vgutierrez: ATS: Re-enable TLS tickets in ulsfo [puppet] - 10https://gerrit.wikimedia.org/r/583948 (https://phabricator.wikimedia.org/T245616) [13:37:48] (03CR) 10Andrew Bogott: [C: 03+2] Keystone: remove member_role_name from keystone.conf [puppet] - 10https://gerrit.wikimedia.org/r/583940 (https://phabricator.wikimedia.org/T248635) (owner: 10Andrew Bogott) [13:39:22] (03CR) 10Vgutierrez: "pcc is happy: https://puppet-compiler.wmflabs.org/compiler1002/21595/" [puppet] - 10https://gerrit.wikimedia.org/r/583948 (https://phabricator.wikimedia.org/T245616) (owner: 10Vgutierrez) [13:41:15] (03CR) 10Hashar: "I guess we then need the ssh key pairs to be generated and added to the secret puppet. I did that for labs/private.git with https://gerri" [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [13:51:20] (03CR) 10Arlolra: [C: 03+1] update_parsoid.sh: Add phpfm restart after new code is pulled [puppet] - 10https://gerrit.wikimedia.org/r/583945 (owner: 10Subramanya Sastry) [13:57:53] (03CR) 10Ema: [C: 03+1] "We agreed to go ahead with this on Monday." [puppet] - 10https://gerrit.wikimedia.org/r/583948 (https://phabricator.wikimedia.org/T245616) (owner: 10Vgutierrez) [13:58:08] (03CR) 10Dzahn: "I added a new keypair in the private repo but there is no "deploy_" prefix. It's just called zuul and zuul.pub." [labs/private] - 10https://gerrit.wikimedia.org/r/582895 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [14:01:10] (03CR) 10Hashar: [V: 03+2 C: 03+2] "We will see, but previously the puppet compiler failed with:" [labs/private] - 10https://gerrit.wikimedia.org/r/582895 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [14:01:47] mutante: hi :-) I think the prefix is needed since the group is "deploy-zuul", but we will see when running the catalog I guess [14:01:51] thx for the key pair :] [14:02:12] (03CR) 10Dzahn: "> Patch Set 1:" [labs/private] - 10https://gerrit.wikimedia.org/r/582895 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [14:02:45] (03CR) 10Vgutierrez: [C: 04-2] "to be merged on Monday" [puppet] - 10https://gerrit.wikimedia.org/r/583948 (https://phabricator.wikimedia.org/T245616) (owner: 10Vgutierrez) [14:02:52] there is no consistency there at all https://wikitech.wikimedia.org/wiki/Keyholder [14:03:04] yeah I could not find a reliable doc :-\ [14:03:21] iircI based my change on the homer one [14:07:02] (03PS1) 10Muehlenhoff: Remove jessie support for puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/583953 [14:13:18] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [14:13:43] (03PS2) 10Dzahn: parsoid/testing: update_parsoid.sh: Add phpfm restart after new code is pulled [puppet] - 10https://gerrit.wikimedia.org/r/583945 (owner: 10Subramanya Sastry) [14:14:54] (03CR) 10Dzahn: [C: 03+2] "scandium-only" [puppet] - 10https://gerrit.wikimedia.org/r/583945 (owner: 10Subramanya Sastry) [14:15:22] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [14:17:29] (03PS1) 10Muehlenhoff: Remove jessie support from osm/maps [puppet] - 10https://gerrit.wikimedia.org/r/583954 [14:19:37] !log updating linux-image-4.9.0-11-amd64 where applicable [14:19:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:20:21] (03PS1) 10Dzahn: match names of zuul fake keys to real private keys [labs/private] - 10https://gerrit.wikimedia.org/r/583956 [14:22:41] !log marostegui@cumin1001 dbctl commit (dc=all): 'Depool db1129 for schema change', diff saved to https://phabricator.wikimedia.org/P10806 and previous config saved to /var/cache/conftool/dbconfig/20200327-142240-marostegui.json [14:22:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:24:32] RECOVERY - Rate of JVM GC Old generation-s runs - cloudelastic1003-cloudelastic-chi-eqiad on cloudelastic1003 is OK: (C)100 gt (W)80 gt 69.56 https://wikitech.wikimedia.org/wiki/Search%23Using_jstack_or_jmap_or_other_similar_tools_to_view_logs https://grafana.wikimedia.org/d/000000462/elasticsearch-memory?orgId=1&var-exported_cluster=cloudelastic-chi-eqiad&var-instance=cloudelastic1003&panelId=37 [14:26:07] (03CR) 10Dzahn: [V: 03+2 C: 03+2] "Just making sure private and fake/private are matching right now." [labs/private] - 10https://gerrit.wikimedia.org/r/583956 (owner: 10Dzahn) [14:26:16] (03PS4) 10Arturo Borrero Gonzalez: toolforge: introduce role/profile for legacy URL redirector [puppet] - 10https://gerrit.wikimedia.org/r/583593 (https://phabricator.wikimedia.org/T247236) [14:26:29] (03PS1) 10L0st3xpl0r3r: Made the run() function always return a list. [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 [14:26:31] (03CR) 10Welcome, new contributor!: "Thank you for making your first contribution to Wikimedia! :) To learn how to get your code changes reviewed faster and more likely to get" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (owner: 10L0st3xpl0r3r) [14:27:19] (03CR) 10Dzahn: "a keypair has been added in the private repo, called "zuul" and "zuul.pub". The "deployment-key-passphrase" has been used to match https:/" [puppet] - 10https://gerrit.wikimedia.org/r/579587 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [14:27:26] mutante: eventually I found out how the key file name is figured out. That is in scap::target [14:27:39] it is based on the deploy user [14:27:55] so if we have: scap::target { 'zuul/deploy': deploy_user => 'deploy-user' } [14:28:20] scap::target forge the filename based on the user, replacing non word characters with underscores [14:28:53] hence deploy-zuul leads to a secret file keyholder/deploy_zuul.pub [14:28:57] though somehow it works fine for many services without that prefix [14:29:41] maybe that is also defined somewhere else [14:31:28] has a user been created yet? [14:32:03] or maybe some of those keys are specified to scap::target . It has a key_name parameter [14:32:05] this stuff (and lack of docs) is why i think scap makes things more complicated [14:33:30] hashar: yea, it is. for example gerrit users key_name => $scap_key_name [14:33:34] uses [14:34:01] so they keys should be named deploy_zuul and deploy_zuul.pub [14:34:03] scap::target { 'gervert/deploy': [14:34:04] deploy_user => $scap_user, [14:34:04] manage_user => false, [14:34:04] key_name => $scap_key_name, [14:34:04] } [14:34:06] after the deployment username [14:34:26] or we instruct puppet to use the key "zuul" [14:35:39] scap::target { 'zuul/deploy': [14:35:51] deploy_user => 'deploy-zuul', [14:35:54] + key_name => 'zuul', [14:35:55] } [14:38:36] 10Operations, 10netops: IRR updates needed - https://phabricator.wikimedia.org/T235886 (10ayounsi) >>! In T235886#6005076, @ayounsi wrote: > From https://github.com/job/irrexplorer/issues/52#issuecomment-371497341 and to make everything clean, I'm considering replacing the current ROAs (some using max-length)... [14:38:38] yea, rename key, rename user or use the parameter [14:39:48] 10Operations, 10MediaWiki-Authentication-and-authorization, 10Security-Team, 10Traffic, 10Security: Investigate usefulness of SameSite cookies for logged-in accounts - https://phabricator.wikimedia.org/T158604 (10Astinson) I keep getting hit by this on multiple tools that I use in my volunteer capacity:... [14:41:26] !log marostegui@cumin1001 dbctl commit (dc=all): 'Repool db1129 after schema change', diff saved to https://phabricator.wikimedia.org/P10807 and previous config saved to /var/cache/conftool/dbconfig/20200327-144125-marostegui.json [14:41:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:43:00] (03PS1) 10Jhedden: ceph: refactor ceph-common for clients with no configuration [puppet] - 10https://gerrit.wikimedia.org/r/583964 (https://phabricator.wikimedia.org/T248610) [14:45:41] mutante: lets just use deploy_zuul and deploy_zuul.pub , that passed the puppet compiler just fine ;) [14:46:12] (03PS1) 10Dzahn: Revert "match names of zuul fake keys to real private keys" [labs/private] - 10https://gerrit.wikimedia.org/r/583967 [14:46:33] (03CR) 10Dzahn: [V: 03+2 C: 03+2] Revert "match names of zuul fake keys to real private keys" [labs/private] - 10https://gerrit.wikimedia.org/r/583967 (owner: 10Dzahn) [14:48:12] hashar: ok. renamed [14:51:02] (03CR) 10Dzahn: [C: 04-1] "before doing this and upgrading the racktables version.. create a database backup!" [puppet] - 10https://gerrit.wikimedia.org/r/583920 (owner: 10Dzahn) [14:51:14] and probably that will work on contint1001 and contint2001 [14:51:24] I think the ssh keys were the last thing needed [14:52:27] (03CR) 10Jcrespo: "Thanks for your contribution!" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (owner: 10L0st3xpl0r3r) [14:54:47] (03PS1) 10Dzahn: add miscweb2002.codfw.wmnet [dns] - 10https://gerrit.wikimedia.org/r/583970 (https://phabricator.wikimedia.org/T247887) [14:59:23] 10Operations, 10Traffic, 10netops: BGP: Investigate isolating codfw and eqiad - https://phabricator.wikimedia.org/T246721 (10ayounsi) 05Open→03Resolved a:03ayounsi Done. [15:00:57] (03CR) 10Vgutierrez: [C: 03+1] acme_chief: expand cp nodes regex [puppet] - 10https://gerrit.wikimedia.org/r/583635 (https://phabricator.wikimedia.org/T247340) (owner: 10BBlack) [15:06:28] (03PS1) 10Vgutierrez: site: Reimage cp2027 as cache::text [puppet] - 10https://gerrit.wikimedia.org/r/583978 (https://phabricator.wikimedia.org/T247340) [15:09:46] PROBLEM - DPKG on snapshot1008 is CRITICAL: DPKG CRITICAL dpkg reports broken packages https://wikitech.wikimedia.org/wiki/Monitoring/dpkg [15:12:16] !log andrew@deploy1001 Started deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens [15:12:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:14:44] (03PS1) 10Dzahn: add IPv6 for miscweb2002 [dns] - 10https://gerrit.wikimedia.org/r/583980 [15:15:51] !log andrew@deploy1001 Finished deploy [horizon/deploy@33e67f9]: fix Identity->Projects with keystone Queens (duration: 03m 35s) [15:15:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:16:13] (03PS2) 10L0st3xpl0r3r: Changed the return value for run() inside transfer.py [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 [15:18:39] (03CR) 10L0st3xpl0r3r: "> Patch Set 1:" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (owner: 10L0st3xpl0r3r) [15:20:14] (03PS3) 10L0st3xpl0r3r: Changed the return value for run() inside transfer.py [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) [15:24:08] (03CR) 10L0st3xpl0r3r: "> Patch Set 2:" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) (owner: 10L0st3xpl0r3r) [15:25:48] (03CR) 10Bstorm: toolforge: introduce role/profile for legacy URL redirector (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583593 (https://phabricator.wikimedia.org/T247236) (owner: 10Arturo Borrero Gonzalez) [15:27:31] (03CR) 10Jcrespo: "> Can you please guide me on how I can run a local build of this project and test it? For now, all I could do is read the code and try to " [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) (owner: 10L0st3xpl0r3r) [15:27:47] (03PS1) 10Ssingh: Add a profile for the cescout role [puppet] - 10https://gerrit.wikimedia.org/r/583984 (https://phabricator.wikimedia.org/T247273) [15:30:00] (03PS2) 10Jhedden: ceph: refactor ceph-common for clients with no configuration [puppet] - 10https://gerrit.wikimedia.org/r/583964 (https://phabricator.wikimedia.org/T248610) [15:30:11] (03CR) 10Bstorm: "I think this looks good! Is the plan to just have this ready to deploy when the communication is up there and all that?" [puppet] - 10https://gerrit.wikimedia.org/r/583593 (https://phabricator.wikimedia.org/T247236) (owner: 10Arturo Borrero Gonzalez) [15:31:28] (03CR) 10Dzahn: [C: 03+1] "looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/583984 (https://phabricator.wikimedia.org/T247273) (owner: 10Ssingh) [15:33:48] (03PS3) 10Jhedden: ceph: refactor ceph-common for clients with no configuration [puppet] - 10https://gerrit.wikimedia.org/r/583964 (https://phabricator.wikimedia.org/T248610) [15:33:50] (03CR) 10Ssingh: [C: 03+2] Add a profile for the cescout role [puppet] - 10https://gerrit.wikimedia.org/r/583984 (https://phabricator.wikimedia.org/T247273) (owner: 10Ssingh) [15:34:21] 10Operations, 10puppet-compiler, 10User-jbond: populate puppetdb fails for unknown hosts - https://phabricator.wikimedia.org/T248689 (10jbond) p:05Triage→03Medium [15:35:11] (03CR) 10Bstorm: [C: 03+1] "Yay! I read the ticket more fully and answered my question. Looks great." [puppet] - 10https://gerrit.wikimedia.org/r/583593 (https://phabricator.wikimedia.org/T247236) (owner: 10Arturo Borrero Gonzalez) [15:35:39] 10Operations, 10puppet-compiler, 10User-jbond: populate puppetdb fails for unknown hosts - https://phabricator.wikimedia.org/T248689 (10jbond) [15:37:04] 10Operations, 10puppet-compiler, 10User-jbond: populate puppetdb fails for unknown hosts - https://phabricator.wikimedia.org/T248689 (10jbond) [15:38:13] 10Operations, 10puppet-compiler, 10User-jbond: populate puppetdb fails for unknown hosts - https://phabricator.wikimedia.org/T248689 (10jbond) [15:38:21] 10Operations, 10puppet-compiler, 10User-jbond: populate puppetdb fails for unknown hosts - https://phabricator.wikimedia.org/T248689 (10jbond) @herron @colewhite @Joe wonder if any of this makes sense to you or if you have hit similar issues before [15:40:10] (03PS2) 10CRusnov: reports/accounting: remove LRU caching of output [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583723 (owner: 10Faidon Liambotis) [15:41:07] (03CR) 10CRusnov: [V: 03+2 C: 03+2] "Minor change from testing, seems to work otherwise." [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583723 (owner: 10Faidon Liambotis) [15:44:24] (03PS2) 10CRusnov: reports/accounting: avoid evaluating formulas [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 (owner: 10Faidon Liambotis) [15:45:19] (03CR) 10Filippo Giunchedi: [C: 03+1] nagios: Remove support for jessie [puppet] - 10https://gerrit.wikimedia.org/r/583931 (owner: 10Muehlenhoff) [15:45:26] (03PS1) 10Andrew Bogott: nova.conf: update conf to suppress deprecation warnings [puppet] - 10https://gerrit.wikimedia.org/r/583989 (https://phabricator.wikimedia.org/T248635) [15:46:22] hashar: oops, forgot to merge on the master.. in case you were wondering. but it's done now thanks to sukhe [15:47:26] mutante: of what? ;) [15:49:05] hashar: of the renaming AND the revert of it so you never noticed, haha [15:49:14] ahh [15:49:16] (03PS1) 10Andrew Bogott: neutron.conf: update a couple of settings in response to deprecation warnings [puppet] - 10https://gerrit.wikimedia.org/r/583990 (https://phabricator.wikimedia.org/T248635) [15:50:04] mutante: seems good ;) [15:50:14] alright! [15:51:12] then I guess the chain is to have the python2 build docker image build which is https://gerrit.wikimedia.org/r/#/c/operations/docker-images/production-images/+/580128/ [15:51:21] then I can merge the patch in zuul/deploy and we can deploy the puppet patch [15:51:28] and there should be something deployable on the hosts [15:52:06] (03CR) 10Jbond: [C: 03+1] Remove jessie support for puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/583953 (owner: 10Muehlenhoff) [15:53:07] ok [15:55:13] (03PS2) 10Andrew Bogott: nova.conf: update conf to suppress deprecation warnings [puppet] - 10https://gerrit.wikimedia.org/r/583989 (https://phabricator.wikimedia.org/T248635) [15:55:15] (03PS2) 10Andrew Bogott: neutron.conf: update a couple of settings in response to deprecation warnings [puppet] - 10https://gerrit.wikimedia.org/r/583990 (https://phabricator.wikimedia.org/T248635) [15:55:17] (03PS1) 10Andrew Bogott: designate.conf: update a few deprecated settings [puppet] - 10https://gerrit.wikimedia.org/r/583994 (https://phabricator.wikimedia.org/T248635) [15:56:04] (03CR) 10Hashar: [C: 03+1] "The beta cluster and integration WMCS projects had their puppet master migrated to Buster." [puppet] - 10https://gerrit.wikimedia.org/r/583953 (owner: 10Muehlenhoff) [15:57:48] (03CR) 10Andrew Bogott: [C: 03+2] nova.conf: update conf to suppress deprecation warnings [puppet] - 10https://gerrit.wikimedia.org/r/583989 (https://phabricator.wikimedia.org/T248635) (owner: 10Andrew Bogott) [16:03:32] (03CR) 10Hashar: [C: 03+1] "From the header served when fetching artifacts, the CSP is:" [puppet] - 10https://gerrit.wikimedia.org/r/582604 (https://phabricator.wikimedia.org/T245658) (owner: 10Brian Wolff) [16:09:06] (03CR) 10Bstorm: [C: 03+2] "This is by far my favorite new feature. It kind of wants a blog post of its own :) All these merges before cutting the new release will m" [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/578411 (owner: 10BryanDavis) [16:10:09] (03Merged) 10jenkins-bot: Introduce command "template" feature [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/578411 (owner: 10BryanDavis) [16:14:23] (03CR) 10Andrew Bogott: [C: 03+2] designate.conf: update a few deprecated settings [puppet] - 10https://gerrit.wikimedia.org/r/583994 (https://phabricator.wikimedia.org/T248635) (owner: 10Andrew Bogott) [16:15:42] (03PS4) 10L0st3xpl0r3r: transfer.py: Convert return for run() from int to list [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) [16:16:00] (03PS5) 10L0st3xpl0r3r: transfer.py: Convert return for run() from int to list [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) [16:16:23] (03CR) 10Hashar: [C: 03+1] "Another note, Jenkins has recently implemented a feature to have artifacts served from an entirely different domain (for example we could " [puppet] - 10https://gerrit.wikimedia.org/r/582604 (https://phabricator.wikimedia.org/T245658) (owner: 10Brian Wolff) [16:17:46] (03PS6) 10L0st3xpl0r3r: transfer.py: Convert return for run() from int to list [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) [16:20:14] (03CR) 10Bstorm: [C: 03+2] "Yay, quotas! We'd never want this in the old cluster." [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/578412 (owner: 10BryanDavis) [16:20:27] (03PS1) 10Andrew Bogott: neutron.conf: fix an earlier typo with auth_uri vs auth_url [puppet] - 10https://gerrit.wikimedia.org/r/584000 (https://phabricator.wikimedia.org/T248635) [16:20:55] (03Merged) 10jenkins-bot: Add support for Kubernetes replica scaling [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/578412 (owner: 10BryanDavis) [16:21:30] (03PS7) 10L0st3xpl0r3r: transfer.py: Convert return for run() from int to list [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) [16:21:34] (03CR) 10Andrew Bogott: [C: 03+2] neutron.conf: fix an earlier typo with auth_uri vs auth_url [puppet] - 10https://gerrit.wikimedia.org/r/584000 (https://phabricator.wikimedia.org/T248635) (owner: 10Andrew Bogott) [16:22:08] (03PS8) 10L0st3xpl0r3r: transfer.py: Convert return for run() from int to list [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) [16:22:19] (03CR) 10Jcrespo: "Thanks, looking good. For approval, I would say just a unit test would be needed in order to merge. Will provide more details at T248590" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) (owner: 10L0st3xpl0r3r) [16:27:32] PROBLEM - nova instance creation test on cloudcontrol1003 is CRITICAL: PROCS CRITICAL: 0 processes with command name python, args nova-fullstack https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [16:32:52] (03PS1) 10Andrew Bogott: Openstack configs: comment out service_token_roles_required = True [puppet] - 10https://gerrit.wikimedia.org/r/584007 (https://phabricator.wikimedia.org/T248635) [16:33:18] (03PS1) 10Ayounsi: Remove prepending in esams and eqsin [homer/public] - 10https://gerrit.wikimedia.org/r/584008 [16:33:52] (03CR) 10Andrew Bogott: [C: 03+2] Openstack configs: comment out service_token_roles_required = True [puppet] - 10https://gerrit.wikimedia.org/r/584007 (https://phabricator.wikimedia.org/T248635) (owner: 10Andrew Bogott) [16:35:25] (03PS3) 10Jforrester: Check out parsoid deploy modules using git::clone, not scap [puppet] - 10https://gerrit.wikimedia.org/r/577656 (owner: 10C. Scott Ananian) [16:37:56] RECOVERY - nova instance creation test on cloudcontrol1003 is OK: PROCS OK: 1 process with command name python, args nova-fullstack https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting [16:38:56] (03PS1) 10Andrew Bogott: Designate: set enabled = True in the [service:worker] section [puppet] - 10https://gerrit.wikimedia.org/r/584011 (https://phabricator.wikimedia.org/T248635) [16:40:13] (03CR) 10Andrew Bogott: [C: 03+2] Designate: set enabled = True in the [service:worker] section [puppet] - 10https://gerrit.wikimedia.org/r/584011 (https://phabricator.wikimedia.org/T248635) (owner: 10Andrew Bogott) [16:42:11] (03PS9) 10Jcrespo: transfer.py: Convert return for run() from int to list [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/583960 (https://phabricator.wikimedia.org/T248661) (owner: 10L0st3xpl0r3r) [16:46:15] (03CR) 10Jforrester: "Maybe add an explicit row for 'bot' group, then?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/582046 (https://phabricator.wikimedia.org/T248177) (owner: 10Reedy) [16:57:33] 10Operations, 10Performance-Team, 10observability: Consolidate performance website and related software - https://phabricator.wikimedia.org/T158837 (10Krinkle) [17:00:35] (03CR) 10Herron: [C: 03+2] ELk7: add curator job to require disktype hdd after 7 days [puppet] - 10https://gerrit.wikimedia.org/r/579422 (https://phabricator.wikimedia.org/T247376) (owner: 10Herron) [17:03:35] (03CR) 10Bstorm: [C: 04-1] Add support for redirecting to toolforge.org (031 comment) [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/578413 (https://phabricator.wikimedia.org/T234617) (owner: 10BryanDavis) [17:05:33] 10Operations, 10DC-Ops, 10decommission, 10fundraising-tech-ops: decommission heka.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T248628 (10Papaul) ` [edit interfaces interface-range disabled] member "ge-[0-1]/0/4" { ... } + member "ge-[0-1]/0/5"; [edit interfaces interface-range vlan-admini... [17:05:35] James_F: all good regarding T248597 ? [17:05:35] T248597: Grant "contint-roots" and "releasers-mediawiki" to user jforrester - https://phabricator.wikimedia.org/T248597 [17:06:07] 10Operations, 10DC-Ops, 10decommission, 10fundraising-tech-ops: decommission heka.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T248628 (10Papaul) [17:07:16] volans: Seems so, yes. Thank you! [17:07:34] ok resolving then [17:08:49] 10Operations, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO, 10SRE-Access-Requests: Grant "contint-roots" and "releasers-mediawiki" to user jforrester - https://phabricator.wikimedia.org/T248597 (10Volans) 05Open→03Resolved Confirmed all looks good so far, resolving. [17:09:04] (03PS4) 10Jhedden: ceph: refactor ceph-common for clients with no configuration [puppet] - 10https://gerrit.wikimedia.org/r/583964 (https://phabricator.wikimedia.org/T248610) [17:10:15] 10Operations, 10DC-Ops, 10decommission: decommission heka.frack.codfw.wmnet - https://phabricator.wikimedia.org/T248627 (10Jgreen) [17:10:17] 10Operations, 10DC-Ops, 10decommission, 10fundraising-tech-ops: decommission heka.frack.eqiad.wmnet - https://phabricator.wikimedia.org/T248628 (10Jgreen) [17:12:42] 10Operations, 10DC-Ops, 10decommission: decommission heka.frack.codfw.wmnet - https://phabricator.wikimedia.org/T248627 (10Papaul) [edit interfaces interface-range disabled] member "ge-[0-1]/0/4" { ... } + member "ge-[0-1]/0/5"; [edit interfaces interface-range vlan-administration] - member "ge-[0... [17:13:08] (03PS6) 10Mstyles: kibana: refactor kibana profile into two profiles [puppet] - 10https://gerrit.wikimedia.org/r/583414 (https://phabricator.wikimedia.org/T246961) [17:13:40] (03CR) 10Jhedden: [C: 03+2] "PCC results https://puppet-compiler.wmflabs.org/compiler1001/21602/" [puppet] - 10https://gerrit.wikimedia.org/r/583964 (https://phabricator.wikimedia.org/T248610) (owner: 10Jhedden) [17:13:42] (03CR) 10Mstyles: kibana: refactor kibana profile into two profiles (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583414 (https://phabricator.wikimedia.org/T246961) (owner: 10Mstyles) [17:13:43] 10Operations, 10ops-codfw, 10DC-Ops, 10decommission: decommission heka.frack.codfw.wmnet - https://phabricator.wikimedia.org/T248627 (10Papaul) [17:19:17] (03PS4) 10RLazarus: profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job [puppet] - 10https://gerrit.wikimedia.org/r/582933 (https://phabricator.wikimedia.org/T211250) [17:19:19] (03PS1) 10RLazarus: systemd: Replace the Datetime regex with a call to systemd-analyze. [puppet] - 10https://gerrit.wikimedia.org/r/584020 [17:22:00] 10Operations, 10Wikimedia-Logstash, 10observability: Logstash: add SSD tier to ELK7 cluster - https://phabricator.wikimedia.org/T247376 (10herron) [17:22:15] 10Operations, 10Wikimedia-Logstash, 10observability: Logstash: add SSD tier to ELK7 cluster - https://phabricator.wikimedia.org/T247376 (10herron) 05Open→03Resolved [17:22:48] (03CR) 10jerkins-bot: [V: 04-1] systemd: Replace the Datetime regex with a call to systemd-analyze. [puppet] - 10https://gerrit.wikimedia.org/r/584020 (owner: 10RLazarus) [17:32:06] (03PS1) 10Jhedden: openstack: add ceph common profile to virt nodes [puppet] - 10https://gerrit.wikimedia.org/r/584022 (https://phabricator.wikimedia.org/T248610) [17:33:01] (03CR) 10Jhedden: "This uses the refactored ceph module from https://gerrit.wikimedia.org/r/c/operations/puppet/+/583964" [puppet] - 10https://gerrit.wikimedia.org/r/584022 (https://phabricator.wikimedia.org/T248610) (owner: 10Jhedden) [17:35:19] (03CR) 10Jhedden: "PCC results: https://puppet-compiler.wmflabs.org/compiler1001/21603/" [puppet] - 10https://gerrit.wikimedia.org/r/584022 (https://phabricator.wikimedia.org/T248610) (owner: 10Jhedden) [17:53:01] (03CR) 10CDanis: [C: 03+1] Remove prepending in esams and eqsin [homer/public] - 10https://gerrit.wikimedia.org/r/584008 (owner: 10Ayounsi) [17:53:06] (03PS2) 10Jhedden: openstack: add ceph common profile to control and virt nodes [puppet] - 10https://gerrit.wikimedia.org/r/584022 (https://phabricator.wikimedia.org/T248610) [17:55:49] (03CR) 10Jhedden: "Updated host list in PCC: https://puppet-compiler.wmflabs.org/compiler1002/21604/" [puppet] - 10https://gerrit.wikimedia.org/r/584022 (https://phabricator.wikimedia.org/T248610) (owner: 10Jhedden) [18:05:03] (03PS8) 10Giuseppe Lavagetto: systemd::timer::job: fix bug re: On(In)?ActiveUnitSec [puppet] - 10https://gerrit.wikimedia.org/r/551281 (owner: 10CDanis) [18:07:39] <_joe_> cdanis: ^^ [18:07:45] yeah I'm looking :) [18:08:22] (03CR) 10CDanis: "This change is ready for review." [puppet] - 10https://gerrit.wikimedia.org/r/551281 (owner: 10CDanis) [18:09:41] (03CR) 10CDanis: [C: 03+1] systemd::timer::job: fix bug re: On(In)?ActiveUnitSec [puppet] - 10https://gerrit.wikimedia.org/r/551281 (owner: 10CDanis) [18:09:48] (03CR) 10Giuseppe Lavagetto: "https://puppet-compiler.wmflabs.org/compiler1001/21605/mwmaint1002.eqiad.wmnet/ seems to DTRT" [puppet] - 10https://gerrit.wikimedia.org/r/551281 (owner: 10CDanis) [18:09:50] _joe_: looks good to me, thank you [18:10:18] <_joe_> I won't merge it at 7 pm on friday [18:10:22] (03CR) 10Arturo Borrero Gonzalez: "Thanks for the review! Will merge this next monday." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/583593 (https://phabricator.wikimedia.org/T247236) (owner: 10Arturo Borrero Gonzalez) [18:10:25] <_joe_> I know better than doing that :P [18:11:07] (03CR) 10CDanis: [C: 03+1] systemd::timer::job: fix bug re: On(In)?ActiveUnitSec (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/551281 (owner: 10CDanis) [18:11:09] ah I did find one thing [18:11:14] which was my mistake, even :D [18:12:24] (03CR) 10Bstorm: [C: 03+2] Remove legacy cluster images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/577818 (owner: 10BryanDavis) [18:13:17] (03Merged) 10jenkins-bot: Remove legacy cluster images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/577818 (owner: 10BryanDavis) [18:16:53] <_joe_> cdanis: I'm amending [18:17:36] ty _joe_ [18:19:48] (03PS9) 10Giuseppe Lavagetto: systemd::timer::job: fix bug re: On(In)?ActiveUnitSec [puppet] - 10https://gerrit.wikimedia.org/r/551281 (owner: 10CDanis) [18:25:32] 10Operations, 10puppet-compiler, 10User-jbond: populate puppetdb fails for unknown hosts - https://phabricator.wikimedia.org/T248689 (10herron) Spent some time on IRC with @jbond reproducing this and indeed puppetdb-populate will fail repeatedly for new hosts until performing a run using an empty manifest t... [18:32:36] (03PS1) 10Bstorm: toolforge: rebuild the docker::builder setup as buster [puppet] - 10https://gerrit.wikimedia.org/r/584027 (https://phabricator.wikimedia.org/T248703) [18:37:51] (03PS3) 10CRusnov: reports/accounting: avoid evaluating formulas [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 (owner: 10Faidon Liambotis) [18:38:28] 10Operations, 10LDAP-Access-Requests: Add Scardenasmolinar to WMF LDAP group - https://phabricator.wikimedia.org/T248521 (10Scardenasmolinar) @Aklapper: I have just connected my MediaWiki user to Phabricator. Let me know if I need to do something else! [18:38:51] (03CR) 10jerkins-bot: [V: 04-1] reports/accounting: avoid evaluating formulas [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 (owner: 10Faidon Liambotis) [18:40:21] (03CR) 10Faidon Liambotis: reports/accounting: avoid evaluating formulas (031 comment) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 (owner: 10Faidon Liambotis) [18:55:48] (03PS2) 10RLazarus: systemd: Replace the Datetime regex with a call to systemd-analyze. [puppet] - 10https://gerrit.wikimedia.org/r/584020 [18:55:50] (03PS5) 10RLazarus: profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job [puppet] - 10https://gerrit.wikimedia.org/r/582933 (https://phabricator.wikimedia.org/T211250) [18:56:37] (03CR) 10RLazarus: "This change is ready for review." [puppet] - 10https://gerrit.wikimedia.org/r/584020 (owner: 10RLazarus) [18:59:40] (03CR) 10jerkins-bot: [V: 04-1] systemd: Replace the Datetime regex with a call to systemd-analyze. [puppet] - 10https://gerrit.wikimedia.org/r/584020 (owner: 10RLazarus) [18:59:53] (03CR) 10Volans: reports/accounting: avoid evaluating formulas (032 comments) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 (owner: 10Faidon Liambotis) [18:59:55] (03PS4) 10CRusnov: reports/accounting: avoid evaluating formulas [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 (owner: 10Faidon Liambotis) [19:00:07] (03CR) 10Jhedden: "Looks good, few minor comments inline" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/584027 (https://phabricator.wikimedia.org/T248703) (owner: 10Bstorm) [19:01:18] (03CR) 10CRusnov: reports/accounting: avoid evaluating formulas (031 comment) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 (owner: 10Faidon Liambotis) [19:02:47] (03PS3) 10Jhedden: openstack: add ceph common profile to control and virt nodes [puppet] - 10https://gerrit.wikimedia.org/r/584022 (https://phabricator.wikimedia.org/T248610) [19:05:46] (03CR) 10Bstorm: toolforge: rebuild the docker::builder setup as buster (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/584027 (https://phabricator.wikimedia.org/T248703) (owner: 10Bstorm) [19:06:22] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:07:05] (03CR) 10Jhedden: [C: 03+2] openstack: add ceph common profile to control and virt nodes [puppet] - 10https://gerrit.wikimedia.org/r/584022 (https://phabricator.wikimedia.org/T248610) (owner: 10Jhedden) [19:08:59] 10Operations, 10LDAP-Access-Requests: Add Scardenasmolinar to WMF LDAP group - https://phabricator.wikimedia.org/T248521 (10Volans) 05Open→03Resolved @Scardenasmolinar that's perfect, all good. Thanks a lot. [19:09:59] (03CR) 10Volans: [C: 03+1] "LGTM" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 (owner: 10Faidon Liambotis) [19:13:45] (03PS5) 10Faidon Liambotis: reports/accounting: avoid evaluating formulas [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 [19:24:09] (03CR) 10CRusnov: [C: 03+2] "LGTM merging :)" (031 comment) [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/583725 (owner: 10Faidon Liambotis) [19:25:05] (03PS3) 10RLazarus: systemd: Replace the Datetime regex with a call to systemd-analyze. [puppet] - 10https://gerrit.wikimedia.org/r/584020 [19:25:07] (03PS6) 10RLazarus: profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job [puppet] - 10https://gerrit.wikimedia.org/r/582933 (https://phabricator.wikimedia.org/T211250) [19:26:06] (03CR) 10RLazarus: [C: 04-1] "Argh, the reason the generate() is failing on perfectly good timestamps is, systemd-analyze didn't exist until buster. This approach might" [puppet] - 10https://gerrit.wikimedia.org/r/584020 (owner: 10RLazarus) [19:26:44] (03PS2) 10Bstorm: toolforge: rebuild the docker::builder setup as buster [puppet] - 10https://gerrit.wikimedia.org/r/584027 (https://phabricator.wikimedia.org/T248703) [19:27:27] (03CR) 10Bstorm: toolforge: rebuild the docker::builder setup as buster (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/584027 (https://phabricator.wikimedia.org/T248703) (owner: 10Bstorm) [19:28:15] (03CR) 10jerkins-bot: [V: 04-1] systemd: Replace the Datetime regex with a call to systemd-analyze. [puppet] - 10https://gerrit.wikimedia.org/r/584020 (owner: 10RLazarus) [19:32:54] (03PS1) 10Faidon Liambotis: reports/accounting: bump Python minimum to 3.7 [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/584041 [19:32:56] (03PS1) 10Faidon Liambotis: reports/accounting: fix a few docstring issues [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/584042 [19:34:59] (03CR) 10Jhedden: [C: 03+1] toolforge: rebuild the docker::builder setup as buster [puppet] - 10https://gerrit.wikimedia.org/r/584027 (https://phabricator.wikimedia.org/T248703) (owner: 10Bstorm) [19:35:50] (03PS1) 10Jhedden: openstack: codfw1dev add ceph common profile to control and virt roles [puppet] - 10https://gerrit.wikimedia.org/r/584043 (https://phabricator.wikimedia.org/T248610) [19:37:06] (03CR) 10Volans: [C: 03+1] "LGTM" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/584041 (owner: 10Faidon Liambotis) [19:38:22] (03CR) 10Bstorm: [C: 03+2] toolforge: rebuild the docker::builder setup as buster [puppet] - 10https://gerrit.wikimedia.org/r/584027 (https://phabricator.wikimedia.org/T248703) (owner: 10Bstorm) [19:39:11] (03CR) 10Volans: [C: 04-1] "Actually, to make this work we need to modify also tox.ini to remove the old python versions, ideally in a separate patch." [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/584041 (owner: 10Faidon Liambotis) [19:39:51] (03CR) 10Volans: [C: 03+1] "Ignore my last comment, I picked the wrong tox.ini locally..." [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/584041 (owner: 10Faidon Liambotis) [19:40:13] (03CR) 10Jhedden: [C: 03+2] openstack: codfw1dev add ceph common profile to control and virt roles [puppet] - 10https://gerrit.wikimedia.org/r/584043 (https://phabricator.wikimedia.org/T248610) (owner: 10Jhedden) [19:41:44] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:54:12] (03PS1) 10Bstorm: toolforge docker: fix image_builder profile [puppet] - 10https://gerrit.wikimedia.org/r/584046 (https://phabricator.wikimedia.org/T248703) [20:08:10] (03PS7) 10RLazarus: profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job [puppet] - 10https://gerrit.wikimedia.org/r/582933 (https://phabricator.wikimedia.org/T211250) [20:19:39] (03CR) 10Bstorm: [C: 03+2] toolforge docker: fix image_builder profile [puppet] - 10https://gerrit.wikimedia.org/r/584046 (https://phabricator.wikimedia.org/T248703) (owner: 10Bstorm) [20:33:38] (03PS8) 10RLazarus: profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job [puppet] - 10https://gerrit.wikimedia.org/r/582933 (https://phabricator.wikimedia.org/T211250) [20:34:04] (03CR) 10Volans: [C: 03+1] "LGTM" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/584042 (owner: 10Faidon Liambotis) [20:43:29] 10Operations, 10Graphoid, 10Code-Stewardship-Reviews, 10Release-Engineering-Team (Code Health), and 2 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Milimetric) Recent events have compelled me to adopt graphoid. I will personally maintain it until we find a right... [20:45:06] (03PS9) 10RLazarus: profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job [puppet] - 10https://gerrit.wikimedia.org/r/582933 (https://phabricator.wikimedia.org/T211250) [20:47:19] 10Operations, 10Graphoid, 10serviceops, 10Core Platform Team (Icebox): Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Milimetric) I am going to pick up graphoid and maintain it for the foreseeable future. My first priority is to not mess up anyone's plans. So, if it's easier to undeploy... [20:53:01] (03PS10) 10RLazarus: profile::mediawiki::maintenance: Migrate pagetriage jobs to periodic_job [puppet] - 10https://gerrit.wikimedia.org/r/582933 (https://phabricator.wikimedia.org/T211250) [20:58:50] (03PS1) 10Andrew Bogott: Upgrade designate in codfw1dev to Rocky [puppet] - 10https://gerrit.wikimedia.org/r/584057 [21:03:30] (03CR) 10RLazarus: "Forgive the jenkins spam; that took... more attempts than I'd like. Now ready for review. PCC: https://puppet-compiler.wmflabs.org/compile" [puppet] - 10https://gerrit.wikimedia.org/r/582933 (https://phabricator.wikimedia.org/T211250) (owner: 10RLazarus) [21:06:27] (03PS2) 10Andrew Bogott: Upgrade designate in codfw1dev to Rocky [puppet] - 10https://gerrit.wikimedia.org/r/584057 [21:06:29] (03PS1) 10Andrew Bogott: Openstack: add Designate manifest and config for version Rocky [puppet] - 10https://gerrit.wikimedia.org/r/584058 (https://phabricator.wikimedia.org/T248635) [21:08:33] (03CR) 10Andrew Bogott: [C: 03+2] Openstack: add Designate manifest and config for version Rocky [puppet] - 10https://gerrit.wikimedia.org/r/584058 (https://phabricator.wikimedia.org/T248635) (owner: 10Andrew Bogott) [21:11:45] (03CR) 1020after4: [C: 03+1] contint: use package_from_component, stop using docker class [puppet] - 10https://gerrit.wikimedia.org/r/566383 (https://phabricator.wikimedia.org/T224591) (owner: 10Dzahn) [21:13:25] (03CR) 10Andrew Bogott: [C: 03+2] Upgrade designate in codfw1dev to Rocky [puppet] - 10https://gerrit.wikimedia.org/r/584057 (owner: 10Andrew Bogott) [21:14:02] (03CR) 10Muehlenhoff: "For the new Buster-based build host which builds the production CI images we've switched to using the Docker package from Buster:" [puppet] - 10https://gerrit.wikimedia.org/r/566383 (https://phabricator.wikimedia.org/T224591) (owner: 10Dzahn) [21:19:35] (03PS1) 10Bstorm: toolforge: remove the old docker builder code [puppet] - 10https://gerrit.wikimedia.org/r/584059 (https://phabricator.wikimedia.org/T248703) [21:32:09] (03CR) 10RLazarus: [C: 03+1] Make configuration of envoy a ConfigMap (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/582777 (https://phabricator.wikimedia.org/T244843) (owner: 10Giuseppe Lavagetto) [21:36:40] (03PS1) 10Bstorm: toolforge-k8s: mount /var/lib/docker on appropriate volume [puppet] - 10https://gerrit.wikimedia.org/r/584061 (https://phabricator.wikimedia.org/T248702) [21:40:49] (03CR) 10Bstorm: [C: 03+2] Introduce jinja2 templating (031 comment) [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/578165 (owner: 10BryanDavis) [21:40:59] (03PS1) 10CDanis: fix check_trafficserver_log_fifo UNKNOWNs due to timeout [puppet] - 10https://gerrit.wikimedia.org/r/584063 [21:41:13] (03Merged) 10jenkins-bot: Introduce jinja2 templating [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/578165 (owner: 10BryanDavis) [21:43:12] (03CR) 10CDanis: [C: 03+2] "PCC looks good https://puppet-compiler.wmflabs.org/compiler1002/21610/cp1089.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/584063 (owner: 10CDanis) [21:44:47] (03CR) 10Bstorm: "I didn't include the control plane nodes because they use very little of their disk in general since they only run 5 pods or so each, and " [puppet] - 10https://gerrit.wikimedia.org/r/584061 (https://phabricator.wikimedia.org/T248702) (owner: 10Bstorm) [21:45:45] (03CR) 10Bstorm: [C: 04-1] "-1 until this is ready to go (which means all the workers have puppet disabled and this can be applied carefully with each worker depooled" [puppet] - 10https://gerrit.wikimedia.org/r/584061 (https://phabricator.wikimedia.org/T248702) (owner: 10Bstorm) [21:47:32] (03CR) 10Bstorm: [C: 03+2] Add rate limiting to profile::toolforge::mailrelay with warn action [puppet] - 10https://gerrit.wikimedia.org/r/379239 (https://phabricator.wikimedia.org/T175964) (owner: 10Herron) [23:07:44] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [23:14:48] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is OK: HTTP OK: HTTP/1.1 200 Ok - 31903 bytes in 3.912 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [23:21:06] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [23:44:12] RECOVERY - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is OK: HTTP OK: HTTP/1.1 200 Ok - 31900 bytes in 9.824 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server [23:50:26] PROBLEM - Ensure traffic_manager binds on 443 and responds to HTTP requests on cp1089 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server