[00:02:28] 10Operations, 10Ops-Access-Reviews, 10Reading-Infrastructure-Team-Backlog, 10Patch-For-Review: Add Michael Holloway (Reading Infrastructure) to maps admin groups - https://phabricator.wikimedia.org/T194404#4199052 (10Dzahn) p:05Triage>03Normal [00:05:29] 10Operations, 10Reading-Infrastructure-Team-Backlog, 10SRE-Access-Requests, 10Patch-For-Review: Add Michael Holloway (Reading Infrastructure) to maps admin groups - https://phabricator.wikimedia.org/T194404#4199054 (10Dzahn) [00:08:02] 10Operations, 10SRE-Access-Requests: Access to people.wikimedia.org for Volker_E - https://phabricator.wikimedia.org/T143465#4199065 (10RobH) [00:08:04] 10Ops-Access-Reviews: basion/rutherfordium access for Volker_E - https://phabricator.wikimedia.org/T143579#4199064 (10RobH) 05Open>03Invalid [00:19:01] 10Operations, 10SRE-Access-Requests: Give Seddon access to the analytics cluster - https://phabricator.wikimedia.org/T194445#4199090 (10Milimetric) [02:51:05] 10Operations, 10SRE-Access-Requests: Give Seddon access to the analytics cluster - https://phabricator.wikimedia.org/T194445#4199090 (10Dzahn) Hi @Jseddon Let's make sure that is really the right group for what you need. What specifically do you need access to? Could you add a little context what this is for?... [02:52:27] 10Operations, 10vm-requests: EQIAD & CODFW: 1 VM in each data center for xhprof/xhgui/other profiling tools - https://phabricator.wikimedia.org/T194390#4199195 (10Dzahn) a:03Dzahn [03:02:01] 10Operations, 10Performance-Team, 10Graphite: Certain graphite data directories should be backed up - https://phabricator.wikimedia.org/T194418#4199198 (10Dzahn) p:05Triage>03Normal [03:05:42] (03PS1) 10Dzahn: graphite: add backup::host and backup::set [puppet] - 10https://gerrit.wikimedia.org/r/432547 (https://phabricator.wikimedia.org/T194418) [03:09:01] (03PS2) 10Dzahn: graphite: add backup of /var/lib/carbon/whisper/coal/ [puppet] - 10https://gerrit.wikimedia.org/r/432547 (https://phabricator.wikimedia.org/T194418) [03:17:53] 10Operations, 10SRE-Access-Requests: Give Seddon access to the analytics cluster - https://phabricator.wikimedia.org/T194445#4199202 (10Jseddon) Hey @Dzahn, My current need access to stat1005 so that I can be able to get near real-time page views data for the target pages used in central-notice campaigns. Th... [03:30:02] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 968.20 seconds [04:18:01] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 101.57 seconds [05:04:26] (03CR) 10Nemo bis: "Thanks for working on this (although the T194032 spammer may or may not be in those lists, right?)." [puppet] - 10https://gerrit.wikimedia.org/r/432168 (https://phabricator.wikimedia.org/T194032) (owner: 10Herron) [05:55:42] (03CR) 10KartikMistry: "recheck" [debs/contenttranslation/apertium-streamparser] - 10https://gerrit.wikimedia.org/r/431553 (https://phabricator.wikimedia.org/T192978) (owner: 10KartikMistry) [05:55:52] (03CR) 10jerkins-bot: [V: 04-1] apertium-streamparser: Initial Debian packaging [debs/contenttranslation/apertium-streamparser] - 10https://gerrit.wikimedia.org/r/431553 (https://phabricator.wikimedia.org/T192978) (owner: 10KartikMistry) [05:58:56] (03PS4) 10KartikMistry: apertium-streamparser: Initial Debian packaging [debs/contenttranslation/apertium-streamparser] - 10https://gerrit.wikimedia.org/r/431553 (https://phabricator.wikimedia.org/T192987) [05:59:05] (03CR) 10jerkins-bot: [V: 04-1] apertium-streamparser: Initial Debian packaging [debs/contenttranslation/apertium-streamparser] - 10https://gerrit.wikimedia.org/r/431553 (https://phabricator.wikimedia.org/T192987) (owner: 10KartikMistry) [06:07:48] (03PS5) 10KartikMistry: apertium-streamparser: Initial Debian packaging [debs/contenttranslation/apertium-streamparser] - 10https://gerrit.wikimedia.org/r/431553 (https://phabricator.wikimedia.org/T192987) [06:08:39] (03PS2) 10Gilles: Add .gitreview file [debs/python-logstash] - 10https://gerrit.wikimedia.org/r/430306 [06:11:11] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1974 bytes in 0.104 second response time [06:45:47] (03PS1) 10Urbanecm: Change logo for wikimania2018wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432549 (https://phabricator.wikimedia.org/T194340) [06:48:01] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1953 bytes in 0.104 second response time [07:00:13] !log depool and upgrade/restart of dbproxy1011 [07:00:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:03:11] PROBLEM - puppet last run on analytics1028 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[initramfs-tools] [07:05:50] this is probably my fault --^ [07:08:31] RECOVERY - puppet last run on analytics1028 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:16:51] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1968 bytes in 0.106 second response time [07:44:30] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM, note that bacula::director::fileset { 'var-lib-carbon-whisper': can be removed instead, it is unused." [puppet] - 10https://gerrit.wikimedia.org/r/432547 (https://phabricator.wikimedia.org/T194418) (owner: 10Dzahn) [07:48:32] (03PS1) 10Urbanecm: Change liwikibooks logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432551 (https://phabricator.wikimedia.org/T193680) [07:49:29] (03PS6) 10Filippo Giunchedi: prometheus: Add varnishrls aggregation rules [puppet] - 10https://gerrit.wikimedia.org/r/432090 (https://phabricator.wikimedia.org/T190978) (owner: 10Krinkle) [07:51:04] (03CR) 10Filippo Giunchedi: [C: 032] prometheus: Add varnishrls aggregation rules [puppet] - 10https://gerrit.wikimedia.org/r/432090 (https://phabricator.wikimedia.org/T190978) (owner: 10Krinkle) [07:59:03] !log reimage analytics1035 to Debian Stretch [07:59:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:20:18] (03PS1) 10Jcrespo: mariadb: Move db1066 from s1 to an s2 master candidate [puppet] - 10https://gerrit.wikimedia.org/r/432552 (https://phabricator.wikimedia.org/T186320) [08:23:14] (03PS1) 10Jcrespo: mariadb-server-package: Upgrade mariadb package and merge with mysql [software] - 10https://gerrit.wikimedia.org/r/432553 [08:24:55] (03PS1) 10Jcrespo: dbhosts: Move db1066 from s1 to s2 [software] - 10https://gerrit.wikimedia.org/r/432554 (https://phabricator.wikimedia.org/T186320) [08:25:52] (03PS2) 10Jcrespo: dbhosts: Move db1066 from s1 to s2 [software] - 10https://gerrit.wikimedia.org/r/432554 (https://phabricator.wikimedia.org/T186320) [08:25:59] (03CR) 10Jcrespo: [V: 032 C: 032] dbhosts: Move db1066 from s1 to s2 [software] - 10https://gerrit.wikimedia.org/r/432554 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [08:26:38] (03CR) 10Jcrespo: [C: 032] mariadb-server-package: Upgrade mariadb package and merge with mysql [software] - 10https://gerrit.wikimedia.org/r/432553 (owner: 10Jcrespo) [08:46:08] (03PS3) 10Ema: prometheus: varnish_thumbnails aggregation rule [puppet] - 10https://gerrit.wikimedia.org/r/431528 (https://phabricator.wikimedia.org/T184942) [08:47:39] (03CR) 10Ema: prometheus: varnish_thumbnails aggregation rule (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/431528 (https://phabricator.wikimedia.org/T184942) (owner: 10Ema) [08:48:56] (03PS2) 10Jcrespo: mariadb: Move db1066 from s1 to an s2 master candidate [puppet] - 10https://gerrit.wikimedia.org/r/432552 (https://phabricator.wikimedia.org/T186320) [08:48:59] (03PS1) 10Jcrespo: mariadb: Revert parallel replication on labsdb [puppet] - 10https://gerrit.wikimedia.org/r/432556 (https://phabricator.wikimedia.org/T194343) [08:51:27] (03PS2) 10Jcrespo: mariadb: Revert parallel replication on labsdb [puppet] - 10https://gerrit.wikimedia.org/r/432556 (https://phabricator.wikimedia.org/T194343) [08:52:49] (03CR) 10Jcrespo: [C: 032] mariadb: Revert parallel replication on labsdb [puppet] - 10https://gerrit.wikimedia.org/r/432556 (https://phabricator.wikimedia.org/T194343) (owner: 10Jcrespo) [08:52:55] (03PS3) 10Jcrespo: mariadb: Revert parallel replication on labsdb [puppet] - 10https://gerrit.wikimedia.org/r/432556 (https://phabricator.wikimedia.org/T194343) [08:56:54] RECOVERY - Check systemd state on labtestcontrol2003 is OK: OK - running: The system is fully operational [08:57:14] (03CR) 10Filippo Giunchedi: [C: 031] prometheus: varnish_thumbnails aggregation rule [puppet] - 10https://gerrit.wikimedia.org/r/431528 (https://phabricator.wikimedia.org/T184942) (owner: 10Ema) [08:58:19] (03PS4) 10Ema: prometheus: varnish_thumbnails aggregation rule [puppet] - 10https://gerrit.wikimedia.org/r/431528 (https://phabricator.wikimedia.org/T184942) [08:58:54] (03CR) 10Ema: [C: 032] prometheus: varnish_thumbnails aggregation rule [puppet] - 10https://gerrit.wikimedia.org/r/431528 (https://phabricator.wikimedia.org/T184942) (owner: 10Ema) [09:01:13] (03PS1) 10Arturo Borrero Gonzalez: prometheus: rabbitmq_exporter: add missing directory [puppet] - 10https://gerrit.wikimedia.org/r/432557 [09:01:16] RECOVERY - puppet last run on labtestcontrol2003 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [09:01:35] (03CR) 10jerkins-bot: [V: 04-1] prometheus: rabbitmq_exporter: add missing directory [puppet] - 10https://gerrit.wikimedia.org/r/432557 (owner: 10Arturo Borrero Gonzalez) [09:02:33] (03PS2) 10Arturo Borrero Gonzalez: prometheus: rabbitmq_exporter: add missing directory [puppet] - 10https://gerrit.wikimedia.org/r/432557 [09:03:28] (03PS3) 10Arturo Borrero Gonzalez: prometheus: rabbitmq_exporter: add missing directory [puppet] - 10https://gerrit.wikimedia.org/r/432557 [09:04:18] (03CR) 10Arturo Borrero Gonzalez: [C: 032] prometheus: rabbitmq_exporter: add missing directory [puppet] - 10https://gerrit.wikimedia.org/r/432557 (owner: 10Arturo Borrero Gonzalez) [09:13:04] (03PS3) 10Jcrespo: mariadb: Move db1066 from s1 to an s2 master candidate [puppet] - 10https://gerrit.wikimedia.org/r/432552 (https://phabricator.wikimedia.org/T186320) [09:14:22] (03CR) 10Jcrespo: [C: 032] mariadb: Move db1066 from s1 to an s2 master candidate [puppet] - 10https://gerrit.wikimedia.org/r/432552 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [09:20:24] (03PS1) 10Jcrespo: mariadb: Depool db1076 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432560 (https://phabricator.wikimedia.org/T186320) [09:22:43] (03CR) 10Jcrespo: [C: 032] mariadb: Depool db1076 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432560 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [09:23:57] (03Merged) 10jenkins-bot: mariadb: Depool db1076 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432560 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [09:25:46] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1076 (duration: 01m 02s) [09:25:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:26:56] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1948 bytes in 0.116 second response time [09:30:01] (03CR) 10jenkins-bot: mariadb: Depool db1076 for maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432560 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [09:30:26] !log stopping db1076 for maintenance [09:30:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:53:21] Random Question: The next update of the localization will be on Monday right? [09:54:12] *deployment [09:59:51] I think so [10:00:04] Or it might be saturday [10:00:28] Nope, Monday, yeah [10:00:28] weekday => ['1', '2', '3', '4'], [10:07:19] !log reimage analytics1052 to Debian Stretch (Hadoop Journal node) [10:07:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:22:07] good day everyone. Asking on behalf of Wikibase Use Group: Is there anyone here I could talk to about creating a mailing list (context: https://phabricator.wikimedia.org/T189674) [10:22:15] robh: would that be you? ^ [10:25:11] 10Operations, 10Wikimedia-Mailing-lists: Create wikibaseug mailing list - https://phabricator.wikimedia.org/T189674#4199824 (10Reedy) [10:26:20] Probably the on duty opsen [10:28:48] Reedy: thanks. that would be who? [10:28:55] mutante? [10:29:01] sorry, I am making lot of noise here [10:37:42] (03PS4) 10Arturo Borrero Gonzalez: openstack: neutron: nova.conf: enable options [puppet] - 10https://gerrit.wikimedia.org/r/432130 (https://phabricator.wikimedia.org/T193657) [10:50:40] (03PS6) 10Rduran: [WIP] Use Cumin to implement the comunication for the transfer [puppet] - 10https://gerrit.wikimedia.org/r/430868 (https://phabricator.wikimedia.org/T156462) [10:59:21] (03CR) 10Arturo Borrero Gonzalez: [C: 04-1] "This change produces catalog errors on labtestcontrol2003.wikimedia.org and labtestvirt2003.codfw.wmnet:" [puppet] - 10https://gerrit.wikimedia.org/r/432130 (https://phabricator.wikimedia.org/T193657) (owner: 10Arturo Borrero Gonzalez) [11:11:35] (03PS3) 10Arturo Borrero Gonzalez: openstack: neutron: api-paste.ini: enable options [puppet] - 10https://gerrit.wikimedia.org/r/432374 (https://phabricator.wikimedia.org/T193657) [11:13:06] (03CR) 10Arturo Borrero Gonzalez: [C: 032] openstack: neutron: api-paste.ini: enable options [puppet] - 10https://gerrit.wikimedia.org/r/432374 (https://phabricator.wikimedia.org/T193657) (owner: 10Arturo Borrero Gonzalez) [11:18:21] (03PS1) 10Elukey: profile::hadoop::worker: drop Debian Jessie support [puppet] - 10https://gerrit.wikimedia.org/r/432564 (https://phabricator.wikimedia.org/T192557) [11:36:20] (03PS1) 10Jcrespo: mariadb: Productionize db1066 after reimage [puppet] - 10https://gerrit.wikimedia.org/r/432565 (https://phabricator.wikimedia.org/T186320) [11:39:09] (03PS1) 10Jcrespo: mariadb: Pool db1066 with low load after reimage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432566 (https://phabricator.wikimedia.org/T186320) [11:42:47] (03CR) 10Jcrespo: [C: 032] mariadb: Productionize db1066 after reimage [puppet] - 10https://gerrit.wikimedia.org/r/432565 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [11:48:57] !log making change_tag_def table on all wikis (T194302) [11:49:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:49:02] T194302: Schema change for new change_tag_def - https://phabricator.wikimedia.org/T194302 [11:52:07] (03CR) 10Imarlier: [C: 031] "Looks good to me." [puppet] - 10https://gerrit.wikimedia.org/r/432547 (https://phabricator.wikimedia.org/T194418) (owner: 10Dzahn) [12:02:55] (03CR) 10Jcrespo: [C: 032] mariadb: Pool db1066 with low load after reimage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432566 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [12:04:07] (03Merged) 10jenkins-bot: mariadb: Pool db1066 with low load after reimage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432566 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [12:08:39] (03PS7) 10Rduran: [WIP] Use Cumin to implement the comunication for the transfer [puppet] - 10https://gerrit.wikimedia.org/r/430868 (https://phabricator.wikimedia.org/T156462) [12:08:41] (03PS1) 10Rduran: [WIP] Refactor code in transfer.py [puppet] - 10https://gerrit.wikimedia.org/r/432569 (https://phabricator.wikimedia.org/T156462) [12:09:37] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1066 with low load (duration: 01m 02s) [12:09:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:09:52] (03CR) 10jenkins-bot: mariadb: Pool db1066 with low load after reimage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432566 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [12:11:01] (03PS1) 10Ladsgroup: mediawiki: remove stopped deleteAutoPatrol script for wikidata [puppet] - 10https://gerrit.wikimedia.org/r/432570 (https://phabricator.wikimedia.org/T189596) [12:26:53] (03PS1) 10Elukey: Release version 0.11.0-1 [debs/druid] (debian) - 10https://gerrit.wikimedia.org/r/432571 (https://phabricator.wikimedia.org/T193712) [12:29:28] (03CR) 10Jcrespo: [C: 032] mediawiki: remove stopped deleteAutoPatrol script for wikidata [puppet] - 10https://gerrit.wikimedia.org/r/432570 (https://phabricator.wikimedia.org/T189596) (owner: 10Ladsgroup) [12:29:56] (03CR) 10Elukey: [C: 032] Release version 0.11.0-1 [debs/druid] (debian) - 10https://gerrit.wikimedia.org/r/432571 (https://phabricator.wikimedia.org/T193712) (owner: 10Elukey) [12:55:45] (03PS1) 10Jcrespo: mariadb: Allow reimage of db107* hosts [puppet] - 10https://gerrit.wikimedia.org/r/432575 (https://phabricator.wikimedia.org/T186320) [12:56:00] (03PS1) 10Jcrespo: Revert "mariadb: Depool db1076 for maintenance" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432576 [12:56:08] (03Abandoned) 10Jcrespo: Revert "mariadb: Depool db1076 for maintenance" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432576 (owner: 10Jcrespo) [12:58:56] (03CR) 10Jcrespo: [C: 032] mariadb: Allow reimage of db107* hosts [puppet] - 10https://gerrit.wikimedia.org/r/432575 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [13:03:20] (03PS1) 10Jcrespo: install_server: Revert db recipe for all databases [puppet] - 10https://gerrit.wikimedia.org/r/432577 (https://phabricator.wikimedia.org/T186320) [13:29:53] (03CR) 10Jcrespo: [C: 032] install_server: Revert db recipe for all databases [puppet] - 10https://gerrit.wikimedia.org/r/432577 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [13:32:45] !log reloading haproxy configuration for dbproxy1011 to point to labsdb1011 [13:32:46] PROBLEM - Device not healthy -SMART- on db1066 is CRITICAL: cluster=mysql device=megaraid,6 instance=db1066:9100 job=node site=eqiad https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=db1066&var-datasource=eqiad%2520prometheus%252Fops [13:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:37:46] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1965 bytes in 0.109 second response time [13:39:53] !log stopping and restarting labsdb1009 for upgrade [13:39:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:42:56] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1960 bytes in 0.107 second response time [13:52:09] (03PS1) 10Jcrespo: mariadb: Pool db1076 with low load, increase db1066 load [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432581 (https://phabricator.wikimedia.org/T186320) [13:53:47] (03PS1) 10Elukey: role::druid::(analytics|public)::worker: upgrade to Druid 0.11.0 [puppet] - 10https://gerrit.wikimedia.org/r/432582 (https://phabricator.wikimedia.org/T193712) [13:55:55] (03PS1) 10Volans: Exclude all tests from distributed packages [software/debmonitor] - 10https://gerrit.wikimedia.org/r/432585 (https://phabricator.wikimedia.org/T167504) [13:57:51] (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler02/11189/druid1001.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/432582 (https://phabricator.wikimedia.org/T193712) (owner: 10Elukey) [13:58:38] (03CR) 10Elukey: "@Andrew: there are a lot of things not required anymore by puppet now, but I think it should be fine. Will try to apply it in labs first." [puppet] - 10https://gerrit.wikimedia.org/r/432582 (https://phabricator.wikimedia.org/T193712) (owner: 10Elukey) [14:00:15] (03CR) 10Jcrespo: [C: 032] mariadb: Pool db1076 with low load, increase db1066 load [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432581 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [14:01:27] (03Merged) 10jenkins-bot: mariadb: Pool db1076 with low load, increase db1066 load [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432581 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [14:01:44] (03CR) 10jenkins-bot: mariadb: Pool db1076 with low load, increase db1066 load [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432581 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [14:01:56] (03CR) 10Volans: [C: 032] Exclude all tests from distributed packages [software/debmonitor] - 10https://gerrit.wikimedia.org/r/432585 (https://phabricator.wikimedia.org/T167504) (owner: 10Volans) [14:02:41] (03Merged) 10jenkins-bot: Exclude all tests from distributed packages [software/debmonitor] - 10https://gerrit.wikimedia.org/r/432585 (https://phabricator.wikimedia.org/T167504) (owner: 10Volans) [14:13:42] !log restart Hadoop daemons on analytics100[12] for openjdk security upgrades [14:13:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:14:49] !log rebooting labvirt1001 for T194258 [14:14:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:25:44] (03PS1) 10Gergő Tisza: Remove $wgNamespacesWithSubpages overrides for NS_TEMPLATE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432587 (https://phabricator.wikimedia.org/T191612) [14:28:14] !log restart kafka brokers on kafka10[20,22,23] to pick up openjdk-7 security upgrades [14:28:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:28:18] (03PS3) 10Andrew Bogott: keystonehooks: Update our create_project monkeypatch to match Mitaka upstream [puppet] - 10https://gerrit.wikimedia.org/r/432040 [14:28:20] (03PS1) 10Andrew Bogott: nova: whitelist a kernel version with spectre fixes [puppet] - 10https://gerrit.wikimedia.org/r/432588 (https://phabricator.wikimedia.org/T194258) [14:29:02] (03CR) 10Andrew Bogott: [C: 032] nova: whitelist a kernel version with spectre fixes [puppet] - 10https://gerrit.wikimedia.org/r/432588 (https://phabricator.wikimedia.org/T194258) (owner: 10Andrew Bogott) [14:30:54] !log reset labsdb proxies to its config defaults after rolling restart for upgrade [14:30:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:34:57] (03CR) 10Anomie: "Change itself looks sane." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432587 (https://phabricator.wikimedia.org/T191612) (owner: 10Gergő Tisza) [14:37:24] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1076 with low load, increase db1066 load (duration: 01m 02s) [14:37:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:04:09] (03PS5) 10Volans: debmonitor: add server side puppettization [puppet] - 10https://gerrit.wikimedia.org/r/430881 (https://phabricator.wikimedia.org/T191299) [15:10:56] (03PS1) 10Volans: Initial working version [software/debmonitor/deploy] - 10https://gerrit.wikimedia.org/r/432597 (https://phabricator.wikimedia.org/T191299) [15:56:07] PROBLEM - Router interfaces on cr1-eqdfw is CRITICAL: CRITICAL: host 208.80.153.198, interfaces up: 35, down: 1, dormant: 0, excluded: 0, unused: 0 [15:58:14] (03PS1) 10Jcrespo: mariadb: Pool db1076 back with full weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432602 (https://phabricator.wikimedia.org/T186320) [16:03:47] RECOVERY - Router interfaces on cr1-eqdfw is OK: OK: host 208.80.153.198, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 [16:06:50] leszek_wmde: please create a ticket with the tag "wikimedia-mailing-lists" [16:07:38] mutante: afaict the ticket already has this tag: https://phabricator.wikimedia.org/T189674 [16:07:44] (03PS1) 10Jcrespo: mariadb: Remove x1 from dbstore2001, trying to minimize load there [puppet] - 10https://gerrit.wikimedia.org/r/432603 [16:11:18] 10Operations, 10ops-eqdfw, 10netops: eqdfw: Patch GTT cross-connect - https://phabricator.wikimedia.org/T194515#4200613 (10ayounsi) p:05Triage>03Normal [16:13:48] leszek_wmde: ok, i'll do it today as part of being on duty [16:14:00] mutante: great, thank you! [16:14:06] (03CR) 10Jcrespo: [C: 032] mariadb: Remove x1 from dbstore2001, trying to minimize load there [puppet] - 10https://gerrit.wikimedia.org/r/432603 (owner: 10Jcrespo) [16:14:21] mutante: I am calling it a day now. Will get back to you on Monday to thank you again! [16:20:57] !log removing x1 from dbstore2001 [16:21:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:26:35] !log restarting mariadb@s8 at dbstore2001 [16:26:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:30:22] PROBLEM - MariaDB Slave SQL: s5 on dbstore2001 is CRITICAL: CRITICAL slave_sql_state could not connect [16:30:31] PROBLEM - MariaDB Slave IO: s7 on dbstore2001 is CRITICAL: CRITICAL slave_io_state could not connect [16:31:11] PROBLEM - MariaDB Slave IO: s5 on dbstore2001 is CRITICAL: CRITICAL slave_io_state could not connect [16:31:11] PROBLEM - MariaDB Slave SQL: s7 on dbstore2001 is CRITICAL: CRITICAL slave_sql_state could not connect [16:33:16] ^that is me, having discovered a bug on the systemd unit [16:33:56] the servers are fine, but the socket was lost on /run dir destroy and recreation [16:34:26] (03PS3) 10Dzahn: graphite: add backup of /var/lib/carbon/whisper/coal/ [puppet] - 10https://gerrit.wikimedia.org/r/432547 (https://phabricator.wikimedia.org/T194418) [16:56:12] (03CR) 10Dzahn: [C: 032] graphite: add backup of /var/lib/carbon/whisper/coal/ [puppet] - 10https://gerrit.wikimedia.org/r/432547 (https://phabricator.wikimedia.org/T194418) (owner: 10Dzahn) [16:57:02] RECOVERY - MariaDB Slave IO: s5 on dbstore2001 is OK: OK slave_io_state Slave_IO_Running: Yes [16:57:11] RECOVERY - MariaDB Slave SQL: s7 on dbstore2001 is OK: OK slave_sql_state Slave_SQL_Running: Yes [16:57:31] RECOVERY - MariaDB Slave SQL: s5 on dbstore2001 is OK: OK slave_sql_state Slave_SQL_Running: Yes [16:57:41] RECOVERY - MariaDB Slave IO: s7 on dbstore2001 is OK: OK slave_io_state Slave_IO_Running: Yes [17:00:31] PROBLEM - Check systemd state on graphite1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [17:05:31] RECOVERY - Check systemd state on graphite1001 is OK: OK - running: The system is fully operational [17:06:32] PROBLEM - Check systemd state on graphite1003 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [17:07:41] PROBLEM - Check systemd state on graphite2002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [17:08:04] well, that is related to me adding backup [17:08:14] and on 1001 it recovered after the next puppet run [17:08:16] 10Operations, 10SRE-Access-Requests: Access to Google Search Console for Go Fish Digital - https://phabricator.wikimedia.org/T192893#4153156 (10JKatzWMF) @RobH Just a friendly nudge, since I believe this is no longer "Awaiting user input" as the column on your board indicates (and is time-sensitive). Thanks! [17:08:28] it's because i'm adding a fileset and using it in the same change [17:08:40] (03PS1) 10Dzahn: bacula: remove unused fileset var-lib-carbon-whisper [puppet] - 10https://gerrit.wikimedia.org/r/432610 (https://phabricator.wikimedia.org/T194418) [17:09:21] PROBLEM - Check systemd state on graphite2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [17:10:12] RECOVERY - Check systemd state on graphite2002 is OK: OK - running: The system is fully operational [17:10:22] RECOVERY - Check systemd state on graphite1003 is OK: OK - running: The system is fully operational [17:10:27] (03PS2) 10Dzahn: bacula: remove unused fileset var-lib-carbon-whisper [puppet] - 10https://gerrit.wikimedia.org/r/432610 (https://phabricator.wikimedia.org/T194418) [17:10:32] RECOVERY - Check systemd state on graphite2001 is OK: OK - running: The system is fully operational [17:10:32] (03CR) 10Dzahn: [C: 032] bacula: remove unused fileset var-lib-carbon-whisper [puppet] - 10https://gerrit.wikimedia.org/r/432610 (https://phabricator.wikimedia.org/T194418) (owner: 10Dzahn) [17:10:48] (03CR) 10Dzahn: [C: 032] "thanks! removing the unused file set here: https://gerrit.wikimedia.org/r/#/c/432610/" [puppet] - 10https://gerrit.wikimedia.org/r/432547 (https://phabricator.wikimedia.org/T194418) (owner: 10Dzahn) [17:11:11] 10Operations, 10ops-eqiad, 10netops, 10Patch-For-Review: Rack/cable/configure asw2-c-eqiad switch stack - https://phabricator.wikimedia.org/T187962#4200809 (10ayounsi) [17:11:31] 10Operations, 10SRE-Access-Requests: Access to Google Search Console for Go Fish Digital - https://phabricator.wikimedia.org/T192893#4200810 (10RobH) @JKatzWMF: Rest assured, I'm well aware of the time sensitivity. This is currently awaiting WMF legal approval within an email thread that includes both @Deska... [17:13:37] 10Operations, 10ops-eqiad, 10DBA: Move db1066 to row A - https://phabricator.wikimedia.org/T193847#4181453 (10jcrespo) db1066 is now pooled on s2, so it will need a depooling before shutting it down (and probably disk changes). [17:13:41] 10Operations, 10SRE-Access-Requests: Access to Google Search Console for Go Fish Digital - https://phabricator.wikimedia.org/T192893#4200815 (10RobH) Please note that in addition to the email thread, I've documented the process for [[ https://wikitech.wikimedia.org/wiki/Google_Search_Console_access | requesti... [17:14:01] 10Operations, 10SRE-Access-Requests: Access to Google Search Console for Go Fish Digital - https://phabricator.wikimedia.org/T192893#4200816 (10RStallman-legalteam) Confirming that the Master Services Agreement we have on file with Go Fish has an NDA, so all set there. [17:15:54] 10Operations, 10ops-eqdfw, 10netops: eqdfw: Patch GTT cross-connect - https://phabricator.wikimedia.org/T194515#4200820 (10ayounsi) [17:16:07] (03CR) 10Jcrespo: [C: 032] mariadb: Pool db1076 back with full weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432602 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [17:17:35] (03Merged) 10jenkins-bot: mariadb: Pool db1076 back with full weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432602 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [17:18:07] 10Operations, 10Performance-Team, 10Graphite, 10Patch-For-Review: Certain graphite data directories should be backed up - https://phabricator.wikimedia.org/T194418#4200826 (10Dzahn) a:03Dzahn [17:19:25] 10Operations, 10SRE-Access-Requests: Access to Google Search Console for Go Fish Digital - https://phabricator.wikimedia.org/T192893#4200827 (10JKatzWMF) /me tiptoes away, embarrassed. Sorry, Rob. Classic outsider move and I'll back off. Thanks for the additional work you're putting into standardizing this,... [17:20:05] (03CR) 10jenkins-bot: mariadb: Pool db1076 back with full weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432602 (https://phabricator.wikimedia.org/T186320) (owner: 10Jcrespo) [17:21:06] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1076 with full weight (duration: 01m 02s) [17:21:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:28:32] (03PS1) 10Elukey: profile::prometheus::alerts: reduce the noise for the Druid alarms [puppet] - 10https://gerrit.wikimedia.org/r/432613 [17:31:49] 10Operations, 10SRE-Access-Requests: Access to Google Search Console for Go Fish Digital - https://phabricator.wikimedia.org/T192893#4200854 (10RobH) >>! In T192893#4200827, @JKatzWMF wrote: > /me tiptoes away, embarrassed. Sorry, @RobH. Classic outsider move and I'll back off. Thanks for the additional wor... [17:36:24] (03PS2) 10Elukey: profile::prometheus::alerts: reduce the noise for the Druid alarms [puppet] - 10https://gerrit.wikimedia.org/r/432613 [17:37:23] (03CR) 10Elukey: [C: 032] profile::prometheus::alerts: reduce the noise for the Druid alarms [puppet] - 10https://gerrit.wikimedia.org/r/432613 (owner: 10Elukey) [17:56:53] 10Operations, 10SRE-Access-Requests: Give Seddon access to the analytics cluster - https://phabricator.wikimedia.org/T194445#4200915 (10Nuria) >get near real-time page views dat So we are clear data is about couple hours delayed, there is no near-real time data on stats machines, unless. that is, two/three ho... [17:58:27] (03PS1) 10Ppchelko: Kafka: increase group.initial.rebalance.delay.ms to 10s. [puppet] - 10https://gerrit.wikimedia.org/r/432615 (https://phabricator.wikimedia.org/T189618) [18:21:50] !log change 'Advertise this list when people ask what lists are on this machine' to no for cloud-admin-l and cloud-admin-feed [18:21:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:25:30] 10Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864#4200959 (10Reedy) [18:27:56] 10Operations, 10ops-eqiad, 10netops, 10Patch-For-Review: Rack/cable/configure asw2-c-eqiad switch stack - https://phabricator.wikimedia.org/T187962#4200965 (10chasemp) >>! In T187962#4198886, @Andrew wrote: > I'm flying on the 29th. If Chase wants to manage these things without me that's fine with me thou... [18:30:49] (03PS1) 10Subramanya Sastry: Enable RemexHtml on wikis with < 100 ns0 errors in high priority cats [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432621 (https://phabricator.wikimedia.org/T193685) [18:32:19] (03PS2) 10Ppchelko: Kafka: increase group.initial.rebalance.delay.ms to 10s. [puppet] - 10https://gerrit.wikimedia.org/r/432615 (https://phabricator.wikimedia.org/T189618) [18:34:44] (03CR) 10Ppchelko: "Puppet compiler: https://puppet-compiler.wmflabs.org/compiler02/11191/" [puppet] - 10https://gerrit.wikimedia.org/r/432615 (https://phabricator.wikimedia.org/T189618) (owner: 10Ppchelko) [18:55:07] (03PS5) 10Herron: lists: rate limit HTTP subscriptions and reject IPs on blocklists [puppet] - 10https://gerrit.wikimedia.org/r/432168 (https://phabricator.wikimedia.org/T194032) [19:03:46] 10Operations, 10Wikimedia-Mailing-lists: Create wikibaseug mailing list - https://phabricator.wikimedia.org/T189674#4049542 (10Dzahn) You have successfully created the mailing list wikibaseug and notification has been sent to the list owner laura@fanhistory.com. You can now: [[ https://lists.wikimedia.org/mai... [19:06:49] (03CR) 10Herron: "> Thanks for working on this (although the T194032 spammer may or may" [puppet] - 10https://gerrit.wikimedia.org/r/432168 (https://phabricator.wikimedia.org/T194032) (owner: 10Herron) [19:11:05] 10Operations, 10Wikimedia-Mailing-lists: Create wikibaseug mailing list - https://phabricator.wikimedia.org/T189674#4201239 (10Dzahn) 05Open>03Resolved a:03Dzahn @LauraHale The list has been created. I changed nothing from default settings besides setting the description to "Wikibase Community User Group... [19:15:21] (03CR) 10Herron: lists: rate limit HTTP subscriptions and reject IPs on blocklists (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/432168 (https://phabricator.wikimedia.org/T194032) (owner: 10Herron) [19:26:40] !log disable puppet and temp block a few IPs I believe are bad actors hammering mailman [19:26:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:26:48] !log on fermium [19:26:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:27:31] (03PS3) 10Merlijn van Deen: Do not connect to SQL server for a dry run [puppet] - 10https://gerrit.wikimedia.org/r/432532 [19:29:02] (03PS4) 10Merlijn van Deen: Do not connect to SQL server for a dry run [puppet] - 10https://gerrit.wikimedia.org/r/432532 [19:29:28] 10Operations, 10ops-codfw, 10DC-Ops: rigel.frack.codfw.wmnet (fundraising codfw bastion) will not boot after a power cycle - https://phabricator.wikimedia.org/T193891#4201303 (10RobH) [19:42:46] 10Operations, 10Graphite, 10Patch-For-Review, 10Performance-Team (Radar): Certain graphite data directories should be backed up - https://phabricator.wikimedia.org/T194418#4201346 (10Imarlier) [19:44:24] (03CR) 10Urbanecm: [C: 04-1] "I don't think sending wrong error code is a good idea." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/432168 (https://phabricator.wikimedia.org/T194032) (owner: 10Herron) [20:27:31] !log temp changes on fermium for T194032 [20:27:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:41:04] (03PS6) 10Herron: lists: rate limit HTTP subscriptions [puppet] - 10https://gerrit.wikimedia.org/r/432168 (https://phabricator.wikimedia.org/T194032) [20:46:57] (03CR) 10Herron: [C: 032] "Pulled out the blocklist portion for this change. Will create a follow-up patch to keep working on that." [puppet] - 10https://gerrit.wikimedia.org/r/432168 (https://phabricator.wikimedia.org/T194032) (owner: 10Herron) [20:47:02] (03PS7) 10Herron: lists: rate limit HTTP subscriptions [puppet] - 10https://gerrit.wikimedia.org/r/432168 (https://phabricator.wikimedia.org/T194032) [20:59:22] !log deployed rate limiting for POST requests to mailman list subscription URIs https://gerrit.wikimedia.org/r/432168 T194032 [20:59:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:01:33] (03PS1) 10Pnorman: Set hieradata variables for maps tm2source [puppet] - 10https://gerrit.wikimedia.org/r/432694 (https://phabricator.wikimedia.org/T194106) [21:22:18] (03PS1) 10Herron: lists: temporarily change subscribe ratelimit to 1/60m [puppet] - 10https://gerrit.wikimedia.org/r/432695 (https://phabricator.wikimedia.org/T194032) [21:24:17] (03CR) 10Herron: [C: 032] lists: temporarily change subscribe ratelimit to 1/60m [puppet] - 10https://gerrit.wikimedia.org/r/432695 (https://phabricator.wikimedia.org/T194032) (owner: 10Herron) [21:28:27] !log updated lists rate limit to 1 subscribe per 60 minutes as a temporary measure until problem requests slow down T194032 [21:28:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:37:52] (03PS1) 10Merlijn van Deen: labs/db: create basic integration test for maintain-meta_p [puppet] - 10https://gerrit.wikimedia.org/r/432698 [21:38:35] (03CR) 10jerkins-bot: [V: 04-1] labs/db: create basic integration test for maintain-meta_p [puppet] - 10https://gerrit.wikimedia.org/r/432698 (owner: 10Merlijn van Deen) [22:01:42] 10Operations, 10Wikimedia-Mailing-lists: mailing list request for USJP Wiki Club - https://phabricator.wikimedia.org/T194010#4201674 (10Dzahn) [22:01:56] 10Operations, 10Wikimedia-Mailing-lists: Request to create a mailing list for Wikimedia Niger Delta - https://phabricator.wikimedia.org/T193342#4201676 (10Dzahn) [22:04:29] 10Operations, 10Wikimedia-Mailing-lists: mailing list request for USJP Wiki Club - https://phabricator.wikimedia.org/T194010#4185747 (10Dzahn) You have successfully created the mailing list usjp-wikiclub and notification has been sent to the list owner mdabir158@gmail.com. You can now: [[ https://lists.wikime... [22:04:47] 10Operations, 10Wikimedia-Mailing-lists: mailing list request for USJP Wiki Club - https://phabricator.wikimedia.org/T194010#4201679 (10Dzahn) 05Open>03Resolved a:03Dzahn [22:08:21] 10Operations, 10Wikimedia-Mailing-lists: Request to create a mailing list for Wikimedia Niger Delta - https://phabricator.wikimedia.org/T193342#4201703 (10Dzahn) You have successfully created the mailing list wikimedia-nd and notification has been sent to the list owner anthony.mcgreat@gmail.com. You can now:... [22:08:25] 10Operations, 10Wikimedia-Mailing-lists: Request to create a mailing list for Wikimedia Niger Delta - https://phabricator.wikimedia.org/T193342#4201704 (10Dzahn) 05Open>03Resolved a:03Dzahn [22:17:24] 10Operations, 10Graphite, 10Patch-For-Review, 10Performance-Team (Radar): Certain graphite data directories should be backed up - https://phabricator.wikimedia.org/T194418#4201709 (10Dzahn) I went to confirm on Bacula that the backups exist: - ssh helium.eqiad.wmnet - sudo bconsole - restore - 5: Se... [22:21:18] 10Operations, 10Wikimedia-Mailing-lists: Delete "ltakb-admins" mailing list - https://phabricator.wikimedia.org/T194461#4201720 (10Dzahn) [22:21:29] 10Operations, 10Wikimedia-Mailing-lists: Delete "ltakb-admins" mailing list - https://phabricator.wikimedia.org/T194461#4199469 (10Dzahn) 05Open>03Resolved a:03Dzahn ``` [fermium:~] $ sudo /usr/lib/mailman/bin/rmlist -a ltakb-admins Removing list info Removing private archives Removing private archives R... [22:23:31] (03PS1) 10Andrew Bogott: wikitech: Don't load OpenStackManager [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432702 (https://phabricator.wikimedia.org/T161553) [22:24:29] 10Operations, 10Wikimedia-Mailing-lists: Reset admin password for Wikipedia-FR-Wikimag mailinglist - https://phabricator.wikimedia.org/T194466#4199570 (10Dzahn) docs: https://wikitech.wikimedia.org/wiki/Mailman#Reset_the_admin_password_of_a_list --- [fermium:~] $ sudo /var/lib/mailman/bin/change_pw -l Wikipe... [22:24:32] 10Operations, 10Wikimedia-Mailing-lists: Reset admin password for Wikipedia-FR-Wikimag mailinglist - https://phabricator.wikimedia.org/T194466#4201737 (10Dzahn) [22:24:43] 10Operations, 10Wikimedia-Mailing-lists: Reset admin password for Wikipedia-FR-Wikimag mailinglist - https://phabricator.wikimedia.org/T194466#4199570 (10Dzahn) 05Open>03Resolved a:03Dzahn [22:24:44] (03CR) 10jerkins-bot: [V: 04-1] wikitech: Don't load OpenStackManager [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432702 (https://phabricator.wikimedia.org/T161553) (owner: 10Andrew Bogott) [22:27:17] (03PS4) 10Andrew Bogott: keystonehooks: Update our create_project monkeypatch to match Mitaka upstream [puppet] - 10https://gerrit.wikimedia.org/r/432040 [22:27:19] (03PS1) 10Andrew Bogott: wikitech: remove OpenStackManager private settings [puppet] - 10https://gerrit.wikimedia.org/r/432703 (https://phabricator.wikimedia.org/T161553) [22:27:57] (03CR) 10jerkins-bot: [V: 04-1] wikitech: remove OpenStackManager private settings [puppet] - 10https://gerrit.wikimedia.org/r/432703 (https://phabricator.wikimedia.org/T161553) (owner: 10Andrew Bogott) [22:28:28] (03PS2) 10Andrew Bogott: wikitech: Don't load OpenStackManager [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432702 (https://phabricator.wikimedia.org/T161553) [22:29:18] (03PS2) 10Andrew Bogott: wikitech: remove OpenStackManager private settings [puppet] - 10https://gerrit.wikimedia.org/r/432703 (https://phabricator.wikimedia.org/T161553) [22:29:40] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1973 bytes in 0.094 second response time [22:29:57] (03CR) 10jerkins-bot: [V: 04-1] wikitech: remove OpenStackManager private settings [puppet] - 10https://gerrit.wikimedia.org/r/432703 (https://phabricator.wikimedia.org/T161553) (owner: 10Andrew Bogott) [22:31:04] (03PS3) 10Andrew Bogott: wikitech: remove OpenStackManager private settings [puppet] - 10https://gerrit.wikimedia.org/r/432703 (https://phabricator.wikimedia.org/T161553) [22:33:07] 10Operations, 10SRE-Access-Requests: Requesting access to stat1004, stat1005, stat1006 for mneisler - https://phabricator.wikimedia.org/T184838#4201750 (10mpopov) [22:36:47] 10Operations, 10Wikimedia-Mailing-lists: Start a new email list called 'Collaboration-Team' (and discontinue the old 'E2' mailing list) - https://phabricator.wikimedia.org/T186824#4201760 (10Dzahn) [22:40:10] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1976 bytes in 0.116 second response time [22:41:10] 10Operations, 10Wikimedia-Mailing-lists: Start a new email list called 'Collaboration-Team' (and discontinue the old 'E2' mailing list) - https://phabricator.wikimedia.org/T186824#3956487 (10Dzahn) You have successfully created the mailing list collaboration-team and notification has been sent to the list owne... [22:41:25] 10Operations, 10Wikimedia-Mailing-lists: Start a new email list called 'Collaboration-Team' (and discontinue the old 'E2' mailing list) - https://phabricator.wikimedia.org/T186824#4201763 (10Dzahn) 05Open>03Resolved a:03Dzahn [22:49:39] 10Operations, 10SRE-Access-Requests: Access to usergroups for Marshall Miller - https://phabricator.wikimedia.org/T194550#4201768 (10MMiller_WMF) [22:52:24] 10Operations, 10Wikimedia-Mailing-lists: Shut down and delete the devnations-l mailing list - https://phabricator.wikimedia.org/T194537#4201783 (10Dzahn) [22:52:47] 10Operations, 10Toolforge-standards-committee, 10Wikimedia-Mailing-lists: Rename (recreate) mailing list for Toolforge-standards-committee - https://phabricator.wikimedia.org/T172624#4201807 (10Dzahn) [22:53:10] 10Operations, 10Wikimedia-Mailing-lists: Shut down and delete the devnations-l mailing list - https://phabricator.wikimedia.org/T194537#4201414 (10Dzahn) ``` [fermium:~] $ sudo /var/lib/mailman/bin/rmlist -a devnations-l Removing list info Removing private archives Removing private archives Removing public arc... [22:53:26] 10Operations, 10Wikimedia-Mailing-lists: Shut down and delete the devnations-l mailing list - https://phabricator.wikimedia.org/T194537#4201811 (10Dzahn) 05Open>03Resolved a:03Dzahn [22:54:45] (03CR) 10Pnorman: [C: 04-1] "This would need some changes in https://github.com/wikimedia/puppet/blob/production/modules/tilerator/manifests/init.pp, and there's discu" [puppet] - 10https://gerrit.wikimedia.org/r/432694 (https://phabricator.wikimedia.org/T194106) (owner: 10Pnorman) [22:55:21] (03Abandoned) 10Pnorman: Set hieradata variables for maps tm2source [puppet] - 10https://gerrit.wikimedia.org/r/432694 (https://phabricator.wikimedia.org/T194106) (owner: 10Pnorman) [22:56:13] 10Operations, 10Mail, 10Wikimedia-Mailing-lists: Reach out to Google about @yahoo.com emails not reaching gmail inboxes (when sent to mailing lists) - https://phabricator.wikimedia.org/T146841#4201816 (10Dzahn) [22:57:22] 10Operations, 10Reading-Infrastructure-Team-Backlog, 10SRE-Access-Requests, 10Patch-For-Review: Add Michael Holloway (Reading Infrastructure) to maps admin groups - https://phabricator.wikimedia.org/T194404#4201818 (10Dzahn) @Mholloway What would you say, who is closest to "project lead of the cluster" in... [22:59:14] 10Operations, 10SRE-Access-Requests: Access to usergroups for Marshall Miller - https://phabricator.wikimedia.org/T194550#4201819 (10Dzahn) p:05Triage>03Normal [23:01:47] 10Operations, 10Analytics, 10SRE-Access-Requests: Access to usergroups for Marshall Miller - https://phabricator.wikimedia.org/T194550#4201768 (10Dzahn) [23:02:38] (03CR) 10Jforrester: "Wonderful to see!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432702 (https://phabricator.wikimedia.org/T161553) (owner: 10Andrew Bogott) [23:14:57] (03CR) 10Krinkle: [C: 031] Add .gitreview file [debs/python-logstash] - 10https://gerrit.wikimedia.org/r/430306 (owner: 10Gilles) [23:33:05] 10Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864#4201887 (10MarcoAurelio) >>! In T52864#3941723, @Legoktm wrote: > I think we need to lobby/convince/remind @faidon and other roadmap deciders to alloca... [23:49:20] 10Operations, 10Analytics, 10SRE-Access-Requests: Access to usergroups for Marshall Miller - https://phabricator.wikimedia.org/T194550#4201903 (10DannyH) I approve Marshall's access. [23:56:00] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1967 bytes in 0.089 second response time [23:57:46] 10Operations, 10Analytics, 10SRE-Access-Requests: Access to usergroups for Marshall Miller - https://phabricator.wikimedia.org/T194550#4201908 (10Nuria) Approved on my end too.