[00:01:38] (03PS1) 10Dzahn: scap: add data types, lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/623078 [00:09:21] (03PS1) 10Dzahn: tlsproxy::instance: switch from hiera() to lookup(), lint fix [puppet] - 10https://gerrit.wikimedia.org/r/623079 [00:32:40] (03PS1) 10Dzahn: phabricator: remove hiera() lookup from module [puppet] - 10https://gerrit.wikimedia.org/r/623080 [00:33:40] (03CR) 10jerkins-bot: [V: 04-1] phabricator: remove hiera() lookup from module [puppet] - 10https://gerrit.wikimedia.org/r/623080 (owner: 10Dzahn) [00:35:28] (03PS1) 10Dzahn: releases: switch the active server from eqiad to codfw [puppet] - 10https://gerrit.wikimedia.org/r/623081 [00:38:28] (03PS1) 10Dzahn: releases: switch backend from eqiad to codfw [dns] - 10https://gerrit.wikimedia.org/r/623082 [00:40:06] (03PS1) 10Dzahn: aptrepo: switch active server from eqiad to codfw [puppet] - 10https://gerrit.wikimedia.org/r/623083 [00:48:52] (03PS1) 10Dzahn: deployment::server: replace hiera() with lookup() [puppet] - 10https://gerrit.wikimedia.org/r/623084 [00:49:53] (03CR) 10jerkins-bot: [V: 04-1] deployment::server: replace hiera() with lookup() [puppet] - 10https://gerrit.wikimedia.org/r/623084 (owner: 10Dzahn) [01:00:24] (03PS1) 10Dzahn: switch deployment_server from eqiad to codfw [puppet] - 10https://gerrit.wikimedia.org/r/623085 [01:01:52] (03PS1) 10Dzahn: deployment::rsync: add data types [puppet] - 10https://gerrit.wikimedia.org/r/623086 [01:02:58] (03PS1) 10Dzahn: planet: switch backend from eqiad to codfw [dns] - 10https://gerrit.wikimedia.org/r/623087 [01:03:50] (03PS1) 10Dzahn: people: switch backend from eqiad to codfw [dns] - 10https://gerrit.wikimedia.org/r/623088 [01:06:30] (03PS1) 10Dzahn: switch mwmaint backend from eqiad to codfw (noc.wikimedia.org) [dns] - 10https://gerrit.wikimedia.org/r/623089 [01:07:25] (03PS1) 10Dzahn: switch webserver_misc_apps from eqiad to codfw [dns] - 10https://gerrit.wikimedia.org/r/623090 [01:08:52] (03PS1) 10Dzahn: rename webserver_misc_apps to miscweb [dns] - 10https://gerrit.wikimedia.org/r/623091 [01:12:49] (03PS1) 10Dzahn: rename webserver_misc_apps to miscweb [puppet] - 10https://gerrit.wikimedia.org/r/623092 [01:14:10] (03CR) 10jerkins-bot: [V: 04-1] rename webserver_misc_apps to miscweb [puppet] - 10https://gerrit.wikimedia.org/r/623092 (owner: 10Dzahn) [01:16:02] (03PS1) 10Dzahn: miscweb: rename misc_apps to miscweb for consistency [puppet] - 10https://gerrit.wikimedia.org/r/623093 [01:17:05] (03CR) 10jerkins-bot: [V: 04-1] miscweb: rename misc_apps to miscweb for consistency [puppet] - 10https://gerrit.wikimedia.org/r/623093 (owner: 10Dzahn) [01:19:18] (03PS2) 10Dzahn: rename webserver_misc_apps to miscweb [puppet] - 10https://gerrit.wikimedia.org/r/623092 [02:12:19] (03PS2) 10Dzahn: deployment::server: replace hiera() with lookup() [puppet] - 10https://gerrit.wikimedia.org/r/623084 [02:16:25] (03PS2) 10Dzahn: phabricator: remove hiera() lookup from module [puppet] - 10https://gerrit.wikimedia.org/r/623080 [02:56:06] (03PS1) 10Keenan Pepper: Add cubic hectometre conversion [mediawiki-config] - 10https://gerrit.wikimedia.org/r/623094 [02:56:08] (03CR) 10Welcome, new contributor!: "Thank you for making your first contribution to Wikimedia! :) To learn how to get your code changes reviewed faster and more likely to get" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/623094 (owner: 10Keenan Pepper) [03:06:00] (03PS1) 10Andrew Bogott: mwopenstackclients: don't pass in 'timeout' when making glance client [puppet] - 10https://gerrit.wikimedia.org/r/623095 [03:18:52] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [03:22:36] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [03:27:15] (03PS2) 10Andrew Bogott: mwopenstackclients: fix glance clients [puppet] - 10https://gerrit.wikimedia.org/r/623095 [03:51:36] (03CR) 10Andrew Bogott: [C: 03+2] mwopenstackclients: fix glance clients [puppet] - 10https://gerrit.wikimedia.org/r/623095 (owner: 10Andrew Bogott) [04:19:28] (03PS1) 10Andrew Bogott: wmcs-ceph-migrate: include a flavor change post-migration [puppet] - 10https://gerrit.wikimedia.org/r/623097 [04:19:53] (03CR) 10jerkins-bot: [V: 04-1] wmcs-ceph-migrate: include a flavor change post-migration [puppet] - 10https://gerrit.wikimedia.org/r/623097 (owner: 10Andrew Bogott) [04:25:51] (03PS2) 10Andrew Bogott: wmcs-ceph-migrate: include a flavor change post-migration [puppet] - 10https://gerrit.wikimedia.org/r/623097 [05:57:40] 10Operations, 10LDAP-Access-Requests, 10SRE-Access-Requests: Requesting access to production shell and wmf ldap access for Razzi Abuissa - https://phabricator.wikimedia.org/T261443 (10Dzahn) Hi @razzi Check out https://wikitech.wikimedia.org/wiki/Bastion and https://wikitech.wikimedia.org/wiki/Production_acc... [06:12:42] RECOVERY - Router interfaces on cr1-codfw is OK: OK: host 208.80.153.192, interfaces up: 134, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [06:13:53] 10Operations, 10InternetArchiveBot, 10Traffic: Support TLSv1.3 in IABot - https://phabricator.wikimedia.org/T251414 (10Reedy) As long as you're using PHP 7.3... https://www.php.net/manual/en/migration73.constants.php You should be able to do something like `curl_setopt($this->curlHandle, CURLOPT_SSLVERSION... [06:14:14] RECOVERY - Router interfaces on cr4-ulsfo is OK: OK: host 198.35.26.193, interfaces up: 77, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [06:29:44] PROBLEM - ores on ores1009 is CRITICAL: connect to address 10.64.48.28 and port 8081: Connection refused https://wikitech.wikimedia.org/wiki/Services/Monitoring/ores [06:50:04] RECOVERY - ores on ores1009 is OK: HTTP OK: HTTP/1.0 200 OK - 6397 bytes in 0.023 second response time https://wikitech.wikimedia.org/wiki/Services/Monitoring/ores [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200829T0700) [10:45:12] PROBLEM - Host mr1-eqiad.oob is DOWN: PING CRITICAL - Packet loss = 100% [10:45:32] PROBLEM - Host mr1-eqiad.oob IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [10:51:26] RECOVERY - Host mr1-eqiad.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 2.78 ms [10:57:16] PROBLEM - Router interfaces on mr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.199, interfaces up: 35, down: 1, dormant: 0, excluded: 1, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [10:59:18] PROBLEM - Host mr1-eqiad.oob IPv6 is DOWN: CRITICAL - Destination Unreachable (2607:f6f0:205::153) [11:00:58] RECOVERY - Router interfaces on mr1-eqiad is OK: OK: host 208.80.154.199, interfaces up: 37, down: 0, dormant: 0, excluded: 1, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [11:03:10] RECOVERY - Host mr1-eqiad.oob is UP: PING OK - Packet loss = 0%, RTA = 0.75 ms [11:05:12] RECOVERY - Host mr1-eqiad.oob IPv6 is UP: PING OK - Packet loss = 0%, RTA = 2.42 ms [11:09:49] 10Operations, 10InternetArchiveBot, 10Traffic: Support TLSv1.3 in IABot - https://phabricator.wikimedia.org/T251414 (10Cyberpower678) Then it’s not happening as Toolforge runs on 7.2 [11:23:42] 10Operations, 10InternetArchiveBot, 10Traffic: Support TLSv1.3 in IABot - https://phabricator.wikimedia.org/T251414 (10Reedy) 05Declined→03Open Depends how/where you run it in Toolforge.. https://wikitech-static.wikimedia.org/wiki/Help:Toolforge/Kubernetes#PHP You can easily run it with a PHP 7.3 conta... [11:50:47] 10Puppet: Puppet resource for creating a postgresql database - https://phabricator.wikimedia.org/T96054 (10Aklapper) [11:50:51] 10Blocked-on-Operations, 10Puppet, 10Product-Infrastructure-Team-Backlog, 10Sentry, and 2 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956 (10Aklapper) [12:56:58] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [13:00:40] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [13:06:14] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [13:09:56] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [14:08:55] (03PS1) 10Ashot1997: Enable Signature button on Wikiproject for hywiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/623109 (https://phabricator.wikimedia.org/T261550) [14:33:40] PROBLEM - IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 is CRITICAL: CRITICAL - failed 70 probes of 559 (alerts on 65) - https://atlas.ripe.net/measurements/1790947/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [14:39:32] RECOVERY - IPv6 ping to eqiad on ripe-atlas-eqiad IPv6 is OK: OK - failed 47 probes of 559 (alerts on 65) - https://atlas.ripe.net/measurements/1790947/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23Atlas_alerts https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas [16:13:54] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=swagger_check_cxserver_cluster_codfw site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [16:15:46] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [16:38:50] (03PS3) 10Andrew Bogott: wmcs-ceph-migrate: include a flavor change post-migration [puppet] - 10https://gerrit.wikimedia.org/r/623097 [17:15:59] (03PS1) 10Mdaniels5757: Allow bureaucrats to remove sysop permissions on Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/623119 (https://phabricator.wikimedia.org/T261481) [17:33:30] 10Operations: Fix "Blog" link on noc.wikimedia.org - https://phabricator.wikimedia.org/T259978 (10Aklapper) Reverted in https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/619377 and I'm currently too lazy to set up the patch again. [17:45:37] !log start of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T261451) [17:45:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:45:41] T261451: Add Wikidata support to jawikivoyage - https://phabricator.wikimedia.org/T261451 [18:05:50] !log end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T261451) [18:05:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:05:54] T261451: Add Wikidata support to jawikivoyage - https://phabricator.wikimedia.org/T261451 [18:33:51] 10Operations, 10serviceops, 10Wikimedia-production-error: PHP7 corruptions (Call on wrong object, Call to undefined method, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) [18:34:17] 10Operations, 10serviceops, 10Wikimedia-production-error: PHP7 corruptions (Call on wrong object, Call to undefined method, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) [18:38:14] 10Operations, 10serviceops, 10Wikimedia-production-error: PHP7 corruptions (Call on wrong object, Call to undefined method, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) Another corruption: ` Uncaught Psr\Log\InvalidArgumentException: Level "400" is not defined, use one of: 0, 1, 2 in /srv/m... [21:11:14] (03CR) 10DannyS712: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/623119 (https://phabricator.wikimedia.org/T261481) (owner: 10Mdaniels5757) [21:19:04] (03PS1) 10Gerrit maintenance bot: Order entries by alphabetical order [dns] - 10https://gerrit.wikimedia.org/r/623140 [21:22:17] (03PS1) 10Gerrit maintenance bot: Order entries by alphabetical order [dns] - 10https://gerrit.wikimedia.org/r/623143 [21:27:42] (03CR) 10Ladsgroup: "Hello, It would be great if you take a look t this. It would make adding new languages easier (T253439)" [dns] - 10https://gerrit.wikimedia.org/r/623143 (owner: 10Gerrit maintenance bot) [21:29:51] (03PS2) 10Ladsgroup: Order entries by alphabetical order [dns] - 10https://gerrit.wikimedia.org/r/623143 (https://phabricator.wikimedia.org/T253439) (owner: 10Gerrit maintenance bot) [21:45:54] PROBLEM - Cxserver LVS eqiad on cxserver.svc.eqiad.wmnet is CRITICAL: /v2/translate/{from}/{to}{/provider} (Machine translate an HTML fragment using TestClient, adapt the links to target language wiki.) timed out before a response was received https://wikitech.wikimedia.org/wiki/CX [21:47:40] RECOVERY - Cxserver LVS eqiad on cxserver.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/CX [21:49:03] (03Abandoned) 10Gerrit maintenance bot: Order entries by alphabetical order [dns] - 10https://gerrit.wikimedia.org/r/623140 (owner: 10Gerrit maintenance bot) [23:33:00] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=swagger_check_cxserver_cluster_codfw site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [23:38:34] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets