[00:01:09] RECOVERY - Host cp3065 is UP: PING OK - Packet loss = 0%, RTA = 83.41 ms [00:05:49] !log jiji@cumin1001 conftool action : set/pooled=yes; selector: name=cp3065.esams.wmnet [00:05:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:06:11] !log pool cp3065 T238305 [00:06:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:06:14] T238305: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 [00:09:52] (03CR) 10DannyS712: Remove "Create a book" link on enwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561403 (https://phabricator.wikimedia.org/T241683) (owner: 10DannyS712) [01:25:47] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=atlas_exporter site=eqiad https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [01:27:31] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [01:55:23] (03PS1) 10Milimetric: [WIP] Migrate pagecounts-ez to hadoop [puppet] - 10https://gerrit.wikimedia.org/r/562079 (https://phabricator.wikimedia.org/T192474) [03:43:25] (03PS1) 10Ladsgroup: mediawiki: Stop rebuildItemTermIndex job [puppet] - 10https://gerrit.wikimedia.org/r/562080 (https://phabricator.wikimedia.org/T219123) [03:53:50] !log ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --batch-size=100 --sleep=2 --file=/tmp/100k.lines (T219301) [03:53:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:53:55] T219301: Migrate to and read from new store for property terms - https://phabricator.wikimedia.org/T219301 [03:57:17] (03PS1) 10Ladsgroup: Set read new for item term store up to Q100K [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562081 (https://phabricator.wikimedia.org/T219123) [04:45:26] (03PS1) 10Ladsgroup: Don’t check constraints on P6685 statements [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562083 (https://phabricator.wikimedia.org/T227865) [04:56:36] 10Operations, 10VisualEditor: VisualEditor broken on Wikitech - https://phabricator.wikimedia.org/T241961 (10ayounsi) p:05Triage→03Normal [05:03:11] 10Operations, 10Design-Research, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Dendelele) Hello. My name is Bethany and I work in the WMF Design Strategy department. I am working with OIT to make my account compliant so that I can make such requests... [05:04:19] 10Operations, 10netops: fastnetmon misreports attack type and protocol - https://phabricator.wikimedia.org/T241374 (10ayounsi) a:05ayounsi→03None Known issue: https://github.com/pavel-odintsov/fastnetmon/issues/787#issuecomment-570740316 I don't see it being solved anytime soon. Also added a "limitations"... [05:11:59] 10Operations, 10netops: Upgrade LibreNMS to 1.59 - https://phabricator.wikimedia.org/T241962 (10ayounsi) p:05Triage→03Low [05:22:51] (03CR) 10Ayounsi: "Planned and inventory can be racked or unracked. I'm wondering if there is a need to do the distinction." [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/561917 (https://phabricator.wikimedia.org/T233183) (owner: 10Volans) [05:23:35] (03CR) 10Ayounsi: [C: 03+1] Fix network devices management records [dns] - 10https://gerrit.wikimedia.org/r/561857 (https://phabricator.wikimedia.org/T239597) (owner: 10Volans) [07:08:27] 10Operations: Use check_dns_query for anycast DNS checks - https://phabricator.wikimedia.org/T241965 (10ayounsi) p:05Triage→03Low [07:37:26] !log ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemTerms.php --wiki=wikidatawiki --batch-size=100 --sleep=2 --file=/tmp/1mio.lines (T219301) [07:37:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:37:30] T219301: Migrate to and read from new store for property terms - https://phabricator.wikimedia.org/T219301 [07:38:11] on screen, in case it misbehaves [07:59:36] (03PS2) 10Ema: cache: hiera setting to apply performance tweaks [puppet] - 10https://gerrit.wikimedia.org/r/561863 [08:03:21] (03CR) 10Ema: [C: 03+2] cache: hiera setting to apply performance tweaks [puppet] - 10https://gerrit.wikimedia.org/r/561863 (owner: 10Ema) [08:07:03] (03PS2) 10Muehlenhoff: Extend the docs for selecting packages to install from a component [puppet] - 10https://gerrit.wikimedia.org/r/561875 [08:11:53] (03CR) 10Muehlenhoff: [C: 03+2] Extend the docs for selecting packages to install from a component [puppet] - 10https://gerrit.wikimedia.org/r/561875 (owner: 10Muehlenhoff) [08:54:36] 10Operations, 10Traffic, 10Cloud-VPS (Debian Jessie Deprecation): "traffic" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236561 (10ema) [09:02:40] (03PS2) 10Ladsgroup: mediawiki: Stop rebuildItemTermIndex job [puppet] - 10https://gerrit.wikimedia.org/r/562080 (https://phabricator.wikimedia.org/T219123) [09:02:54] 10Operations, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor broken on Wikitech - https://phabricator.wikimedia.org/T241961 (10Peachey88) [09:03:38] 10Operations, 10observability: Use check_dns_query for anycast DNS checks - https://phabricator.wikimedia.org/T241965 (10Peachey88) [09:03:48] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: NDA for Superset Request from WMDE Employee - Kris Litson - https://phabricator.wikimedia.org/T241722 (10Franziska_Heine) Approved. [09:04:35] (03CR) 10Gehel: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/561866 (owner: 10Muehlenhoff) [09:04:50] (03CR) 10Ema: [C: 03+2] mediawiki: Stop rebuildItemTermIndex job [puppet] - 10https://gerrit.wikimedia.org/r/562080 (https://phabricator.wikimedia.org/T219123) (owner: 10Ladsgroup) [09:08:17] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10Franziska_Heine) Approved. [09:15:01] (03CR) 10Muehlenhoff: [C: 03+2] profile::mediawiki::videoscaler: Switch to apt::package_from_component [puppet] - 10https://gerrit.wikimedia.org/r/561824 (owner: 10Muehlenhoff) [09:17:55] (03CR) 10Ema: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1002/20195/" [puppet] - 10https://gerrit.wikimedia.org/r/561878 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [09:28:59] (03PS2) 10Ema: cache: make backend_services default to 'ats-be' [puppet] - 10https://gerrit.wikimedia.org/r/561881 (https://phabricator.wikimedia.org/T241239) [09:29:17] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=varnish-text site=ulsfo https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [09:29:40] looking, that's likely due to the rename of role text_ats to text ^ [09:32:16] !log reimaging mw2282 to validate correctness of apt::package_from_component for fresh installs [09:32:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:40:47] PROBLEM - Varnish traffic drop between 30min ago and now at eqsin on icinga1001 is CRITICAL: 15.83 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [09:51:20] !log jmm@cumin2001 START - Cookbook sre.hosts.downtime [09:51:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:53:29] !log jmm@cumin2001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [09:53:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:56:50] (03PS1) 10Ema: prometheus: re-add varnish-text_mtail jobs [puppet] - 10https://gerrit.wikimedia.org/r/562225 (https://phabricator.wikimedia.org/T241239) [10:03:41] (03PS1) 10Muehlenhoff: librenms: Switch to apt::package_from_component [puppet] - 10https://gerrit.wikimedia.org/r/562226 [10:03:44] (03CR) 10Ema: [C: 03+2] prometheus: re-add varnish-text_mtail jobs [puppet] - 10https://gerrit.wikimedia.org/r/562225 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [10:04:35] (03CR) 10jerkins-bot: [V: 04-1] librenms: Switch to apt::package_from_component [puppet] - 10https://gerrit.wikimedia.org/r/562226 (owner: 10Muehlenhoff) [10:07:51] RECOVERY - Varnish traffic drop between 30min ago and now at eqsin on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [10:09:31] PROBLEM - Postgres Replication Lag on maps1003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 33459512 and 7 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:11:01] RECOVERY - Postgres Replication Lag on maps1003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 0 and 13 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [10:11:57] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [10:15:28] (03PS2) 10Muehlenhoff: librenms: Switch to apt::package_from_component [puppet] - 10https://gerrit.wikimedia.org/r/562226 [10:16:29] (03CR) 10jerkins-bot: [V: 04-1] librenms: Switch to apt::package_from_component [puppet] - 10https://gerrit.wikimedia.org/r/562226 (owner: 10Muehlenhoff) [10:19:16] (03PS3) 10Ema: cache: make backend_services default to 'ats-be' [puppet] - 10https://gerrit.wikimedia.org/r/561881 (https://phabricator.wikimedia.org/T241239) [10:22:47] (03CR) 10Ema: [C: 03+2] cache: make backend_services default to 'ats-be' [puppet] - 10https://gerrit.wikimedia.org/r/561881 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [10:23:55] (03CR) 10Volans: "> Patch Set 1:" [software/netbox-extras] - 10https://gerrit.wikimedia.org/r/561917 (https://phabricator.wikimedia.org/T233183) (owner: 10Volans) [10:39:26] !log installing libbsd security updates [10:39:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:39:36] (03PS3) 10Muehlenhoff: librenms: Switch to apt::package_from_component [puppet] - 10https://gerrit.wikimedia.org/r/562226 [10:41:37] (03PS1) 10Muehlenhoff: Add library hint for libbsd [puppet] - 10https://gerrit.wikimedia.org/r/562230 [10:42:54] (03PS1) 10Ema: cache: remove profile::cache::varnish::backend and cron_restart [puppet] - 10https://gerrit.wikimedia.org/r/562231 (https://phabricator.wikimedia.org/T241239) [10:46:50] (03PS2) 10Ema: cache: remove profile::cache::varnish::backend and cron_restart [puppet] - 10https://gerrit.wikimedia.org/r/562231 (https://phabricator.wikimedia.org/T241239) [10:46:52] (03PS1) 10Ema: cache: remove varnish-backend-restart [puppet] - 10https://gerrit.wikimedia.org/r/562232 (https://phabricator.wikimedia.org/T241239) [10:53:42] (03CR) 10Muehlenhoff: [C: 03+2] Add library hint for libbsd [puppet] - 10https://gerrit.wikimedia.org/r/562230 (owner: 10Muehlenhoff) [10:56:50] !log urbanecm@deploy1001 Synchronized private/PrivateSettings.php: Revert T227416 mitigations (duration: 01m 05s) [10:56:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:11:10] PROBLEM - Unmerged changes on repository puppet on labtestpuppetmaster2001 is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet, ref HEAD..origin/production). https://wikitech.wikimedia.org/wiki/Monitoring/unmerged_changes [11:18:00] (03PS1) 10Muehlenhoff: Remove conditional application of apt pin [puppet] - 10https://gerrit.wikimedia.org/r/562234 [11:21:45] 10Operations, 10vm-requests: eqiad/codfw: 2 VM request for URL downloaders - https://phabricator.wikimedia.org/T241979 (10MoritzMuehlenhoff) [11:22:00] (03PS2) 10Muehlenhoff: Remove conditional application of apt pin [puppet] - 10https://gerrit.wikimedia.org/r/562234 [11:25:08] 10Operations, 10Dumps-Generation, 10SDC General, 10Wikidata: Capacity planning for Commons Structured Data - https://phabricator.wikimedia.org/T226093 (10Cparle) [11:30:05] jan_drewniak: It is that lovely time of the day again! You are hereby commanded to deploy Wikimedia Portals Update. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200106T1130). [11:31:18] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562236 (https://phabricator.wikimedia.org/T128546) [11:33:31] (03CR) 10Jdrewniak: [C: 03+2] Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562236 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [11:34:17] (03CR) 10Muehlenhoff: [C: 03+2] Remove conditional application of apt pin [puppet] - 10https://gerrit.wikimedia.org/r/562234 (owner: 10Muehlenhoff) [11:34:33] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562236 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [11:36:14] RECOVERY - Unmerged changes on repository puppet on labtestpuppetmaster2001 is OK: No changes to merge. https://wikitech.wikimedia.org/wiki/Monitoring/unmerged_changes [11:39:28] (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] Don’t check constraints on P6685 statements (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562083 (https://phabricator.wikimedia.org/T227865) (owner: 10Ladsgroup) [11:41:19] !log jdrewniak@deploy1001 Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:562236| Bumping portals to master (T128546)]] (duration: 00m 55s) [11:41:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:41:23] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [11:42:15] !log jdrewniak@deploy1001 Synchronized portals: Wikimedia Portals Update: [[gerrit:562236| Bumping portals to master (T128546)]] (duration: 00m 55s) [11:42:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:00:04] Amir1, Lucas_WMDE, awight, and Urbanecm: #bothumor My software never has bugs. It just develops random features. Rise for European Mid-day SWAT(Max 6 patches). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200106T1200). [12:00:04] tassu and Amir1: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [12:00:10] o/ [12:00:14] * Urbanecm around [12:00:29] Amir1: since you have most of patches in this window, want to lead this SWAT? [12:00:38] sure [12:00:44] :) [12:00:48] I can do the swat today [12:01:37] (03CR) 10Ladsgroup: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561286 (https://phabricator.wikimedia.org/T241304) (owner: 10Majavah) [12:01:50] PROBLEM - mediawiki-installation DSH group on mw2282 is CRITICAL: Host mw2282 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [12:02:37] (03Merged) 10jenkins-bot: Enable wgCiteResponsiveReferences on cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561286 (https://phabricator.wikimedia.org/T241304) (owner: 10Majavah) [12:04:07] tassu: it's live in mwdebug1001, please check [12:04:12] sure [12:04:49] it's working [12:06:23] cool deploying [12:07:03] !log ladsgroup@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [[gerrit:561286|Enable wgCiteResponsiveReferences on cswiki (T241304)]] (duration: 00m 56s) [12:07:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:07:05] T241304: Make references responsive by default at the Czech Wikipedia - https://phabricator.wikimedia.org/T241304 [12:07:10] (03CR) 10Ladsgroup: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562081 (https://phabricator.wikimedia.org/T219123) (owner: 10Ladsgroup) [12:11:43] (03PS2) 10Ladsgroup: Set read new for item term store up to Q100K [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562081 (https://phabricator.wikimedia.org/T219123) [12:11:51] (03CR) 10Ladsgroup: [C: 03+2] Set read new for item term store up to Q100K [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562081 (https://phabricator.wikimedia.org/T219123) (owner: 10Ladsgroup) [12:12:52] (03Merged) 10jenkins-bot: Set read new for item term store up to Q100K [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562081 (https://phabricator.wikimedia.org/T219123) (owner: 10Ladsgroup) [12:15:00] !log ladsgroup@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:562081|Set read new for item term store up to Q100K (T219123)]] (duration: 00m 55s) [12:15:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:15:05] T219123: Migrate to and read from new store for item terms - https://phabricator.wikimedia.org/T219123 [12:15:08] (03PS2) 10Ladsgroup: Don’t check constraints on P6685 statements [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562083 (https://phabricator.wikimedia.org/T227865) [12:15:16] (03CR) 10Ladsgroup: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562083 (https://phabricator.wikimedia.org/T227865) (owner: 10Ladsgroup) [12:15:40] (03PS1) 10Jakob: termbox: update test service to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/562238 [12:15:40] (03PS1) 10Jakob: termbox: update staging service to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/562239 [12:15:42] (03PS1) 10Jakob: termbox: update codfw service to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/562240 [12:16:13] (03PS1) 10Jakob: termbox: update eqiad service to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/562241 [12:16:15] (03Merged) 10jenkins-bot: Don’t check constraints on P6685 statements [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562083 (https://phabricator.wikimedia.org/T227865) (owner: 10Ladsgroup) [12:18:04] Amir1: I can test the constraints config change if you want [12:18:05] (03PS2) 10Ladsgroup: Revert "Add a bit for forcing LC caching backend in cli mode" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560341 [12:18:21] Lucas_WMDE: yeah, it goes live in a minute [12:18:26] (03PS3) 10Ammarpad: Re-add localized Wikipedia wordmark for szlwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/557584 (https://phabricator.wikimedia.org/T233104) [12:18:41] !log ladsgroup@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:562083|Don’t check constraints on P6685 statements (T227865)]] (duration: 00m 55s) [12:18:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:18:44] (03CR) 10Jakob: "This change is ready for review." [deployment-charts] - 10https://gerrit.wikimedia.org/r/562238 (owner: 10Jakob) [12:18:45] T227865: Don’t check constraints on “Wikidata property example for media” statements - https://phabricator.wikimedia.org/T227865 [12:19:17] uh [12:19:19] without mwdebug? [12:19:37] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] termbox: update test service to latest [deployment-charts] - 10https://gerrit.wikimedia.org/r/562238 (owner: 10Jakob) [12:19:51] Lucas_WMDE: yeah [12:20:12] (03CR) 10Ladsgroup: "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560341 (owner: 10Ladsgroup) [12:20:28] Amir1: missed CR+2 :-) [12:20:38] oops [12:20:47] well, it doesn’t work as far as I can tell [12:20:57] (03CR) 10Ladsgroup: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560341 (owner: 10Ladsgroup) [12:21:04] (03PS6) 10Ammarpad: Add new namespace and aliases for zh.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/558827 (https://phabricator.wikimedia.org/T241023) [12:21:27] Lucas_WMDE: okay, let me check after this one [12:22:07] (03Merged) 10jenkins-bot: Revert "Add a bit for forcing LC caching backend in cli mode" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560341 (owner: 10Ladsgroup) [12:22:22] Lucas_WMDE: ah, I remember the sync cache is behind master one deployment [12:22:31] it means it's not deployed yet [12:22:38] ok [12:22:48] (03PS3) 10Ammarpad: Add sandboxlink for eswikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559576 (https://phabricator.wikimedia.org/T241163) [12:22:55] https://phabricator.wikimedia.org/T236104 [12:23:08] !log jakob@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'test' . [12:23:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:23:35] Amir1: ping me if there's some time left, please [12:23:42] sure [12:25:38] !log ladsgroup@deploy1001 Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:560341|Revert "Add a bit for forcing LC caching backend in cli mode"]] (duration: 00m 54s) [12:25:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:28:05] !log ladsgroup@deploy1001 Synchronized php-1.35.0-wmf.11/maintenance/rebuildLocalisationCache.php: SWAT: [[gerrit:560596|Add option to override storeClass in rebuildLocalisationCache (T105683 T99740)]] (duration: 00m 55s) [12:28:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:28:09] T99740: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 [12:28:09] T105683: Add Scap support for static-array format of LCStore - https://phabricator.wikimedia.org/T105683 [12:29:18] (03CR) 10Ladsgroup: "I think I fixed it why it's not there :/" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559161 (https://phabricator.wikimedia.org/T238154) (owner: 10Ladsgroup) [12:31:00] !log ladsgroup@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:562083|Don’t check constraints on P6685 statements]] Bypassing T236104 (duration: 00m 55s) [12:31:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:31:03] T236104: Cache of wmf-config/InitialiseSettings often 1 step behind - https://phabricator.wikimedia.org/T236104 [12:31:30] (03PS1) 10Muehlenhoff: Pass the package list to the apt::pin [puppet] - 10https://gerrit.wikimedia.org/r/562246 (https://phabricator.wikimedia.org/T230324) [12:32:18] (03CR) 10jerkins-bot: [V: 04-1] Pass the package list to the apt::pin [puppet] - 10https://gerrit.wikimedia.org/r/562246 (https://phabricator.wikimedia.org/T230324) (owner: 10Muehlenhoff) [12:33:11] Urbanecm: the floor is yours [12:33:20] thank you Amir1 [12:34:10] (03CR) 10Urbanecm: [C: 03+2] Add throttle exception for Amical Wikimedia Workshop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561576 (https://phabricator.wikimedia.org/T241705) (owner: 10Ammarpad) [12:35:14] (03PS1) 10Jakob: Revert "Update termbox test service to latest" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562247 [12:35:14] (03Merged) 10jenkins-bot: Add throttle exception for Amical Wikimedia Workshop [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561576 (https://phabricator.wikimedia.org/T241705) (owner: 10Ammarpad) [12:35:15] (03CR) 10jerkins-bot: [V: 04-1] Revert "Update termbox test service to latest" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562247 (owner: 10Jakob) [12:35:49] (03Abandoned) 10Jakob: Revert "Update termbox test service to latest" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562247 (owner: 10Jakob) [12:36:19] (03PS1) 10Jakob: Revert "termbox: update test service to latest" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562248 [12:36:56] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Revert "termbox: update test service to latest" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562248 (owner: 10Jakob) [12:37:10] !log urbanecm@deploy1001 Synchronized wmf-config/throttle.php: SWAT: 0ac5032: Add throttle exception for Amical Wikimedia Workshop (T241705) (duration: 00m 56s) [12:37:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:37:13] T241705: Request: temporary lift of IP cap in cawiki on 2020-01-17 - https://phabricator.wikimedia.org/T241705 [12:37:31] (03CR) 10Urbanecm: [C: 03+2] Add new namespace and aliases for zh.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/558827 (https://phabricator.wikimedia.org/T241023) (owner: 10Ammarpad) [12:37:50] (03PS7) 10Urbanecm: Add new namespace and aliases for zh.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/558827 (https://phabricator.wikimedia.org/T241023) (owner: 10Ammarpad) [12:37:57] (03CR) 10Urbanecm: [C: 03+2] Add new namespace and aliases for zh.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/558827 (https://phabricator.wikimedia.org/T241023) (owner: 10Ammarpad) [12:38:29] (03PS2) 10Urbanecm: Set Author and Author_talk aliases for Autore NS at napwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561359 (https://phabricator.wikimedia.org/T231880) [12:38:45] I go run for a couple of hours, will be back [12:39:01] (03Merged) 10jenkins-bot: Add new namespace and aliases for zh.wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/558827 (https://phabricator.wikimedia.org/T241023) (owner: 10Ammarpad) [12:39:10] !log jakob@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'test' . [12:39:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:39:54] !log urbanecm@deploy1001 sync-file aborted: SWAT: 0ac5032: Add throttle exception for Amical Wikimedia Workshop (T241705) (duration: 00m 01s) [12:39:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:40:02] (03PS2) 10Muehlenhoff: Pass the package list to the apt::pin [puppet] - 10https://gerrit.wikimedia.org/r/562246 (https://phabricator.wikimedia.org/T230324) [12:41:29] (03PS3) 10Urbanecm: Set Author and Author_talk aliases for Autore NS at napwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561359 (https://phabricator.wikimedia.org/T231880) [12:41:33] (03CR) 10Urbanecm: [C: 03+2] Set Author and Author_talk aliases for Autore NS at napwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561359 (https://phabricator.wikimedia.org/T231880) (owner: 10Urbanecm) [12:41:34] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 0baf554: Add new namespace and aliases for zh.wiktionary (T241023) (duration: 00m 54s) [12:41:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:41:37] T241023: Create Rhymes: namespace for zh.wiktionary - https://phabricator.wikimedia.org/T241023 [12:42:33] (03Merged) 10jenkins-bot: Set Author and Author_talk aliases for Autore NS at napwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561359 (https://phabricator.wikimedia.org/T231880) (owner: 10Urbanecm) [12:43:12] !log mwscript namespaceDupes.php --wiki=zhwiktionary --fix (T241023) [12:43:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:44:41] thank you :) [12:45:19] yw WhitePhosphorus [12:45:53] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 864a2f8: Set Author and Author_talk aliases for Autore NS at napwikisource (T231880) (duration: 00m 55s) [12:45:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:45:56] T231880: Author and Author_talk aliases needed on nap.wikisource - https://phabricator.wikimedia.org/T231880 [12:46:32] !log mwscript namespaceDupes.php --wiki=napwikisource --fix (T231880) [12:46:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:47:56] (03PS3) 10Urbanecm: Enable subpages for the main namespace on ge.wikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560339 (https://phabricator.wikimedia.org/T241329) (owner: 10MarcoAurelio) [12:48:02] (03CR) 10Urbanecm: [C: 03+2] Enable subpages for the main namespace on ge.wikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560339 (https://phabricator.wikimedia.org/T241329) (owner: 10MarcoAurelio) [12:49:04] (03Merged) 10jenkins-bot: Enable subpages for the main namespace on ge.wikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560339 (https://phabricator.wikimedia.org/T241329) (owner: 10MarcoAurelio) [12:50:50] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: c44b4ff: Enable subpages for the main namespace on ge.wikimedia (T241329) (duration: 00m 55s) [12:50:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:50:53] T241329: Please enable subpages for WMGE - https://phabricator.wikimedia.org/T241329 [12:51:40] (03PS3) 10Urbanecm: Added throttle rule for University of Derby mini-editathon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559612 (https://phabricator.wikimedia.org/T240845) (owner: 10NicholasG04) [12:51:43] 10Operations, 10Wikimedia-Mailing-lists: Migrate archives of the OKFN-hosted Open-GLAM mailing list to Wikimedia's mailman - https://phabricator.wikimedia.org/T240929 (10SandraF_WMF) Thanks both! I just emailed the .mbox file to @jcrespo - it's not small (44MB) so let's see how this goes. [12:52:30] (03PS4) 10Urbanecm: Added throttle rule for University of Derby mini-editathon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559612 (https://phabricator.wikimedia.org/T240845) (owner: 10NicholasG04) [12:52:42] (03PS5) 10Urbanecm: Added throttle rule for University of Derby mini-editathon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559612 (https://phabricator.wikimedia.org/T240845) (owner: 10NicholasG04) [12:53:15] (03PS6) 10Urbanecm: Add basic transwiki sources for ltwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559965 (https://phabricator.wikimedia.org/T241288) (owner: 10Vogone) [12:53:20] (03CR) 10Urbanecm: [C: 03+2] Add basic transwiki sources for ltwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559965 (https://phabricator.wikimedia.org/T241288) (owner: 10Vogone) [12:54:44] (03Merged) 10jenkins-bot: Add basic transwiki sources for ltwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559965 (https://phabricator.wikimedia.org/T241288) (owner: 10Vogone) [12:55:07] (03PS3) 10Zoranzoki21: Enable GeoData extension in ruwikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561657 (https://phabricator.wikimedia.org/T239000) [12:56:31] Hi, who doing SWAT? [12:56:36] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 88c800c: Add basic transwiki sources for ltwiki (T241288) (duration: 00m 54s) [12:56:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:56:40] T241288: Add basic import sources for ltwiki - https://phabricator.wikimedia.org/T241288 [12:57:01] Zoranzoki21: I, last four minutes, what's up? [12:57:12] (03PS3) 10Muehlenhoff: Pass the package list to the apt::pin [puppet] - 10https://gerrit.wikimedia.org/r/562246 (https://phabricator.wikimedia.org/T230324) [12:57:38] Nothing, I wanted to request merge of some throttle rule patches, but ok, I will request for morning SWAT [12:57:40] (03PS3) 10Ema: cache: remove profile::cache::varnish::backend and cron_restart [puppet] - 10https://gerrit.wikimedia.org/r/562231 (https://phabricator.wikimedia.org/T241239) [12:57:42] (03PS2) 10Ema: cache: remove varnish-backend-restart [puppet] - 10https://gerrit.wikimedia.org/r/562232 (https://phabricator.wikimedia.org/T241239) [12:57:44] (03PS1) 10Ema: cache: remove directors.vcl.tpl.erb [puppet] - 10https://gerrit.wikimedia.org/r/562253 (https://phabricator.wikimedia.org/T241239) [12:57:46] (03PS1) 10Ema: cache: remove varnish-be from profile::cache::base::default_weights [puppet] - 10https://gerrit.wikimedia.org/r/562254 (https://phabricator.wikimedia.org/T241239) [12:57:48] (03PS1) 10Ema: cache: remove varnish::logging::backend [puppet] - 10https://gerrit.wikimedia.org/r/562255 (https://phabricator.wikimedia.org/T241239) [12:57:50] (03PS1) 10Ema: cache: remove varnishmtail-backend [puppet] - 10https://gerrit.wikimedia.org/r/562256 (https://phabricator.wikimedia.org/T241239) [12:57:52] (03PS1) 10Ema: cache: remove check_varnish_expiry_mailbox_lag.sh [puppet] - 10https://gerrit.wikimedia.org/r/562257 (https://phabricator.wikimedia.org/T241239) [12:57:55] (03PS1) 10Ema: cache: do not pick varnish-be as origins for varnish-fe [puppet] - 10https://gerrit.wikimedia.org/r/562258 (https://phabricator.wikimedia.org/T241239) [12:58:04] !log EU SWAT done [12:58:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:06:07] (03PS1) 10Ema: cache: merge common_vcl_config into fe_vcl_config [puppet] - 10https://gerrit.wikimedia.org/r/562260 (https://phabricator.wikimedia.org/T241239) [13:11:52] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:13:53] (03PS2) 10Ema: cache: do not pick varnish-be as origins for varnish-fe [puppet] - 10https://gerrit.wikimedia.org/r/562258 (https://phabricator.wikimedia.org/T241239) [13:14:13] (03PS2) 10Ema: cache: merge common_vcl_config into fe_vcl_config [puppet] - 10https://gerrit.wikimedia.org/r/562260 (https://phabricator.wikimedia.org/T241239) [13:15:46] (03CR) 10Zoranzoki21: [C: 03+1] "Scheduled for today Morning SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) (owner: 10Superzerocool) [13:16:17] (03CR) 10Muehlenhoff: "PCC: https://puppet-compiler.wmflabs.org/compiler1001/20200/" [puppet] - 10https://gerrit.wikimedia.org/r/562246 (https://phabricator.wikimedia.org/T230324) (owner: 10Muehlenhoff) [13:16:25] 10Operations, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Ghostscript outputs errors to stdout despite -q, preventing Thumbor from generating some thumbnails properly - https://phabricator.wikimedia.org/T236240 (10Gilles) p:05Normal→03Low [13:17:13] (03CR) 10Zoranzoki21: [C: 03+1] "Scheduled for today Morning SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559612 (https://phabricator.wikimedia.org/T240845) (owner: 10NicholasG04) [13:19:25] (03CR) 10Muehlenhoff: [C: 03+2] Pass the package list to the apt::pin [puppet] - 10https://gerrit.wikimedia.org/r/562246 (https://phabricator.wikimedia.org/T230324) (owner: 10Muehlenhoff) [13:19:44] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=atlas_exporter site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [13:20:29] (03PS1) 10Zoranzoki21: Add www.digital.archives.go.jp to the wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562262 (https://phabricator.wikimedia.org/T238476) [13:21:31] 10Operations, 10Design-Research, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Aklapper) Hi and welcome @Dendelele! You may want to tell //WMF Design Strategy// to follow https://office.wikimedia.org/wiki/Guide_for_new_hires#Account_creation in the... [13:21:32] RECOVERY - Prometheus jobs reduced availability on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [13:21:38] (03PS2) 10Zoranzoki21: Add www.digital.archives.go.jp to the wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562262 (https://phabricator.wikimedia.org/T238476) [13:24:23] 10Operations, 10Traffic: servers freeze across the caching cluster - https://phabricator.wikimedia.org/T238305 (10ema) [13:24:53] (03CR) 10Ema: [C: 03+2] cache: remove profile::cache::varnish::backend and cron_restart [puppet] - 10https://gerrit.wikimedia.org/r/562231 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [13:27:53] PROBLEM - Check for VMs leaked by the nova-fullstack test on cloudcontrol1003 is CRITICAL: 10 instances in the admin-monitoring project https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting%23Nova-fullstack [13:30:33] (03CR) 10Ema: [C: 03+2] cache: remove varnish-backend-restart [puppet] - 10https://gerrit.wikimedia.org/r/562232 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [13:31:22] (03PS1) 10Zoranzoki21: Add mediaphoto.mnhn.fr to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562265 (https://phabricator.wikimedia.org/T241637) [13:32:35] (03PS2) 10Zoranzoki21: Add mediaphoto.mnhn.fr to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562265 (https://phabricator.wikimedia.org/T241637) [13:33:15] RECOVERY - Check for VMs leaked by the nova-fullstack test on cloudcontrol1003 is OK: 0 instances in the admin-monitoring project https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting%23Nova-fullstack [13:35:12] !log reimaging mw2282 to validate correctness of apt::package_from_component for fresh installs [13:35:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:36:28] (03CR) 10Tchanders: [C: 03+1] [officewiki] Grant ipblock-exempt to all users on officewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559811 (https://phabricator.wikimedia.org/T231943) (owner: 10Jforrester) [13:37:24] (03CR) 10Ema: [C: 03+2] cache: remove directors.vcl.tpl.erb [puppet] - 10https://gerrit.wikimedia.org/r/562253 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [13:41:19] (03CR) 10Ema: [C: 03+2] cache: remove varnish-be from profile::cache::base::default_weights [puppet] - 10https://gerrit.wikimedia.org/r/562254 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [13:45:48] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:52:38] (03CR) 10Ema: [C: 03+2] cache: remove varnish::logging::backend [puppet] - 10https://gerrit.wikimedia.org/r/562255 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [13:54:27] !log jmm@cumin2001 START - Cookbook sre.hosts.downtime [13:54:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:56:38] !log jmm@cumin2001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [13:56:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:59:03] (03CR) 10Ema: [C: 03+2] cache: remove varnishmtail-backend [puppet] - 10https://gerrit.wikimedia.org/r/562256 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [13:59:05] (03PS1) 10Jakob: Termbox: adjust entry point paths [deployment-charts] - 10https://gerrit.wikimedia.org/r/562272 [14:00:12] (03CR) 10Jakob: "This change is ready for review." [deployment-charts] - 10https://gerrit.wikimedia.org/r/562272 (owner: 10Jakob) [14:02:56] (03CR) 10Ema: [C: 03+2] cache: remove check_varnish_expiry_mailbox_lag.sh [puppet] - 10https://gerrit.wikimedia.org/r/562257 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [14:06:16] PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 24565912 and 3 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [14:07:34] RECOVERY - Postgres Replication Lag on maps2002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 0 and 5 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [14:09:24] PROBLEM - Postgres Replication Lag on maps1002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 44523864 and 4 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [14:10:19] (03CR) 10Ema: [C: 03+2] cache: do not pick varnish-be as origins for varnish-fe [puppet] - 10https://gerrit.wikimedia.org/r/562258 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [14:10:48] RECOVERY - Postgres Replication Lag on maps1002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 7256 and 18 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [14:16:27] (03CR) 10Ema: [C: 03+2] cache: merge common_vcl_config into fe_vcl_config [puppet] - 10https://gerrit.wikimedia.org/r/562260 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [14:16:34] (03PS3) 10Ema: cache: merge common_vcl_config into fe_vcl_config [puppet] - 10https://gerrit.wikimedia.org/r/562260 (https://phabricator.wikimedia.org/T241239) [14:17:12] jouncebot: now [14:17:12] No deployments scheduled for the next 3 hour(s) and 42 minute(s) [14:17:15] jouncebot: next [14:17:15] In 3 hour(s) and 42 minute(s): Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200106T1800) [14:18:20] (03PS2) 10Reedy: Enable WebAuthn extension on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560227 [14:26:21] (03CR) 10Ottomata: [C: 03+1] profile::analytics::client::limits: add cpu limits to Analytics clients [puppet] - 10https://gerrit.wikimedia.org/r/561675 (https://phabricator.wikimedia.org/T240440) (owner: 10Elukey) [14:28:36] (03CR) 10Reedy: [C: 03+2] Enable WebAuthn extension on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560227 (owner: 10Reedy) [14:28:49] (03PS2) 10Jakob: Termbox: adjust entry point paths [deployment-charts] - 10https://gerrit.wikimedia.org/r/562272 [14:29:37] (03Merged) 10jenkins-bot: Enable WebAuthn extension on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560227 (owner: 10Reedy) [14:31:00] !log reedy@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Enable WebAuthn everywhere (duration: 00m 57s) [14:31:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:32:30] !log reedy@deploy1001 Synchronized wmf-config/InitialiseSettings-labs.php: Enable WebAuthn everywhere (duration: 00m 54s) [14:32:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:33:29] 10Operations, 10Wikimedia-Mailing-lists: Request changes to Malaysia Mailing List - https://phabricator.wikimedia.org/T241988 (10Exec8) [14:33:53] 10Operations, 10Wikimedia-Mailing-lists: Request changes to Malaysia Mailing List - https://phabricator.wikimedia.org/T241988 (10Exec8) [14:40:29] Reedy: Huh, didn't realise WebAuthn was ready for prod use. Nice! [14:41:31] James_F: We fixed the issue with it using the wrong DB... [14:41:46] It doesn't increase the pool of people that can use it... [14:41:47] that could be a problem [14:42:26] Reedy: Well, all the people with 2FA who currently lose their reset tokens and come crying to T&S now have a second way to break their account. ;-) [14:42:46] 10Operations, 10Wikimedia-Mailing-lists: Request changes to Malaysia Mailing List - https://phabricator.wikimedia.org/T241988 (10Exec8) [14:42:57] James_F: bd808 did give me a t-shirt for a reason [14:43:06] Reedy: But did Jan? ;-) [14:44:03] He's aware :D [14:44:29] U2F is only slightly worse than OATH as it doesn't have recovery keys... [14:44:34] * James_F nods. [14:45:25] Rolling both out to more users is a TBC issue :) [14:45:33] what stops an implementation from supporting a recovery keys? [14:45:51] Nothing particularly [14:45:57] But it needs some refactoring [14:46:02] 10Operations, 10serviceops: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T240684 (10jijiki) p:05Triage→03Normal [14:46:16] U2F does support multiple keys though [14:46:19] Which reminds me [14:46:28] * Reedy goes to look at the progress of the solo keys U2F [14:48:30] (03PS1) 10Ema: cache: remove varnishbackend.mtail [puppet] - 10https://gerrit.wikimedia.org/r/562279 (https://phabricator.wikimedia.org/T241239) [14:49:52] start giving out yubikeys for free to people that enrolls to use them [14:50:02] that would rapidly increase the number of users [14:50:10] it's not particularly cheap, though :P [14:50:10] They're not free to buy though ;P [14:51:26] I know, despite the per-volume discounts [14:51:34] * James_F wanders off from the office to get breakfast. [14:51:40] They do hand them out like candy at some conferences and stuff [14:51:49] Maybe if they're listening, they might want to consider donating a load :) [14:54:33] (03CR) 10Ammarpad: [C: 04-1] "Needs rebasing and commit summary update. The older exceptions have been removing in I9431e7d" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) (owner: 10Superzerocool) [14:55:04] (03CR) 10Ammarpad: [C: 04-1] "*removed" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) (owner: 10Superzerocool) [14:56:40] !log remove BGP to AS13285 in eqiad (IXP not listed in peeringdb anymore) [14:56:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:02:06] (03PS1) 10ArielGlenn: Don't fail on symlink cleanup if symlink disappears before removal [dumps] - 10https://gerrit.wikimedia.org/r/562281 [15:04:24] (03CR) 10ArielGlenn: [C: 03+2] Don't fail on symlink cleanup if symlink disappears before removal [dumps] - 10https://gerrit.wikimedia.org/r/562281 (owner: 10ArielGlenn) [15:04:45] !log remove BGP to AS13285 in ulsfo (IXP not listed in peeringdb anymore) [15:04:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:04:54] 10Operations, 10ops-eqiad: Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T241881 (10JHedden) [15:04:59] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T241884 (10JHedden) [15:05:09] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T241884 (10JHedden) Looks like we're missing drives in slot 2 and 9 on this host. ` megacli -PDList -aALL | egrep 'Slot|Firmware' Slot Number: 0 Firmware state... [15:05:38] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T241884 (10JHedden) And slot 4! [15:06:16] !log ariel@deploy1001 Started deploy [dumps/dumps@db81d78]: avoid aborts on some symlink cleanup failures [15:06:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:06:23] !log ariel@deploy1001 Finished deploy [dumps/dumps@db81d78]: avoid aborts on some symlink cleanup failures (duration: 00m 06s) [15:06:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:06:45] 10Operations: Migrate URL downloaders to Stretch/Buster - https://phabricator.wikimedia.org/T224551 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff [15:17:52] 10Operations, 10vm-requests: eqiad/codfw: 2 VM request for URL downloaders - https://phabricator.wikimedia.org/T241979 (10Reedy) [15:18:16] (03CR) 10Bstorm: "> Patch Set 2:" [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561262 (owner: 10Legoktm) [15:19:32] 10Operations, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor broken on Wikitech - https://phabricator.wikimedia.org/T241961 (10Reedy) Yeah, this is known and "expected"... Pending more migrations I can't find the relevant task offhand though [15:20:55] 10Operations, 10Wikimedia-Mailing-lists: Request changes to Malaysia Mailing List - https://phabricator.wikimedia.org/T241988 (10Reedy) a:05Exec8→03None [15:21:14] (03PS1) 10Muehlenhoff: Add DNS entries for urldownloader* [dns] - 10https://gerrit.wikimedia.org/r/562283 (https://phabricator.wikimedia.org/T224551) [15:21:35] (03CR) 10jerkins-bot: [V: 04-1] Add DNS entries for urldownloader* [dns] - 10https://gerrit.wikimedia.org/r/562283 (https://phabricator.wikimedia.org/T224551) (owner: 10Muehlenhoff) [15:21:48] 10Operations, 10Parsoid, 10VisualEditor, 10wikitech.wikimedia.org: VisualEditor was removed from Wikitech because Parsoid/PHP isn't yet compatible with how Wikitech is set up - https://phabricator.wikimedia.org/T241961 (10Jdforrester-WMF) [15:22:36] (03PS1) 10Muehlenhoff: Add urldownloader* to partman config [puppet] - 10https://gerrit.wikimedia.org/r/562284 [15:24:42] (03PS2) 10Jforrester: [officewiki] Grant ipblock-exempt to all users on officewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559811 (https://phabricator.wikimedia.org/T231943) [15:27:19] (03CR) 10Muehlenhoff: [C: 03+2] Add urldownloader* to partman config [puppet] - 10https://gerrit.wikimedia.org/r/562284 (owner: 10Muehlenhoff) [15:27:26] (03CR) 10Jforrester: [C: 03+2] [officewiki] Grant ipblock-exempt to all users on officewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559811 (https://phabricator.wikimedia.org/T231943) (owner: 10Jforrester) [15:28:28] (03Merged) 10jenkins-bot: [officewiki] Grant ipblock-exempt to all users on officewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559811 (https://phabricator.wikimedia.org/T231943) (owner: 10Jforrester) [15:30:19] !log jforrester@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [officewiki] Grant ipblock-exempt to all users T231943 (duration: 00m 56s) [15:30:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:30:22] T231943: Uncheck "Automatically block the last IP address" by default on Special:Block on officewiki - https://phabricator.wikimedia.org/T231943 [15:35:42] (03PS1) 10Muehlenhoff: Switch Gerrit/Phabricator to standard Partman recipes [puppet] - 10https://gerrit.wikimedia.org/r/562285 (https://phabricator.wikimedia.org/T156955) [15:44:05] 10Operations, 10Traffic, 10fixcopyright.wikimedia.org, 10Core Platform Team Workboards (Clinic Duty Team), and 3 others: Retire fixcopyright.wikimedia.org - https://phabricator.wikimedia.org/T238803 (10Jdforrester-WMF) [15:44:07] (03PS2) 10Muehlenhoff: Add DNS entries for urldownloader* [dns] - 10https://gerrit.wikimedia.org/r/562283 (https://phabricator.wikimedia.org/T224551) [15:44:29] (03CR) 10jerkins-bot: [V: 04-1] Add DNS entries for urldownloader* [dns] - 10https://gerrit.wikimedia.org/r/562283 (https://phabricator.wikimedia.org/T224551) (owner: 10Muehlenhoff) [15:45:19] !log milimetric@deploy1001 Started deploy [analytics/refinery@09133cf]: Fix for geoeditors monthly [15:45:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:45:59] (03PS3) 10Muehlenhoff: Add DNS entries for urldownloader* [dns] - 10https://gerrit.wikimedia.org/r/562283 (https://phabricator.wikimedia.org/T224551) [15:47:53] !log migrating mx1001 to seconday ganeti node T240906 [15:47:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:47:56] T240906: CA App Synthetic Monitor Mail (SMTP): Connection timed out; connect(): -2 - https://phabricator.wikimedia.org/T240906 [15:56:52] (03PS8) 10Ottomata: New eventstreams chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/551843 (https://phabricator.wikimedia.org/T238658) [15:57:09] !log milimetric@deploy1001 Finished deploy [analytics/refinery@09133cf]: Fix for geoeditors monthly (duration: 11m 49s) [15:57:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:07:09] 10Operations, 10observability: Add RIPE atlas data to Prometheus - https://phabricator.wikimedia.org/T167689 (10CDanis) a:03CDanis The steps outlined in [[ https://phabricator.wikimedia.org/T167689#3341476 | Filippo's comment ]] happened, with the difference that I chose to use the netmon* machines for this... [16:07:33] PROBLEM - mediawiki-installation DSH group on mw2282 is CRITICAL: Host mw2282 is not in mediawiki-installation dsh group https://wikitech.wikimedia.org/wiki/Monitoring/check_dsh_groups [16:09:16] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Termbox: adjust entry point paths [deployment-charts] - 10https://gerrit.wikimedia.org/r/562272 (owner: 10Jakob) [16:15:15] 10Operations, 10ops-codfw, 10serviceops: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet - https://phabricator.wikimedia.org/T241796 (10Papaul) 05Open→03Declined @elukey we can close this task and keep T239249 [16:15:17] 10Operations, 10serviceops: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T240684 (10Papaul) [16:22:17] 10Operations, 10Wikimedia-Mailing-lists: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Aklapper) [16:22:53] 10Operations, 10Wikimedia-Mailing-lists: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Aklapper) Hi @Exec8, thanks for taking the time to report this and welcome to Wikimedia Phabricator! Please see https://meta.wikimedia.org/wiki/Mailing_lists... [16:23:05] 10Operations, 10CX-cxserver, 10Citoid, 10Core Platform Team, and 9 others: Make services swagger specs standard compliant - https://phabricator.wikimedia.org/T218217 (10MSantos) [16:23:50] (03PS1) 10Jakob: Revert "Revert "termbox: update test service to latest"" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562291 [16:24:45] 10Operations, 10Wikimedia-Mailing-lists: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Taufik) My email is mohdtaufikrosman@gmail.com. [16:26:27] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Revert "Revert "termbox: update test service to latest"" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562291 (owner: 10Jakob) [16:26:48] (03PS2) 10Ladsgroup: Revert "Revert "termbox: update test service to latest"" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562291 (owner: 10Jakob) [16:26:54] (03CR) 10Ladsgroup: [V: 03+2 C: 03+2] Revert "Revert "termbox: update test service to latest"" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562291 (owner: 10Jakob) [16:27:13] !log jakob@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'test' . [16:27:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:28:21] (03PS4) 10Herron: ganeti: apply ferm regardless of ganeti_cluster fact [puppet] - 10https://gerrit.wikimedia.org/r/559879 [16:30:36] (03PS9) 10Ottomata: New eventstreams chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/551843 (https://phabricator.wikimedia.org/T238658) [16:30:58] (03PS3) 10EBernhardson: Perform weekly dumps of all public media urls [puppet] - 10https://gerrit.wikimedia.org/r/561356 (https://phabricator.wikimedia.org/T240520) [16:32:28] (03CR) 10Herron: "https://puppet-compiler.wmflabs.org/compiler1002/20217/" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/559879 (owner: 10Herron) [16:38:29] (03PS1) 10Herron: profile::ganeti switch from hiera() to lookup() [puppet] - 10https://gerrit.wikimedia.org/r/562294 [16:42:42] 10Operations, 10ops-codfw: (No Need By Date Provided) codfw: rack/setup/install elastic20{55,56,57,58,59,60}.wikimedia.org - https://phabricator.wikimedia.org/T241337 (10EBernhardson) In terms of "how close", I would have to check but probably 2 years worth of server replacements before the full cluster is 10... [16:43:20] (03CR) 10Herron: "https://puppet-compiler.wmflabs.org/compiler1003/20219/ -- looks good but has diff due to unmerged parent. should re-run after thats merg" [puppet] - 10https://gerrit.wikimedia.org/r/562294 (owner: 10Herron) [16:48:42] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T241884 (10wiki_willy) a:03Jclark-ctr [16:48:43] 10Operations, 10ops-eqiad: Degraded RAID on cloudvirt1014 - https://phabricator.wikimedia.org/T241494 (10wiki_willy) a:03Jclark-ctr [16:50:15] (03CR) 10Ema: [C: 03+2] cache: remove varnishbackend.mtail [puppet] - 10https://gerrit.wikimedia.org/r/562279 (https://phabricator.wikimedia.org/T241239) (owner: 10Ema) [16:51:28] robh: I noticed that T241784 (like the hardware request) says "Please place all 3 new hosts in row C, in different 1G racks", was that copypasta, or were we still going to discuss this? :) [16:51:28] T241784: (No Need By Date) rack/setup/install restbase1029, restbase1029, restbase1030 - https://phabricator.wikimedia.org/T241784 [16:51:55] urandom: i just took that off the procurement task [16:52:01] if that isnt going to be right, please put on task asap [16:52:07] via comment so jclark-ctr doesnt rack that way [16:52:18] all of the racking details are listed on the procurmenet task at the time of order =] [16:52:41] I know, I updated that tasks (via comment) at the time :) [16:52:53] https://phabricator.wikimedia.org/T238580#5726426 [16:53:15] ahh, i recall now [16:53:23] so row b is 10G contrained [16:53:29] since cloudvirt takes up a lot [16:53:39] i'll list that we want this there, but they may not be able to accomodate [16:54:05] 10Operations, 10ops-eqiad, 10Core Platform Team: (No Need By Date) rack/setup/install restbase1029, restbase1029, restbase1030 - https://phabricator.wikimedia.org/T241784 (10RobH) >>! In T238580#5710739, @Eevans wrote: >>>! In T238580#5709953, @RobH wrote: >> Also note I assumed details for the racking/hostn... [16:54:15] (03CR) 10Jdlrobson: [C: 03+1] "config looks right to me. Thanks @ammarpad! la wiki will be very grateful!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/557439 (https://phabricator.wikimedia.org/T240728) (owner: 10Ammarpad) [16:54:17] (03PS1) 10Jakob: Revert "Revert "Revert "termbox: update test service to latest""" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562295 [16:54:34] updated [16:54:35] (03CR) 10Jakob: [C: 03+2] Revert "Revert "Revert "termbox: update test service to latest""" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562295 (owner: 10Jakob) [16:54:37] 10Operations, 10ops-eqiad, 10Core Platform Team: (No Need By Date) rack/setup/install restbase1029, restbase1029, restbase1030 - https://phabricator.wikimedia.org/T241784 (10RobH) [16:54:39] urandom: refresh [16:54:51] if they cannot find room for a 1u in 10G in row b they'll comment on task [16:54:53] (03Merged) 10jenkins-bot: Revert "Revert "Revert "termbox: update test service to latest""" [deployment-charts] - 10https://gerrit.wikimedia.org/r/562295 (owner: 10Jakob) [16:55:11] (03CR) 10Jdlrobson: [C: 03+1] "yay! one less wiki using the mobilefrontend legacy formatter! Responsive main pages are the best!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562025 (https://phabricator.wikimedia.org/T241888) (owner: 10Ammarpad) [16:55:12] 10Operations, 10Wikimedia-Mailing-lists: Migrate archives of the OKFN-hosted Open-GLAM mailing list to Wikimedia's mailman - https://phabricator.wikimedia.org/T240929 (10jcrespo) I have forwarded the link to @herron [16:55:21] robh: \o/ [16:55:22] thanks [16:56:03] !log jakob@deploy1001 helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'test' . [16:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:57:09] 10Operations, 10ops-codfw, 10Core Platform Team: (No Need By Date Provided) rack/setup/install restbase202[123] - https://phabricator.wikimedia.org/T241790 (10Eevans) [16:58:06] 10Operations, 10ops-codfw, 10Core Platform Team: (No Need By Date Provided) rack/setup/install restbase202[123] - https://phabricator.wikimedia.org/T241790 (10Eevans) >>! In T238580#5710739, @Eevans wrote: >>>! In T238580#5709953, @RobH wrote: >> Also note I assumed details for the racking/hostnames and woul... [16:58:20] robh: I had a matching change to T241790 as well [16:58:21] T241790: (No Need By Date Provided) rack/setup/install restbase202[123] - https://phabricator.wikimedia.org/T241790 [16:58:41] cool [16:59:07] 10Operations, 10Wikimedia-Mailing-lists: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Exec8) You can make me a co-admin. My email address is bustrias@gmail.com [17:01:03] (03PS10) 10Ottomata: New eventstreams chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/551843 (https://phabricator.wikimedia.org/T238658) [17:04:59] 10Puppet, 10Horizon: Invalid yaml in horizon hiera editor results in confusing error message - https://phabricator.wikimedia.org/T241999 (10bd808) [17:10:15] 10Operations, 10ops-eqiad, 10Discovery-Search (Current work): (No Need By Date) rack/setup/install relforge100[34] - https://phabricator.wikimedia.org/T241791 (10EBernhardson) regarding the racking proposal > 1G network rack. These should be 10G (I verified PO contains 10G cards as well). [17:10:22] 10Operations, 10LDAP-Access-Requests: Allow LDAP access to Moushira Elamrawy - https://phabricator.wikimedia.org/T242000 (10Moushira) [17:11:28] 10Operations, 10LDAP-Access-Requests: Allow LDAP access to Moushira Elamrawy - https://phabricator.wikimedia.org/T242000 (10Moushira) a:05elukey→03None [17:11:54] !log jakob@deploy1001 helmfile [STAGING] Ran 'sync' command on namespace 'termbox' for release 'test' . [17:11:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:16:35] 10Operations, 10LDAP-Access-Requests: Allow LDAP access to superset dashboards for Moushira Elamrawy - https://phabricator.wikimedia.org/T242000 (10Aklapper) [17:17:07] 10Operations, 10LDAP-Access-Requests: Allow LDAP access to superset dashboards for Moushira Elamrawy - https://phabricator.wikimedia.org/T242000 (10Aklapper) Hi, what is your user name used on wikitech? See https://phabricator.wikimedia.org/tag/ldap-access-requests/ [17:28:02] 10Puppet, 10Horizon: Invalid yaml in horizon hiera editor results in confusing error message - https://phabricator.wikimedia.org/T241999 (10bd808) [17:30:40] (03CR) 10Jforrester: "Oops. :-)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/560521 (https://phabricator.wikimedia.org/T240455) (owner: 10Reedy) [17:31:27] (03PS1) 10EBernhardson: Revert "Reduce query load on cirrus elastic clusters" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562308 [17:31:41] (03CR) 10jerkins-bot: [V: 04-1] Revert "Reduce query load on cirrus elastic clusters" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562308 (owner: 10EBernhardson) [17:33:31] (03PS2) 10EBernhardson: Revert "Reduce query load on cirrus elastic clusters" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562308 [17:33:49] (03PS1) 10EBernhardson: Revert "cirrus: Shift more_like to codfw cirrus cluster" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562309 [17:34:44] (03PS2) 10EBernhardson: Revert "cirrus: Shift more_like to codfw cirrus cluster" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562309 [17:35:30] (03PS11) 10Ottomata: New eventstreams chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/551843 (https://phabricator.wikimedia.org/T238658) [17:45:39] 10Operations, 10Design-Research, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Dendelele) Thank you. I am not on the internal contact list but I am [[ https://wikimediafoundation.org/role/staff-contractors/ | here ]]. I will work with OIT. [17:46:44] (03PS4) 10EBernhardson: Perform weekly dumps of all public media urls [puppet] - 10https://gerrit.wikimedia.org/r/561356 (https://phabricator.wikimedia.org/T240520) [17:51:52] (03PS1) 10Ottomata: Remove now unused deployment-prep evenstreams config [puppet] - 10https://gerrit.wikimedia.org/r/562310 (https://phabricator.wikimedia.org/T238658) [17:52:25] (03PS2) 10Ottomata: Remove now unused deployment-prep eventstreams config [puppet] - 10https://gerrit.wikimedia.org/r/562310 (https://phabricator.wikimedia.org/T238658) [17:54:02] (03CR) 10Ottomata: [C: 03+2] Remove now unused deployment-prep eventstreams config [puppet] - 10https://gerrit.wikimedia.org/r/562310 (https://phabricator.wikimedia.org/T238658) (owner: 10Ottomata) [17:54:52] (03PS12) 10Ottomata: New eventstreams chart [deployment-charts] - 10https://gerrit.wikimedia.org/r/551843 (https://phabricator.wikimedia.org/T238658) [17:56:40] 10Operations, 10ops-eqsin: rack/setup/install ganeti500[123].eqsin.wmnet - https://phabricator.wikimedia.org/T228099 (10RobH) Please note that while these hosts are responsive to ssh on mgmt, I cannot login to them with the asset tag (per initial request) old mgmt pass, or new mgmt pass. The new PDUs for EQSI... [18:00:04] gehel and onimisionipe: Dear deployers, time to do the Wikidata Query Service weekly deploy deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200106T1800). [18:00:52] 10Operations, 10Research, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to analytics-privatedata-users and researchers for Aroraakhil - https://phabricator.wikimedia.org/T241096 (10leila) >>! In T241096#5763697, @jcrespo wrote: > @Leila researchers typically have time-limited MOUs, is this t... [18:09:38] 10Operations, 10SRE-Access-Requests: Restore access for bmansurov - https://phabricator.wikimedia.org/T241089 (10leila) @jcrespo thanks for working on this while most of us were away! :) [18:09:45] PROBLEM - Postgres Replication Lag on maps1002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 49960872 and 3 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [18:11:31] RECOVERY - Postgres Replication Lag on maps1002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 63176 and 94 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [18:11:34] 10Operations, 10ops-eqiad, 10DC-Ops, 10cloud-services-team (Kanban): cloudvirt1016 crash - https://phabricator.wikimedia.org/T241882 (10wiki_willy) a:03Jclark-ctr [18:13:06] 10Operations, 10ops-eqiad, 10DC-Ops, 10cloud-services-team (Kanban): cloudvirt1016 crash - https://phabricator.wikimedia.org/T241882 (10wiki_willy) @jclark-ctr looks like this one is warranty until May 2020, so you can just RMA this with Dell. Thanks, Willy [18:16:54] !log ebernhardson@deploy1001 Started deploy [search/airflow@8ae8500]: Require apache-airflow[kerberos] python package [18:16:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:17:21] !log ebernhardson@deploy1001 Finished deploy [search/airflow@8ae8500]: Require apache-airflow[kerberos] python package (duration: 00m 27s) [18:17:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:24:08] 10Operations, 10ops-codfw, 10serviceops: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet - https://phabricator.wikimedia.org/T241796 (10Papaul) [18:24:26] 10Operations, 10ops-codfw: (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet - https://phabricator.wikimedia.org/T239249 (10Papaul) [18:24:38] 10Operations, 10ops-eqiad, 10DC-Ops, 10cloud-services-team (Kanban): cloudvirt1013: server down for no reason (power issue?) - https://phabricator.wikimedia.org/T241313 (10wiki_willy) a:03Jclark-ctr [18:26:24] 10Operations, 10ops-eqiad, 10Patch-For-Review, 10cloud-services-team (Kanban): Degraded RAID on cloudvirt1024 - https://phabricator.wikimedia.org/T241884 (10wiki_willy) @Jclark-ctr - if no errors pop up after firmware and bios upgrades, maybe it a bad raid controller. [18:34:46] (03CR) 10BBlack: [C: 03+2] varnish: Minor wording update for browsersec/sec-warning [puppet] - 10https://gerrit.wikimedia.org/r/558735 (https://phabricator.wikimedia.org/T238038) (owner: 10Krinkle) [18:43:52] (03PS1) 10Joal: Add tables to analytics regular sqoop list [puppet] - 10https://gerrit.wikimedia.org/r/562322 (https://phabricator.wikimedia.org/T242015) [18:48:08] !log ebernhardson@deploy1001 Started deploy [search/airflow@8db442c]: match cryptography package with debian buster [18:48:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:49:02] !log ebernhardson@deploy1001 Finished deploy [search/airflow@8db442c]: match cryptography package with debian buster (duration: 00m 53s) [18:49:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:49:11] 10Operations, 10ops-codfw, 10Core Platform Team: (No Need By Date Provided) rack/setup/install restbase202[123] - https://phabricator.wikimedia.org/T241790 (10Papaul) [18:57:30] 10Operations, 10Traffic, 10hardware-requests: Add 10G NICs to core site DNS servers (6 servers, 3 per site) - https://phabricator.wikimedia.org/T239675 (10Papaul) [18:59:24] jouncebot: now [18:59:24] No deployments scheduled for the next 0 hour(s) and 0 minute(s) [19:00:05] RoanKattouw, Niharika, and Urbanecm: How many deployers does it take to do Morning SWAT(Max 6 patches) deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200106T1900). [19:00:05] Ammarpad and Zoranzoki21: A patch you scheduled for Morning SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [19:00:16] I am here [19:00:17] Zoranzoki21: hi! [19:01:02] If Ammarpad isn't here I can help with his patches [19:01:14] You will SWAT Urbanecm? [19:01:16] Zoranzoki21: kinda reculant to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/559612, Nicholas might want to schedule it on their own with GCI [19:01:20] yes, I can SWAT today! [19:01:35] Zoranzoki21: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/562056 has merge config [19:01:45] (03PS4) 10Urbanecm: Enable GeoData extension in ruwikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561657 (https://phabricator.wikimedia.org/T239000) (owner: 10Zoranzoki21) [19:01:56] (03CR) 10Urbanecm: [C: 03+2] Enable GeoData extension in ruwikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561657 (https://phabricator.wikimedia.org/T239000) (owner: 10Zoranzoki21) [19:02:08] Will fix merge conflict, let me clone operations/mediawiki-config [19:02:56] Ammarpad's patches are mostly inactionable, one wrong-linked, one merged already [19:03:05] (03Merged) 10jenkins-bot: Enable GeoData extension in ruwikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561657 (https://phabricator.wikimedia.org/T239000) (owner: 10Zoranzoki21) [19:03:24] But he removed some of my patches, and moved for Evening SWAT [19:03:58] Zoranzoki21: check 561657 at mwdebug1001 [19:04:17] Will check now [19:04:43] You can +2 https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/559576/ I will check it on mwdebug [19:04:55] (03PS1) 10EBernhardson: airflow: Remove PrivateTmp from airflow-kerberos unit [puppet] - 10https://gerrit.wikimedia.org/r/562329 [19:06:04] Urbanecm: Ok is [19:06:47] syncing [19:07:02] (03CR) 10Urbanecm: [C: 03+2] Add sandboxlink for eswikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559576 (https://phabricator.wikimedia.org/T241163) (owner: 10Ammarpad) [19:07:08] (03PS4) 10Urbanecm: Add sandboxlink for eswikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559576 (https://phabricator.wikimedia.org/T241163) (owner: 10Ammarpad) [19:07:14] (03CR) 10Urbanecm: Add sandboxlink for eswikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559576 (https://phabricator.wikimedia.org/T241163) (owner: 10Ammarpad) [19:07:19] (03CR) 10Urbanecm: [C: 03+2] Add sandboxlink for eswikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559576 (https://phabricator.wikimedia.org/T241163) (owner: 10Ammarpad) [19:08:18] (03Merged) 10jenkins-bot: Add sandboxlink for eswikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/559576 (https://phabricator.wikimedia.org/T241163) (owner: 10Ammarpad) [19:08:41] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: d7a19ca: Enable GeoData extension in ruwikinews (T239000) (duration: 00m 56s) [19:08:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:08:44] T239000: Deploying GeoData extension in the Russian Wikinews - https://phabricator.wikimedia.org/T239000 [19:09:07] Zoranzoki21: please test 559576 at mwdebug1001 [19:09:23] OK, I will [19:10:18] It's so hard to load [19:10:43] Zoranzoki21: what do you mean? [19:10:49] It loads so slow [19:11:18] 10Operations, 10Wikimedia-Mailing-lists: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Dzahn) Hello, i'll take this since i'm on rotating clinic duty this week. Would it be ok with you if we simply create a new list with a new admin? Since it'... [19:11:20] Main page is so big [19:11:33] Zoranzoki21: agreed, but that's another issue [19:11:42] Ok loaded [19:12:47] All ok, sync [19:13:27] syncing [19:14:46] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: c3a3248: Add sandboxlink for eswikivoyage (T241163) (duration: 00m 58s) [19:14:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:14:50] T241163: Enable SandboxLink extension in Spanish Wikivoyage - https://phabricator.wikimedia.org/T241163 [19:15:32] (03PS2) 10Zoranzoki21: Add throttle rule for ECLAC editathon in Santiago, Chile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) (owner: 10Superzerocool) [19:15:49] (03PS3) 10Urbanecm: Add www.digital.archives.go.jp to the wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562262 (https://phabricator.wikimedia.org/T238476) (owner: 10Zoranzoki21) [19:15:51] Conflict for https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/562056/ is resolved [19:15:59] (03CR) 10Urbanecm: [C: 03+2] Add www.digital.archives.go.jp to the wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562262 (https://phabricator.wikimedia.org/T238476) (owner: 10Zoranzoki21) [19:16:31] (03PS3) 10Urbanecm: Add throttle rule for ECLAC editathon in Santiago, Chile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) (owner: 10Superzerocool) [19:16:47] (03CR) 10Urbanecm: [C: 03+2] Add throttle rule for ECLAC editathon in Santiago, Chile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) (owner: 10Superzerocool) [19:17:12] (03PS3) 10Urbanecm: Add mediaphoto.mnhn.fr to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562265 (https://phabricator.wikimedia.org/T241637) (owner: 10Zoranzoki21) [19:17:21] (03CR) 10Urbanecm: [C: 03+2] Add mediaphoto.mnhn.fr to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562265 (https://phabricator.wikimedia.org/T241637) (owner: 10Zoranzoki21) [19:17:23] (03Merged) 10jenkins-bot: Add www.digital.archives.go.jp to the wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562262 (https://phabricator.wikimedia.org/T238476) (owner: 10Zoranzoki21) [19:17:58] (03Merged) 10jenkins-bot: Add throttle rule for ECLAC editathon in Santiago, Chile [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562056 (https://phabricator.wikimedia.org/T241414) (owner: 10Superzerocool) [19:18:44] (03Merged) 10jenkins-bot: Add mediaphoto.mnhn.fr to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562265 (https://phabricator.wikimedia.org/T241637) (owner: 10Zoranzoki21) [19:19:02] syncing the throttle rule now [19:19:40] ok [19:19:43] I closed tasks as resolved [19:19:47] !log urbanecm@deploy1001 Synchronized wmf-config/throttle.php: SWAT: 1324af9: Add throttle rule for ECLAC editathon in Santiago, Chile (T241414) (duration: 00m 54s) [19:19:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:19:50] T241414: Lift IP limit - WMCL Editathon 2020-01-27 - https://phabricator.wikimedia.org/T241414 [19:21:20] Thanks for fixing calendar [19:21:47] yw [19:21:54] syncing the whitelists too [19:22:01] You can +2 https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/555621/ too after syncing others [19:22:28] 10Operations, 10vm-requests: eqiad/codfw: 2 VM request for URL downloaders - https://phabricator.wikimedia.org/T241979 (10Dzahn) a:03Dzahn [19:22:33] (03CR) 10Urbanecm: [C: 03+2] Enable local uploads on inh.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/555621 (https://phabricator.wikimedia.org/T239925) (owner: 10Ammarpad) [19:22:39] !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: 7722ff3: 0bff587: Add www.digital.archives.go.jp/mediaphoto.mnhn.fr to the wgCopyUploadsDomains (T238476, T241637) (duration: 00m 54s) [19:22:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:22:43] T241637: Add mediaphoto.mnhn.fr to $wgCopyUploadsDomains - https://phabricator.wikimedia.org/T241637 [19:22:43] T238476: Add *.archives.go.jp to the wgCopyUploadsDomains whitelist of Wikimedia Commons - https://phabricator.wikimedia.org/T238476 [19:22:51] 10Operations, 10vm-requests: eqiad/codfw: 2 VM request for URL downloaders - https://phabricator.wikimedia.org/T241979 (10Dzahn) p:05Triage→03Normal [19:23:30] (03Merged) 10jenkins-bot: Enable local uploads on inh.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/555621 (https://phabricator.wikimedia.org/T239925) (owner: 10Ammarpad) [19:23:42] 10Operations, 10Wikimedia-Mailing-lists: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Dzahn) p:05Triage→03Normal a:03Dzahn [19:23:43] (03CR) 10Jforrester: "I'd be sad to lose this. Peter's call, though." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/561649 (owner: 10Bartosz Dziewoński) [19:25:42] !log urbanecm@deploy1001 Synchronized dblists/commonsuploads.dblist: SWAT: 0f045c3: Enable local uploads on inh.wiki (T239925) (duration: 00m 54s) [19:25:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:25:45] T239925: Non-free content on Ingush Wikipedia - https://phabricator.wikimedia.org/T239925 [19:25:57] Zoranzoki21: the local upload synced too [19:26:00] !log Morning SWAT done [19:26:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:26:04] Oh I wanted just to ask when it will be on mwdebug, but super.. Thanks! [19:26:12] We done great job as always :) [19:28:29] Bye [19:28:31] * Zoranzoki21 waves [19:31:53] 10Operations, 10vm-requests: eqiad/codfw: 2 VM request for URL downloaders - https://phabricator.wikimedia.org/T241979 (10MoritzMuehlenhoff) a:05Dzahn→03MoritzMuehlenhoff @Dzahn I'm already on it via https://phabricator.wikimedia.org/T224551 [19:32:30] !log ebernhardson@deploy1001 Started deploy [search/mjolnir/deploy@41a22b8]: Bump to latest master [19:32:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:32:36] (03PS1) 10RobH: ganeti500* setup info [puppet] - 10https://gerrit.wikimedia.org/r/562339 (https://phabricator.wikimedia.org/T228099) [19:32:57] (03CR) 10RobH: [C: 03+2] ganeti500* setup info [puppet] - 10https://gerrit.wikimedia.org/r/562339 (https://phabricator.wikimedia.org/T228099) (owner: 10RobH) [19:33:42] 10Operations, 10vm-requests: eqiad/codfw: 2 VM request for URL downloaders - https://phabricator.wikimedia.org/T241979 (10Dzahn) @MoritzMuehlenhoff Alright, i thought i'd take it as part of clinic duty (just creating the VM with OS). Let me know if it would help or take it back. [19:35:43] 10Operations, 10LDAP-Access-Requests: Allow LDAP access to superset dashboards for Moushira Elamrawy - https://phabricator.wikimedia.org/T242000 (10Dzahn) p:05Triage→03Normal a:03Dzahn [19:36:53] 10Operations, 10ops-eqsin: rack/setup/install ganeti500[123].eqsin.wmnet - https://phabricator.wikimedia.org/T228099 (10RobH) [19:39:25] 10Operations, 10ops-eqiad, 10Core Platform Team: (No Need By Date) rack/setup/install restbase1029, restbase1029, restbase1030 - https://phabricator.wikimedia.org/T241784 (10RobH) [19:39:28] !log ebernhardson@deploy1001 Finished deploy [search/mjolnir/deploy@41a22b8]: Bump to latest master (duration: 06m 57s) [19:39:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:39:33] 10Operations, 10ops-codfw, 10Core Platform Team: (No Need By Date Provided) rack/setup/install restbase202[123] - https://phabricator.wikimedia.org/T241790 (10RobH) [19:40:25] 10Operations, 10LDAP-Access-Requests: Allow LDAP access to superset dashboards for Moushira Elamrawy - https://phabricator.wikimedia.org/T242000 (10Dzahn) Hi @Moushira could you please try if you can login on the [[ https://wikitech.wikimedia.org | Wikitech wiki ]] with the existing `Moushirael`user? If that... [19:42:28] 10Operations, 10Gerrit, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10thcipriani) Looks like we have a few options available: * Use some spare hardware that is already available * Setup a ganeti VM (also fine from my perspective) Adding @herron to unstick us... [19:43:34] 10Operations, 10Traffic, 10HTTPS, 10Voice & Tone: sec-warning page uses the term "Wikipedia" incorrectly - https://phabricator.wikimedia.org/T241656 (10Dzahn) p:05Triage→03Normal [19:45:13] PROBLEM - Prometheus jobs reduced availability on icinga1001 is CRITICAL: job=mjolnir site=codfw https://wikitech.wikimedia.org/wiki/Prometheus%23Prometheus_job_unavailable https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets [19:51:15] (03CR) 10Gehel: [C: 03+2] airflow: Remove PrivateTmp from airflow-kerberos unit [puppet] - 10https://gerrit.wikimedia.org/r/562329 (owner: 10EBernhardson) [19:52:02] 10Operations, 10Design-Research, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Dzahn) Hi @Dendelele In general if you want WMF to buy a domain name please you would first contact @CRoslof at legal to actually buy it and then SRE to add it to DNS.... [19:55:26] 10Operations, 10Design-Research, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Dzahn) @Dendele Please ask OIT to get you office wiki access and a contractor email address. They will help you with that. When it comes to getting the domain name you'l... [19:57:15] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: NDA for Superset Request from WMDE Employee - Kris Litson - https://phabricator.wikimedia.org/T241722 (10Dzahn) a:05Franziska_Heine→03Dzahn [19:58:32] (03CR) 10Dzahn: [C: 03+2] "has approval now" [puppet] - 10https://gerrit.wikimedia.org/r/561899 (https://phabricator.wikimedia.org/T241722) (owner: 10Dzahn) [20:02:51] !log LDAP - added 'krli' (Kris Litson) to 'wmde' and 'nda' for superset access (T241722) [20:02:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:02:54] T241722: NDA for Superset Request from WMDE Employee - Kris Litson - https://phabricator.wikimedia.org/T241722 [20:03:17] (03PS1) 10EBernhardson: airflow: Share kerberos cache via /var/run [puppet] - 10https://gerrit.wikimedia.org/r/562349 [20:04:23] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: NDA for Superset Request from WMDE Employee - Kris Litson - https://phabricator.wikimedia.org/T241722 (10Dzahn) 05Open→03Resolved Thanks @Franziska_Heine ! @Kris_Litson_WMDE You have been added to "wmde" and "nda", the same groups that Raja has.... [20:06:00] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10Dzahn) a:03Dzahn [20:06:02] (03CR) 10Gehel: [C: 03+2] airflow: Share kerberos cache via /var/run [puppet] - 10https://gerrit.wikimedia.org/r/562349 (owner: 10EBernhardson) [20:08:31] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10Dzahn) @RStallman-legalteam Kai has signed an NDA with WMF Legal back in September 2017 (see T168046#3591921). Can i assume that is still valid? [20:13:27] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10Dzahn) Hi, @kai.nissen i'll handle this, i'm on clinic duty this week. +@Nuria @elukey for analytics data requests Does "view EventLogging data for analyzing WMDE's fundrais... [20:14:45] 10Operations, 10Research, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to analytics-privatedata-users and researchers for Aroraakhil - https://phabricator.wikimedia.org/T241096 (10Dzahn) a:05Nuria→03Dzahn [20:15:00] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10Ottomata) > Does "view EventLogging data for analyzing WMDE's fundraising banner campaigns" require shell access today as it did in 2017? I'd assume so! Since we've we no long... [20:15:43] 10Operations, 10Research, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to analytics-privatedata-users and researchers for Aroraakhil - https://phabricator.wikimedia.org/T241096 (10Dzahn) Taking over as the one on rotating clinic duty. Will amend the expiration date to the changeset and foll... [20:15:52] 10Operations, 10Wikimedia-Incident: Investigate all-site outage on 2019-12-30 (HTTP 504 error) - https://phabricator.wikimedia.org/T241573 (10Krinkle) [20:16:43] (03PS1) 10EBernhardson: Add airflow-kerberos unit to airflow mgmt privs [puppet] - 10https://gerrit.wikimedia.org/r/562354 [20:16:44] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10Dzahn) Thanks for the clarification @Ottomata , alright! [20:17:19] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10Dzahn) [20:19:30] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10Dzahn) Adjusted requested groups. So: NOT `researchers`, `analytics` but instead ONLY analytics-privatedata. @kai.nissen See above, you'll need to use Hive nowadays instea... [20:24:45] (03PS3) 10Dzahn: admin: Add production access to Aroraakhil, including private data [puppet] - 10https://gerrit.wikimedia.org/r/560604 (https://phabricator.wikimedia.org/T241096) (owner: 10Jcrespo) [20:25:39] (03PS2) 10EBernhardson: Add airflow-kerberos unit to airflow mgmt privs [puppet] - 10https://gerrit.wikimedia.org/r/562354 [20:25:41] (03PS1) 10EBernhardson: [airflow] Move PrivateTmp to correct section of unit [puppet] - 10https://gerrit.wikimedia.org/r/562358 [20:26:34] (03PS1) 10Dbarratt: Enable Partial Blocks on every wiki excluding those that have opted-out [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562359 (https://phabricator.wikimedia.org/T218626) [20:27:26] (03CR) 10Jforrester: [C: 03+1] install_server: switch contint2001 from jessie to buster [puppet] - 10https://gerrit.wikimedia.org/r/560365 (https://phabricator.wikimedia.org/T224591) (owner: 10Dzahn) [20:37:31] 10Operations, 10vm-requests: eqiad/codfw: 2 VM request for URL downloaders - https://phabricator.wikimedia.org/T241979 (10MoritzMuehlenhoff) >>! In T241979#5779246, @Dzahn wrote: > @MoritzMuehlenhoff Alright, i thought i'd take it as part of clinic duty (just creating the VM with OS). Let me know if it would h... [20:37:47] (03CR) 10DannyS712: [C: 03+1] "Does what it says" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562359 (https://phabricator.wikimedia.org/T218626) (owner: 10Dbarratt) [20:40:32] (03CR) 10Dzahn: [C: 03+2] "going ahead, as Jaime already pointed out on ticket this was just missing approval and then we got that and then it was just missing the e" [puppet] - 10https://gerrit.wikimedia.org/r/560604 (https://phabricator.wikimedia.org/T241096) (owner: 10Jcrespo) [20:40:47] (03PS4) 10Dzahn: admin: Add production access to Aroraakhil, including private data [puppet] - 10https://gerrit.wikimedia.org/r/560604 (https://phabricator.wikimedia.org/T241096) (owner: 10Jcrespo) [20:42:04] (03CR) 10Dzahn: [C: 03+2] "UID confirmed, key matches ticket, email confirmed" [puppet] - 10https://gerrit.wikimedia.org/r/560604 (https://phabricator.wikimedia.org/T241096) (owner: 10Jcrespo) [20:45:49] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10RStallman-legalteam) Yes, the NDA on file will suffice. Thanks! [20:47:01] (03CR) 10Tchanders: [C: 03+1] "Change matches the commit message... We should update the task description to be clearer that enwiki and commonswiki are the exceptions" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/562359 (https://phabricator.wikimedia.org/T218626) (owner: 10Dbarratt) [20:52:42] 10Operations, 10Research, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to analytics-privatedata-users and researchers for Aroraakhil - https://phabricator.wikimedia.org/T241096 (10Dzahn) 05Open→03Resolved @Aroraakhil Hi, your shell access has been granted and you should be able to login... [20:54:01] (03PS2) 10Legoktm: [WIP] Add html webservice type [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561753 (https://phabricator.wikimedia.org/T241817) [20:56:16] (03PS1) 10Legoktm: [WIP] Switch d/changelog to use jessie [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/562362 [21:00:04] cscott, arlolra, subbu, halfak, and accraze: Time to snap out of that daydream and deploy Services – Graphoid / Parsoid / Citoid / ORES. Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200106T2100). [21:00:17] (03PS2) 10Legoktm: Switch d/changelog to use jessie [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/562362 [21:02:49] (03PS1) 10Paladox: Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/562363 [21:04:03] (03CR) 10jerkins-bot: [V: 04-1] Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/562363 (owner: 10Paladox) [21:04:57] (03CR) 10Bstorm: [C: 03+2] Switch d/changelog to use jessie [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/562362 (owner: 10Legoktm) [21:09:07] PROBLEM - Postgres Replication Lag on maps1002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 164022968 and 28 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:09:25] (03PS1) 10RobH: ganeti500[123] production dns entries [dns] - 10https://gerrit.wikimedia.org/r/562366 (https://phabricator.wikimedia.org/T228099) [21:10:55] RECOVERY - Postgres Replication Lag on maps1002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 68200 and 43 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [21:11:55] (03CR) 10Nuria: "Looks good." [puppet] - 10https://gerrit.wikimedia.org/r/562322 (https://phabricator.wikimedia.org/T242015) (owner: 10Joal) [21:13:54] 10Operations, 10Design-Research, 10Domains, 10Traffic: Register a domain and redirect URL - https://phabricator.wikimedia.org/T241944 (10Dzahn) p:05Triage→03Normal [21:14:41] 10Operations, 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Scap, 10serviceops: On beta, scap can't clear opcache on some mw servers - https://phabricator.wikimedia.org/T237033 (10Dzahn) p:05Triage→03Normal [21:16:17] (03CR) 10RobH: [C: 03+2] ganeti500[123] production dns entries [dns] - 10https://gerrit.wikimedia.org/r/562366 (https://phabricator.wikimedia.org/T228099) (owner: 10RobH) [21:19:38] 10Operations, 10ops-eqsin: rack/setup/install ganeti500[123].eqsin.wmnet - https://phabricator.wikimedia.org/T228099 (10RobH) [21:20:19] 10Operations, 10netops: Network issues reaching phabricator on IPv6 (Comcast/Portland OR) - https://phabricator.wikimedia.org/T240488 (10Dzahn) The packet loss seems to be at hops in the Comcast network and not on our end afaict. @brion Is it safe to assume we are talking about https as protocol (since phabr... [21:23:26] 10Operations, 10netops: Network issues reaching phabricator on IPv6 (Comcast/Portland OR) - https://phabricator.wikimedia.org/T240488 (10brion) 05Open→03Resolved a:03brion Definitely saw it on https, on phab. The ipv6 timeout behavior makes it hard to see reliably so not sure if it was hitting other site... [21:27:44] (03PS2) 10Dzahn: install_server: switch contint2001 from jessie to buster [puppet] - 10https://gerrit.wikimedia.org/r/560365 (https://phabricator.wikimedia.org/T224591) [21:27:50] (03CR) 10Dzahn: [C: 03+2] install_server: switch contint2001 from jessie to buster [puppet] - 10https://gerrit.wikimedia.org/r/560365 (https://phabricator.wikimedia.org/T224591) (owner: 10Dzahn) [21:28:08] !log starting rebuild of holes in new term store from Q1Mio to Q10Mio using screen in mwmaint1002 (T219123) [21:28:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:28:12] T219123: Migrate to and read from new store for item terms - https://phabricator.wikimedia.org/T219123 [21:34:12] I'm done for the day, let me know if there's any issue [21:49:19] (03CR) 10BryanDavis: [C: 04-1] "-1 for the new grid queue (comment inline)" (032 comments) [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561753 (https://phabricator.wikimedia.org/T241817) (owner: 10Legoktm) [21:52:24] (03CR) 10BryanDavis: [C: 04-1] [WIP] Add html webservice type (031 comment) [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561753 (https://phabricator.wikimedia.org/T241817) (owner: 10Legoktm) [22:00:04] Reedy and sbassett: (Dis)respected human, time to deploy Weekly Security deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200106T2200). Please do the needful. [22:01:20] 10Operations, 10SRE-Access-Requests: Requesting access to EventLogging data for knissen - https://phabricator.wikimedia.org/T241838 (10Nuria) @kai.nissen Have you ever used hadoop/hive before? The dat is no longer in a MySQL data store. [22:06:54] 10Operations, 10Performance-Team, 10Traffic, 10Patch-For-Review: 15% response start regression as of 2019-11-11 (Varnish->ATS) - https://phabricator.wikimedia.org/T238494 (10Krinkle) [22:09:36] 10Operations, 10Traffic, 10Patch-For-Review, 10Performance-Team (Radar): Improve ATS backend connection reuse against origin servers - https://phabricator.wikimedia.org/T241145 (10Gilles) [22:24:19] 10Operations, 10ops-eqsin: rack/setup/install ganeti500[123].eqsin.wmnet - https://phabricator.wikimedia.org/T228099 (10RobH) [22:34:51] 10Operations, 10Wikimedia-Mailing-lists: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Exec8) It will be ok. The old list will then must be deactivated to avoid public confusion having two mailing lists. If name "WikimediaMY" can't be reused, "W... [22:39:59] mutante: Next step for T224591 is to re-image contint2001? [22:39:59] T224591: Migrate contint* hosts to Buster - https://phabricator.wikimedia.org/T224591 [22:43:22] James_F: well.. the first thought is that but the second thought is "first use the role on cloud VPS on buster and remove puppet errors early" [22:43:37] Ha. [22:43:45] not fully sure. the price is either how long does it take to fix in codfw [22:43:45] Yes, not-erroring sounds good. [22:43:53] and how high is the price for setting up cloud VPS [22:43:53] * James_F nods. [22:44:33] some stuff..like apache2/php version i already know will break like on all the others [22:45:08] but having the VPS is nicer... i can use the new "devtools" project [22:45:19] since it'll be a throw-away VPS [22:45:53] and then probably there will be a (temp) blocker "some package is missing in buster" [22:47:05] then package building and adding apt-repo via puppet [22:48:08] (03PS1) 10Bstorm: wmcs: stop the rpcbind service [puppet] - 10https://gerrit.wikimedia.org/r/562377 (https://phabricator.wikimedia.org/T241710) [22:49:23] (03PS2) 10Bstorm: wmcs: stop the rpcbind service [puppet] - 10https://gerrit.wikimedia.org/r/562377 (https://phabricator.wikimedia.org/T241710) [22:54:52] (03CR) 10Bstorm: "This compiles as a noop, but that's partly because I don't think a custom fact that isn't available to the master yet will work? I can te" [puppet] - 10https://gerrit.wikimedia.org/r/562377 (https://phabricator.wikimedia.org/T241710) (owner: 10Bstorm) [22:57:21] 10Operations, 10Wikimedia-Mailing-lists: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Dzahn) Alright. If you look at https://lists.wikimedia.org/mailman/listinfo and scroll down you'll see the standard (which hasn't always existed) moved to W... [23:00:16] 10Operations, 10Wikimedia-Mailing-lists: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Dzahn) regarding archives: There are literally 2 messages. And only one of them was written by a human and never got a reply. https://lists.wikimedia.org/pip... [23:04:23] 10Operations, 10Mail: Add security-team@wikimedia.org as recipient any abuse@ emails - https://phabricator.wikimedia.org/T242049 (10Dsharpe) [23:11:48] 10Operations, 10Mail: Add security-team@wikimedia.org as recipient of any abuse@ emails - https://phabricator.wikimedia.org/T242049 (10Dsharpe) [23:12:27] !log mailman - running /usr/local/sbin/rename_list wikimediamy wikimedia-my (T241988) [23:12:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:12:30] T241988: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 [23:19:35] (03PS1) 10Dzahn: exim: add listserver_alias for renamed list wikimediamy [puppet] - 10https://gerrit.wikimedia.org/r/562384 (https://phabricator.wikimedia.org/T241988) [23:20:42] (03PS2) 10Dzahn: exim: add listserver_alias for renamed list wikimediamy [puppet] - 10https://gerrit.wikimedia.org/r/562384 (https://phabricator.wikimedia.org/T241988) [23:20:45] (03PS1) 10CDanis: fastnetmon: include hostnames (not just IPs) in emails [puppet] - 10https://gerrit.wikimedia.org/r/562385 [23:21:06] (03CR) 10Dzahn: [C: 03+2] "the standard naming for chapter/ug lists is with the hyphen" [puppet] - 10https://gerrit.wikimedia.org/r/562384 (https://phabricator.wikimedia.org/T241988) (owner: 10Dzahn) [23:26:02] (03PS1) 10Dzahn: mailman: add http redirect for renamed listinfo page of wikimediamy [puppet] - 10https://gerrit.wikimedia.org/r/562386 (https://phabricator.wikimedia.org/T241988) [23:27:20] (03CR) 10Dzahn: [C: 03+2] mailman: add http redirect for renamed listinfo page of wikimediamy [puppet] - 10https://gerrit.wikimedia.org/r/562386 (https://phabricator.wikimedia.org/T241988) (owner: 10Dzahn) [23:33:38] 10Operations, 10netops: fastnetmon fired for routine text-lb.esams traffic - https://phabricator.wikimedia.org/T241281 (10CDanis) 05Open→03Resolved [23:34:10] 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Dzahn) Hi again @Exec8, @Taufik, despite what i said earlier i did a proper rename. - wikimediamy has been renamed to wikimedia-my - t... [23:35:27] 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Rename and transfer admin of Malaysia Mailing List (wikimediamy) - https://phabricator.wikimedia.org/T241988 (10Dzahn) 05Open→03Resolved [23:38:38] (03PS1) 10CDanis: fastnetmon: add UDP/ICMP bw limits, greatly increase pps limits [puppet] - 10https://gerrit.wikimedia.org/r/562387 [23:40:16] (03CR) 10CDanis: "These are somewhat arbitrary -- and I could see increasing the udp/icmp pps figures by another order of magnitude or two -- but looking at" [puppet] - 10https://gerrit.wikimedia.org/r/562387 (owner: 10CDanis) [23:41:25] (03CR) 10Bstorm: "Yes, this does the thing in toolsbeta when patched in:" [puppet] - 10https://gerrit.wikimedia.org/r/562377 (https://phabricator.wikimedia.org/T241710) (owner: 10Bstorm) [23:48:34] 10Operations, 10Mail: Add security-team@wikimedia.org as recipient of any abuse@ emails - https://phabricator.wikimedia.org/T242049 (10Dzahn) The list of domains for which exim aliases files (like the i edited for wikimedia.org to add security@ previously) exist currently is: benefactors.wikimedia.org pr.wiki...