[02:15:19] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Dzahn) 05Open→03Stalled currently blocked on T244438 , an installer issue on stretch that only happens on stretch and buster would not have a problem [06:44:42] 10serviceops, 10Operations, 10Wikimedia-Mailing-lists: Allow list admins to train spam filters - https://phabricator.wikimedia.org/T244241 (10Aklapper) Oh darrn! Thanks Reedy! I never realized that this is a custom patch in GNOME's Mailman instance, sorry! Feel free to decline if this is too much maintenanc... [08:54:01] 10serviceops, 10Services, 10Core Platform Team Workboards (Clinic Duty Team): scb2003 reports 'Internal error in changeprop' - https://phabricator.wikimedia.org/T244069 (10Joe) 05Resolved→03Open Maybe change it to use a 64 bit integer instead? [11:44:39] 10serviceops, 10Operations, 10observability: Stream a subset of mediawiki apache logs to logstash - https://phabricator.wikimedia.org/T244472 (10jijiki) [12:18:06] 10serviceops, 10Operations: Test and deploy mcrouter 0.41 - https://phabricator.wikimedia.org/T244476 (10jijiki) [12:19:50] 10serviceops, 10Operations: Test and deploy mcrouter 0.41 - https://phabricator.wikimedia.org/T244476 (10jijiki) [12:19:55] 10serviceops, 10Operations, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T240684 (10jijiki) [12:34:26] 10serviceops, 10Release-Engineering-Team-TODO, 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Test-Coverage: Add pcov PHP extension to wikimedia apt so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10Daimona) >>! In T243847#5854056, @J... [14:27:09] serviceopsen: fyi I still don't have internet at home 😓 I'll join the meeting from my phone, but I'm expecting a tech between 15-17:00 UTC, so I may miss part of the meeting depending when they show up [14:42:16] rlazarus: no worries [14:47:03] 10serviceops, 10Operations, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T240684 (10elukey) Just updated https://grafana.wikimedia.org/d/000000317/memcache-slabs adding a new row at the bottom '1.5.x metrics' with all the new m... [14:47:05] have some coffee prepared for the tech [14:47:13] we want techs on our side [14:47:58] that's a good point but I don't know if they want coffee or tea?? I'll start the kettle at least [14:56:08] lol [14:56:14] 10serviceops, 10Release Pipeline, 10Wikimedia-Portals, 10Release-Engineering-Team (Pipeline): Migrate www.wikimedia.org (the portal) to be hosted as a service - https://phabricator.wikimedia.org/T238747 (10thcipriani) [15:21:13] 10serviceops, 10Services, 10Core Platform Team Workboards (Clinic Duty Team): scb2003 reports 'Internal error in changeprop' - https://phabricator.wikimedia.org/T244069 (10Pchelolo) >>! In T244069#5855237, @Joe wrote: > Maybe change it to use a 64 bit integer instead? The HTCP protocol only allows us 32 bit... [15:55:56] 10serviceops, 10Core Platform Team Workboards (Clinic Duty Team): restrouter.svc.{eqiad,codfw}.wmnet in a failed state - https://phabricator.wikimedia.org/T242461 (10WDoranWMF) @Eevans Sorry this got lost in my inbox, yep, I agree. [17:08:38] 10serviceops, 10Operations: Test and deploy mcrouter 0.41 - https://phabricator.wikimedia.org/T244476 (10jijiki) [17:10:01] rlazarus _joe_ akosiaris mutante apergos mark can we work something a bit earlier ? [17:10:17] earlier than when? pitch us a time [17:10:20] fine by me, that's 100% a question for mutante I think [17:10:35] eg 15:00 UTC if mutante is onboard [17:11:35] <_joe_> I'm good at 6 AM UTC usually, how much earlier :D [17:12:33] yes, that's ok with me [17:12:54] cool, 1500 UTC it is [17:13:08] I could do 6 UTC am I'm usually typing by then [17:13:14] *am [17:13:39] if you want to do even earlier and decide without me present that is also ok with me [17:16:29] thank you all ! [17:17:32] mutante: since 15:00 is ok for you, we are all set [17:17:48] ok [17:49:02] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2312.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [18:11:14] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2312.codfw.wmnet'] ` and were **ALL** successful. [18:19:32] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2313.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [18:40:27] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2313.codfw.wmnet'] ` and were **ALL** successful. [18:41:41] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2314.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [19:03:40] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2314.codfw.wmnet'] ` and were **ALL** successful. [19:05:12] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2315.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [19:27:16] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2315.codfw.wmnet'] ` and were **ALL** successful. [20:48:34] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2316.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [21:10:01] 10serviceops, 10Wikifeeds: wikifeeds - fix the CPU limits so that it doesn't get starved - https://phabricator.wikimedia.org/T244535 (10Dzahn) [21:10:23] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2316.codfw.wmnet'] ` and were **ALL** successful. [21:10:43] 10serviceops, 10Wikifeeds: wikifeeds - fix the CPU limits so that it doesn't get starved - https://phabricator.wikimedia.org/T244535 (10Dzahn) [21:12:44] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2317.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [21:20:20] 10serviceops, 10Wikifeeds, 10Patch-For-Review, 10Wikimedia-Incident: wikifeeds - fix the CPU limits so that it doesn't get starved - https://phabricator.wikimedia.org/T244535 (10Dzahn) [21:34:45] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2317.codfw.wmnet'] ` and were **ALL** successful. [21:37:02] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2318.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [21:58:00] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2318.codfw.wmnet'] ` and were **ALL** successful. [21:59:00] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2319.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [22:12:22] 10serviceops, 10Services, 10Core Platform Team Workboards (Clinic Duty Team): scb2003 reports 'Internal error in changeprop' - https://phabricator.wikimedia.org/T244069 (10Pchelolo) 05Open→03Resolved @Clarakosi has fixed the underlying issue and a `htcp-purge@0.3.1` was published and will be deployed on... [22:17:24] i turned mw2163 and mw2271 into canary appservers for codfw. that means: they get mediawiki-testers shell users group, they do NOT get scap sql scripts and nginx keepalive_requests changes from 100 to 1000 [22:17:28] that is all the difference i see [22:18:21] together with existing mwdebug2* this gives us 4 canaries for codfw .. and the other day the same happened for canary API.. but there it's not an actual puppet change at all to switch roles [22:20:31] 10serviceops, 10Operations, 10Patch-For-Review: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10Dzahn) [22:23:04] 10serviceops, 10Operations, 10Patch-For-Review: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10Dzahn) mw2163 and mw2271 have been turned into canary appservers now. As opposed to canary API appservers this means actual puppet changes which are: - mediawiki-testers shell access... [22:23:15] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2320.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [22:23:38] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2319.codfw.wmnet'] ` and were **ALL** successful. [22:24:04] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2321.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [22:24:07] 10serviceops, 10Operations, 10Patch-For-Review: No mw canary servers in codfw - https://phabricator.wikimedia.org/T242606 (10Dzahn) @jijiki What do you think ? Is this good now? 4 of each type and in different rows/racks. [22:38:53] the install issues with new mw servers have been solved and papaul is on them now.. so soon! [22:46:21] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2320.codfw.wmnet'] ` and were **ALL** successful. [22:47:29] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2321.codfw.wmnet'] ` and were **ALL** successful. [22:56:33] 10serviceops: Add x-request-id to httpd (apache) logs - https://phabricator.wikimedia.org/T244545 (10Dzahn) [22:58:08] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2322.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [22:58:26] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2323.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [22:58:30] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2323.codfw.wmnet'] ` Of which those **FAILED**: ` ['mw2323.codfw.wmnet'] ` [23:02:42] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2320.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [23:03:17] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2320.codfw.wmnet'] ` Of which those **FAILED**: ` ['mw2320.codfw.wmnet'] ` [23:04:39] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2323.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [23:20:11] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2322.codfw.wmnet'] ` and were **ALL** successful. [23:20:34] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2324.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [23:26:37] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2323.codfw.wmnet'] ` and were **ALL** successful. [23:27:35] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2325.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [23:41:41] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2324.codfw.wmnet'] ` and were **ALL** successful. [23:48:35] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2325.codfw.wmnet'] ` and were **ALL** successful. [23:53:57] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2326.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [23:54:21] 10serviceops, 10Operations, 10ops-codfw: rack/setup/install new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts: ` mw2327.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020...