[06:16:29] 10serviceops, 10Operations, 10Patch-For-Review, 10User-Elukey: Reimage one memcached shard to Buster - https://phabricator.wikimedia.org/T252391 (10elukey) Note after checking slab distribution on the gutter pool. The last slab sizes seem to not follow the prediction made by the script: ` STAT 49:chunk_si... [09:27:55] 10serviceops, 10Operations, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10MoritzMuehlenhoff) Since we don't use helm3 in production yet, just adding it to the task. There's a new vulnerability specific to 3.x, so let's upgrade to 3.2.4 before we roll th... [12:45:51] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install kubernetes20[07-14].codfw.wmnet and kubestage200[1-2].codfw.wmnet. - https://phabricator.wikimedia.org/T252185 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by akosiaris on cumin2001.codfw.wmnet for... [13:08:26] 10serviceops, 10Operations, 10ops-codfw, 10Patch-For-Review: (Need by: TBD) rack/setup/install kubernetes20[07-14].codfw.wmnet and kubestage200[1-2].codfw.wmnet. - https://phabricator.wikimedia.org/T252185 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['kubernetes2013.codfw.wmnet', 'kubernetes2... [13:18:33] 10serviceops, 10CX-cxserver, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10akosiaris) [13:18:43] 10serviceops, 10CX-cxserver, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10akosiaris) p:05Triage→03Medium [13:19:24] 10serviceops, 10CX-cxserver, 10Release-Engineering-Team (Pipeline): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10akosiaris) [13:39:53] 10serviceops: Replace rdb200[34] with rdb200[78] - https://phabricator.wikimedia.org/T255250 (10akosiaris) [14:27:52] 10serviceops, 10Operations, 10Patch-For-Review: move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet) - https://phabricator.wikimedia.org/T247021 (10ops-monitoring-bot) Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 5 host(s) and their services with reason: new_install ` mw[2335... [14:29:26] 10serviceops: Replace rdb200[34] with rdb200[78] - https://phabricator.wikimedia.org/T255250 (10akosiaris) [14:52:50] 10serviceops, 10Operations, 10Patch-For-Review: move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet) - https://phabricator.wikimedia.org/T247021 (10ops-monitoring-bot) Icinga downtime for 1:00:00 set by dzahn@cumin1001 on 5 host(s) and their services with reason: new_install ` mw[2335... [15:24:25] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Dzahn) >>! In T241852#6211870, @Papaul wrote: > @Dzahn the 5 servers in C3 are ready for services They are now in production. (details in T247021) [15:25:02] 10serviceops, 10Operations: move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet) - https://phabricator.wikimedia.org/T247021 (10Dzahn) mw2335 through mw2339 in rack C3 have also been taken into production now. This should complete the ticket. All hosts from mw2291 through mw2376 are po... [15:30:09] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Dzahn) @Papaul This ticket talks about mw2377 but it seems to me we never had a host mw2377 (only up to mw2376). Can you confirm that? [15:32:51] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Dzahn) 76 servers are pooled as appservers. 10 have been used for kubernetes. Adds up to 86. [15:33:12] 10serviceops, 10Operations: move all 86 new codfw appservers into production (mw2[291-2377].codfw.wmnet) - https://phabricator.wikimedia.org/T247021 (10Dzahn) 05Open→03Resolved 76 servers are pooled as appservers. 10 have been used for kubernetes. Adds up to 86. [15:33:15] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Dzahn) [17:36:29] 10serviceops, 10Operations, 10ops-codfw: (Need by: TBD) rack/setup/install 86 new codfw mw systems - https://phabricator.wikimedia.org/T241852 (10Papaul) @Dzahn confirm we have up to mw2376 [18:57:54] 10serviceops, 10Operations, 10Release-Engineering-Team, 10Developer Productivity, and 2 others: All debug hosts give (likely spurious) message: PHP Fatal error: The UdpSocket to 127.0.0.1:10514 has been closed (from Monolog/SyslogUdp) - https://phabricator.wikimedia.org/T214734 (10Krinkle) Debug-only, not...