[00:49:42] 06Traffic, 06DC-Ops, 10ops-esams, 10ops-magru, 13Patch-For-Review: CPU temperature issues in cp hosts - https://phabricator.wikimedia.org/T373993#11002922 (10BCornwall) [00:51:02] 06Traffic: Unify CDN ats/haproxy/varnish upgrade cookbooks - https://phabricator.wikimedia.org/T390094#11002923 (10BCornwall) 05In progress→03Resolved We decided on keeping the cookbooks separate and merged this in with https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1129882 which, in retrospect,... [00:51:57] 06Traffic, 07Browser-Support-Apple-Safari, 07Browser-Support-Firefox, 07Browser-Support-Google-Chrome, 07User-notice: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#11002925 (10BCornwall) a:05BCornwall→03None [00:57:52] 06Traffic, 07PM: Create useful Phabricator task workflow for Traffic - https://phabricator.wikimedia.org/T370557#11002929 (10BCornwall) 05Stalled→03In progress [01:24:43] 06Traffic, 10MediaWiki-Core-AuthManager, 06MediaWiki-Platform-Team, 05FY2025-26 KR 5.1: Decide how to expose session information outside of MediaWiki - https://phabricator.wikimedia.org/T394012#11002960 (10Tgr) >>! In T394012#10980179, @Tgr wrote: > Anyway, we do have auth.wikimedia.org now, but could just... [01:25:21] 06Traffic, 10MediaWiki-Core-AuthManager, 06MediaWiki-Platform-Team, 05FY2025-26 KR 5.1: Decide how to expose session information outside of MediaWiki - https://phabricator.wikimedia.org/T394012#11002961 (10Tgr) 05Open→03Resolved [05:54:13] 06Traffic: Googlebot Commons 429 throttling - https://phabricator.wikimedia.org/T398668#11003105 (10tstarling) 05Open→03Resolved Thanks @Joe. Looks good in Turnilo. [07:52:22] 06Traffic, 06MediaWiki-Platform-Team, 06MW-Interfaces-Team, 06serviceops, and 3 others: API Rate Limiting Architecture - https://phabricator.wikimedia.org/T399291#11003297 (10daniel) [10:11:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Netbox: remove old cr2-codfw Switch Control Board inventory items - https://phabricator.wikimedia.org/T398940#11003801 (10ayounsi) Netbox is unfortunately not made to track inventory items (as in on a shelf). There are some plugins tha... [12:57:03] 10netops, 06Infrastructure-Foundations, 07sre-alert-triage: Alert in need of triage: BGP status (instance cr2-drmrs) - https://phabricator.wikimedia.org/T393991#11004304 (10ayounsi) 05Open→03Resolved [13:11:22] 06Traffic, 06DC-Ops, 10ops-codfw, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11004337 (10elukey) Just sent the email to Willy explaining the issue, fingers crossed to get some help from Dell :) [13:27:17] Hey folks, really sorry but we are still stuck with the new cp nodes, hope that the Dell's tech support will unblock the situation [13:29:23] I know... thanks for taking care of this elukey! [15:44:13] 06Traffic: Improve runbooks for OCSP-related alerts - https://phabricator.wikimedia.org/T292397#11005103 (10Vgutierrez) 05Open→03Declined OCSP has been deprecated [15:48:06] 06Traffic: 429 Error from cp5022 when accessing Wikimedia project - https://phabricator.wikimedia.org/T397804#11005124 (10BCornwall) 05Open→03Declined As there hasn't been any update in over a week, I'm closing this. If more information can be provided, please re-open. Thanks! [15:48:53] 06Traffic: 429 Error from cp5022 when accessing Wikimedia project - https://phabricator.wikimedia.org/T397804#11005131 (10BCornwall) 05Declined→03Invalid [15:55:24] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11005152 (10BCornwall) 05Resolved→03Open Ah, @VRiley-WMF, it seems that connectivity is no longer through the Mellanox card: ` [ 9.128067] mlx5_core 0000:3b:00.0: Port module event: module 0... [15:55:55] 06Traffic: Write a cookbook that performs a rolling restart of HAProxy - https://phabricator.wikimedia.org/T375232#11005158 (10ssingh) 05Open→03Resolved Done in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1167222. [16:27:28] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11005406 (10VRiley-WMF) Orginally put the cable into the onboard port. Once it was able to reimage, that's when I just moved it over. It should be all set now. [16:33:53] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11005483 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs1017.eqiad.wmnet with OS bookworm [17:34:26] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11005847 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host lvs1017.eqiad.wmnet with OS bookworm [17:49:38] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11005905 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs1017.eqiad.wmnet with OS bookworm executed with errors: - lvs1017 (**FAIL**) - Downtimed... [18:12:06] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11005979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host lvs1017.eqiad.wmnet with OS bookworm completed: - lvs1017 (**PASS**) - Removed from Puppet... [18:16:35] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 - https://phabricator.wikimedia.org/T387145#11005982 (10BCornwall) 05Open→03Resolved The link was re-connected to the Mellanox card; We then reconfigured the interface with: ` $ sudo -i cookbook sre.dns.netbox -t T387145 'update lvs1... [20:33:19] 06Traffic, 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team: beta cluster: profile::cache::varnish::frontend needs to reload varnish-frontend.service when /etc/varnish/blocked-nets.inc.vcl changes - https://phabricator.wikimedia.org/T358887#11006592 (10bd808) I had a user report today that the block...