[07:57:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [08:02:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2014 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [09:12:21] 10netops, 06Infrastructure-Foundations, 06SRE: Audit and verify all cloudcephosd have their primary interface tagged and access to cloud-storage vlan - https://phabricator.wikimedia.org/T409690 (10fgiunchedi) 03NEW [09:15:03] 10netops, 06Infrastructure-Foundations, 06SRE: Cloudcephosd: migrate to single network uplink - https://phabricator.wikimedia.org/T399180#11357477 (10fgiunchedi) >>! In T399180#11310972, @cmooney wrote: >>>! In T399180#11310845, @fgiunchedi wrote: >> @taavi @Andrew @cmooney what do you think of the above? >... [09:28:46] 06Traffic, 06MediaWiki-Platform-Team, 07Upstream: Telegram previews broken since unified mobile routing - https://phabricator.wikimedia.org/T409575#11357515 (10Vgutierrez) a quick check using `https://en.wikipedia.org/wiki/Main_Page?vgutierrez=tg` resulted in telegram bot visiting `https://en.m.wikipedia.org... [11:14:15] 06Traffic, 06MediaWiki-Platform-Team, 06serviceops, 07OKR-Work: api-gateway helm chart: rest routes should return retry-after when a rate limit applies. - https://phabricator.wikimedia.org/T405636#11357980 (10Clement_Goubert) After thinking about it a bit, I tend to agree with @pmiazga Adding #traffic for... [12:09:25] 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11358221 (10Sfaci) > I think we should have some analytics instrumentation. At minimum we should h... [13:47:49] 10netops, 06Infrastructure-Foundations, 06SRE: lsw1-d6-eqiad reboot failed, stuck in UEFI shell - https://phabricator.wikimedia.org/T409731 (10cmooney) 03NEW p:05Triage→03High [13:53:28] 10netops, 06Infrastructure-Foundations, 06SRE: Audit and verify all cloudcephosd have their primary interface tagged and access to cloud-storage vlan - https://phabricator.wikimedia.org/T409690#11358656 (10cmooney) I can take a look at this unless there is another plan? [14:20:12] FYI, I just disabled exception of default thumbnail size in nlwiki. It'll take a month to propagate through various layers of caching but that should significantly improve caching efficiency of upload cluster in esams [14:20:51] Amir1: is nlwiki that relevant? [14:21:43] it's a rather large wiki in terms of reads but also the problem is that they were duplicating all default images so the impact was way out of proportion [14:22:22] not halving the efficiency but decent amount [14:41:59] 10netops, 06Infrastructure-Foundations, 06SRE: Audit and verify all cloudcephosd have their primary interface tagged and access to cloud-storage vlan - https://phabricator.wikimedia.org/T409690#11358897 (10fgiunchedi) Yes please @cmooney, much appreciated! Note that this is currently not a blocker / not high... [14:49:09] 06Traffic, 06SRE: Meta query about why we map 31.13.103.0/24 to US - https://phabricator.wikimedia.org/T409735 (10cmooney) 03NEW p:05Triage→03Medium [15:40:26] 10netops, 06Infrastructure-Foundations, 06SRE: Audit and verify all cloudcephosd have their primary interface tagged and access to cloud-storage vlan - https://phabricator.wikimedia.org/T409690#11359233 (10LSobanski) p:05Triage→03Medium [15:41:06] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10Toolforge: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309#11359235 (10cmooney) p:05Triage→03Medium [15:42:26] 06Traffic, 06MediaWiki-Platform-Team (Radar), 07Upstream: Telegram previews broken since unified mobile routing - https://phabricator.wikimedia.org/T409575#11359246 (10OWresch-WMF) [15:46:31] 10netops, 06Infrastructure-Foundations, 06SRE: Sporadic RST drops in the ulogd logs - https://phabricator.wikimedia.org/T238823#11359271 (10LSobanski) 05Open→03Resolved a:03LSobanski Resolving as part of backlog review. There have been changes to the network and Puppet since the creation of this ta... [16:31:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: lsw1-d6-eqiad reboot failed, stuck in UEFI shell - https://phabricator.wikimedia.org/T409731#11359494 (10Jclark-ctr) [17:25:36] 06Traffic, 06Security-Team, 10WMF-General-or-Unknown, 07ContentSecurityPolicy, 13Patch-For-Review: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618#11359861 (10sbassett) >>! In T117618#11224896, @ssingh wrote: > Commenting from Traffic's side: this is in some ways,... [18:54:38] 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11360329 (10Sfaci) [19:01:42] 06Traffic, 06SRE: Meta query about why we map 31.13.103.0/24 to US - https://phabricator.wikimedia.org/T409735#11360362 (10ssingh) Thanks for filing this task @cmooney! The geofeed link above is very helpful. So it seems from the above (57.141.8.0/24, 57.141.8.0/24), we are missing the entries in the geo-maps... [19:25:17] 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11360445 (10Sfaci) [19:26:07] 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11360450 (10Sfaci) [20:28:30] 06Traffic, 06Security-Team, 10WMF-General-or-Unknown, 07ContentSecurityPolicy, 13Patch-For-Review: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618#11360663 (10ssingh) >>! In T117618#11359861, @sbassett wrote: >>>! In T117618#11224896, @ssingh wrote: >> Commenting f... [21:44:30] 06Traffic, 10Hiddenparma, 13Patch-For-Review: Introduce known-client identity objects and integrate with requestctl - https://phabricator.wikimedia.org/T403220#11361007 (10Scott_French) [23:42:12] 06Traffic, 10MobileFrontend (Tracking): All Wikimedia projects provides mobile frontend on Huawei Laptops running HarmonyOS - https://phabricator.wikimedia.org/T408567#11361520 (10Jdlrobson-WMF) [23:45:19] 06Traffic, 07Essential-Work, 06Experimentation Lab (Experiment Platform Sprint 14), 13Patch-For-Review: Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11361528 (10Sfaci) The change above creates the experiment-specific instrume...