[00:00:08] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552156 (10ssingh) Thank for reporting. Can you please try again from your home connection (without VPN) and let us know if that works? We depooled... [00:01:28] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552158 (10AlexisJazz) Works now, thanks! [00:02:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp3070:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=esams&var-instance=cp3070 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:05:00] RESOLVED: [32x] PurgedHighEventLag: High event process lag with purged on cp3066:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [00:05:31] yeah that's fine [00:07:36] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552161 (10AlexisJazz) {F71607077} There was a visible dip in editing and surge in error responses. [00:17:40] RESOLVED: VarnishHighThreadCount: Varnish's thread count on cp3070:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=esams&var-instance=cp3070 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [01:19:27] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552172 (10ssingh) We had a transient link failure between eqiad and esams that resulted in this issue. It should be resolved now, and esams is poo... [01:33:42] 10netops, 06Traffic, 06DC-Ops, 06Infrastructure-Foundations, and 3 others: ESAMS 502 broken pipe connection issues - https://phabricator.wikimedia.org/T415473#11552173 (10Peachey88) [10:42:22] 06Traffic, 06SRE, 07Documentation: Documentation error about TLS 1.2 on Wikimedia DNS DoH on metawiki - https://phabricator.wikimedia.org/T415449#11553069 (10ssingh) Yes, thanks for doing that @Naruse_shiroha. That was a typo, and well, a big one at that. For DoH (and DoT before that), the rationale for... [10:54:38] 06Traffic, 06SRE, 07Documentation: Documentation error about TLS 1.2 on Wikimedia DNS DoH on metawiki - https://phabricator.wikimedia.org/T415449#11553163 (10Cuthead) > no DoH client in theory should not support anything other than 1.3 Actually yes, there's one as I've mentioned in the main post: [OkHtt... [13:13:05] 06Traffic, 06SRE, 07Documentation: Documentation error about TLS 1.2 on Wikimedia DNS DoH on metawiki - https://phabricator.wikimedia.org/T415449#11553611 (10Naruse_shiroha) > no DoH client in theory should not support anything other than 1.3 OkHttp DoH client may not support 1.3, too depending on the p... [13:43:24] 06Traffic, 06Wikipedia-Android-App-Backlog, 10Wikisource Reader App: Integrate Wikimedia DNS into Android apps - https://phabricator.wikimedia.org/T415531#11553737 (10ssingh) [13:44:19] 06Traffic, 10DNS, 06SRE, 06Abstract Wikipedia team (26Q3 (Jan–Mar)), and 2 others: Set up DNS for abstract.wikipedia.org to be recognised - https://phabricator.wikimedia.org/T411724#11553739 (10Dzahn) `abstract.wikipedia.org` has been added to DNS [13:45:10] 06Traffic, 10DNS, 06SRE, 06Abstract Wikipedia team (26Q3 (Jan–Mar)), and 2 others: Set up DNS for abstract.wikipedia.org to be recognised - https://phabricator.wikimedia.org/T411724#11553743 (10Jdforrester-WMF) 05Open→03Resolved a:03Dzahn Thank you! [13:57:18] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#11553795 (10ssingh) Hi @RobH: thanks for the update and for pursuing this. And yeah, that works; waiting for the week after the offsite is also fine since it is just one host. Thanks! [14:14:53] 06Traffic, 06Wikipedia-Android-App-Backlog, 10Wikisource Reader App: Integrate Wikimedia DNS into Android apps - https://phabricator.wikimedia.org/T415531#11553841 (10Dbrant) Is this different from T327286? [14:44:42] 06Traffic, 06Wikipedia-Android-App-Backlog, 10Wikisource Reader App: Integrate Wikimedia DNS into Android apps - https://phabricator.wikimedia.org/T415531#11553969 (10Naruse_shiroha) > Is this different from T327286? That would require more efforts. Let me quote this... > **Does this project solve all inte... [14:49:21] 06Traffic, 06SRE: All github action tests of Pywikibot fails due to 429 status code (TOO MANY REQUESTS) - https://phabricator.wikimedia.org/T414173#11553987 (10Peter17) Hello, I am encountering the same error with my bot, based on Pywikibot... I am still unable to fetch more than a few pages from Wikipedia. T... [15:16:44] 06Traffic, 06Commons: HTTP 429 error on original image requests on Commons (iOS app by default hiding the Referrer header) - https://phabricator.wikimedia.org/T413570#11554072 (10Nylki) Btw. I released on new version of CommonsFinder that sends referer-headers with all network requests: https://github.com/nylk... [21:55:15] 06Traffic, 06SRE: High rate of broken thumbnails on Discord embeds of links to Wikimedia sites - https://phabricator.wikimedia.org/T415598#11555698 (10taavi) [21:55:40] 06Traffic, 06SRE: High rate of broken thumbnails on Discord embeds of links to Wikimedia sites - https://phabricator.wikimedia.org/T415598#11555699 (10AntiCompositeNumber) This started happening within the last month or so. It doesn't happen for every request for every thumbnail, but seems to be more likely af... [22:30:05] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11555784 (10Izno) I just fixed https://en.wikipedia.org/wiki/User:Bradv/Scripts/ExpandDiffs.js#L-20 which was previously showing a bla... [23:03:13] 06Traffic, 06SRE: High rate of broken thumbnails on Discord embeds of links to Wikimedia sites - https://phabricator.wikimedia.org/T415598#11555931 (10TheDJ) The ogp.me tag is lis... [23:03:37] 06Traffic, 06SRE: High rate of broken thumbnails on Discord embeds of links to Wikimedia sites - https://phabricator.wikimedia.org/T415598#11555944 (10TheDJ) [23:03:39] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11555945 (10TheDJ) [23:04:33] 06Traffic, 06SRE: OGP lists thumbnail version of fullsize instead the fullsize version itself - https://phabricator.wikimedia.org/T415598#11555947 (10TheDJ) [23:11:52] 06Traffic, 06SRE: OGP lists fullsize thumbnail version of original instead the original itself - https://phabricator.wikimedia.org/T415598#11555956 (10TheDJ) [23:13:18] 06Traffic, 06SRE: OGP lists fullsize thumbnail version of original instead the original itself - https://phabricator.wikimedia.org/T415598#11555961 (10AntiCompositeNumber) >>! In T415598#11555931, @TheDJ wrote: > The ogp.me tag is listing the thumbnail variant of the fullsize, instead of the fullsize I'm not...