[08:43:02] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10997642 (10Xqt) [08:44:24] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#10997646 (10Xqt) [08:58:15] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#10997671 (10Xqt) [09:24:04] (03CR) 10Hashar: [C:03+2] dockerfiles: remove EXECUTOR_NUMBER from tox example [integration/config] - 10https://gerrit.wikimedia.org/r/1168188 (https://phabricator.wikimedia.org/T399283) (owner: 10Hashar) [09:24:25] (03CR) 10Hashar: [C:04-1] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1168190 (https://phabricator.wikimedia.org/T399283) (owner: 10Hashar) [09:26:31] (03Merged) 10jenkins-bot: dockerfiles: remove EXECUTOR_NUMBER from tox example [integration/config] - 10https://gerrit.wikimedia.org/r/1168188 (https://phabricator.wikimedia.org/T399283) (owner: 10Hashar) [09:46:16] 10Gerrit: Unable to push commit whose parent is not the latest commit for the parent patch - https://phabricator.wikimedia.org/T399241#10997755 (10hashar) It is possible but unlikely imho. I think it never worked and that looks very similar to https://issues.gerritcodereview.com/issues/40011702 That describes h... [11:12:36] 10Gerrit: Unable to push commit whose parent is not the latest commit for the parent patch - https://phabricator.wikimedia.org/T399241#10997897 (10hashar) Repro which does not edit the UI ` git clone https://gerrit.wikimedia.org/r/test/gerrit-ping cd gerrit-ping ` Create two commits and push for review, saving t... [11:17:13] 10Continuous-Integration-Config: Create a non-voting CI job to flag deprecated code - https://phabricator.wikimedia.org/T388948#10997913 (10Tacsipacsi) Adding yet another job is doable and easy, but wasteful on CI infrastructure resources. Adding support for three-state jobs would be easy on resources, but defin... [11:32:15] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#10997926 (10Xqt) 05Open→03Resolved a:03Xqt Thank you @hashar! [13:35:59] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#10998122 (10hashar) >>! In T399367#10997926, @Xqt wrote: > Thank you @hashar! You are welcome! [15:26:30] 10Gerrit: Unable to push commit whose parent is not the latest commit for the parent patch - https://phabricator.wikimedia.org/T399241#10998334 (10hashar) I have tried to reproduce by writing at test which was unexpectedly working, or at least not reproducing the issue: ` lang=java,name=javatests/com/google/gerr... [15:28:50] (03PS1) 10Hashar: Disable receive.createNewChangeForAllInTarget again [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1168621 (https://phabricator.wikimedia.org/T399241) [15:29:10] (03CR) 10Hashar: [V:03+2 C:03+2] Disable receive.createNewChangeForAllInTarget again [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1168621 (https://phabricator.wikimedia.org/T399241) (owner: 10Hashar) [15:32:25] 10Gerrit: Unable to push commit whose parent is not the latest commit for the parent patch - https://phabricator.wikimedia.org/T399241#10998344 (10hashar) a:03hashar I have confirmed the fix with the chain of commits I did on `test/gerrit-ping`. I have managed to send a patchset 2 on https://gerrit.wikimedia.... [15:32:33] nothing like hacking stuff on Sunday [15:32:43] AND finding the issue :) [15:58:20] 10Gerrit, 07Upstream: Unable to push commit whose parent is not the latest commit for the parent patch - https://phabricator.wikimedia.org/T399241#10998355 (10hashar) [15:58:50] 10Gerrit, 07Upstream: Unable to push commit whose parent is not the latest commit for the parent patch - https://phabricator.wikimedia.org/T399241#10998357 (10hashar) https://gerrit-review.googlesource.com/c/gerrit/+/489941 Test review rejectes when parent is obsolete [16:34:51] Project beta-code-update-eqiad build #556519: 04FAILURE in 4.7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/556519/ [16:45:11] Yippee, build fixed! [16:45:11] Project beta-code-update-eqiad build #556520: 09FIXED in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/556520/ [17:41:46] 10Phabricator (Upstream), 10Release-Engineering-Team (Radar), 07Developer Productivity, 07Upstream: Add Open Graph support to Phabricator Maniphest Tasks to have link preview on Telegram, Slack, and other messaging apps - https://phabricator.wikimedia.org/T288117#10998448 (10Aklapper) This task is stalled... [18:30:01] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Unable to generate family for wpbeta:zh with github action (ClientError: (403) Request forbidden) - https://phabricator.wikimedia.org/T399415 (10Xqt) 03NEW [18:32:20] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Unable to generate family for wpbeta:zh with github action (ClientError: (403) Request forbidden) - https://phabricator.wikimedia.org/T399415#10998503 (10Xqt) [18:32:21] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10998504 (10Xqt) [19:31:14] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management, 07Beta-Cluster-reproducible: Inconsistent user permissions for users who were recently added to a new group (June 2025 edition) - https://phabricator.wikimedia.org/T396061#10998514 (10Lobo77) 05Open→03Resolved a:03Lobo77 [19:32:09] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management, 07Beta-Cluster-reproducible: Inconsistent user permissions for users who were recently added to a new group (June 2025 edition) - https://phabricator.wikimedia.org/T396061#10998516 (10Ladsgroup) 05Resolved→03Open a:05Lobo77→03None [19:35:36] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#10998520 (10Xqt) a:05Xqt→03None [19:36:53] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#10998521 (10hashar) That is slight different though, this time it is GitHub IP range that got blocked. We'd need to know their r... [21:42:49] 10Beta-Cluster-Infrastructure: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 - https://phabricator.wikimedia.org/T399419 (10bd808) 03NEW [21:43:06] 10Beta-Cluster-Infrastructure: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 - https://phabricator.wikimedia.org/T399419#10998594 (10bd808) [21:51:49] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10998597 (10Krinkle) Well, that didn't help. I still can't get a packet through. `name=Locally krinkle@deployment-memc13:~$ echo "delete krinkle0" | nc deployme... [22:03:18] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10998601 (10Paladox) From your logged message, you added 11212 rather than 11211. [22:14:42] 10Beta-Cluster-Infrastructure: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 - https://phabricator.wikimedia.org/T399419#10998602 (10bd808) I think that the directory linked to by /etc/acmecerts/unified/live is the only one that is actually used. Let's try getting rid... [22:16:52] 10Beta-Cluster-Infrastructure: Warning about /etc/acmecerts/unified contents during puppet run on deployment-cache-text08 - https://phabricator.wikimedia.org/T399419#10998603 (10bd808) >>! In T399419#10998602, @bd808 wrote: > I think that the directory linked to by /etc/acmecerts/unified/live is the only one tha... [22:37:41] 10Beta-Cluster-Infrastructure: Logins seem not to work after switch to *.beta.wmcloud.org canonical domains - https://phabricator.wikimedia.org/T399349#10998606 (10bd808) >>! In T399349#10997413, @Tgr wrote: > So, some kind of firewall issue? The security groups that @Krinkle modified should only affect traffic... [22:38:37] 10Beta-Cluster-Infrastructure: Login broken by memcached local firewall - https://phabricator.wikimedia.org/T399349#10998609 (10bd808) [22:44:04] 10Beta-Cluster-Infrastructure: Login broken by memcached local firewall - https://phabricator.wikimedia.org/T399349#10998610 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/531c91d329807702d416cfee13913a4443ac2531%5E%21/#F0 ` diff --git a/deployment-prep/deployment-memc.yaml b/de... [22:46:57] 10Beta-Cluster-Infrastructure: Login broken by memcached local firewall - https://phabricator.wikimedia.org/T399349#10998611 (10Krinkle) >>! In T399349#10998601, @Paladox wrote: > From your logged message, you added 11212 rather than 11211. >>! In T399349#10998606, @bd808 wrote: > The security groups that @Krin... [22:47:13] 10Beta-Cluster-Infrastructure: Login broken by memcached local firewall - https://phabricator.wikimedia.org/T399349#10998612 (10bd808) After forcing a puppet run on deployment-memc11 and deployment-memc12 I seem to be able to login again. [22:48:43] 10Beta-Cluster-Infrastructure: Login broken by memcached local firewall - https://phabricator.wikimedia.org/T399349#10998613 (10Krinkle) Fixed! ` krinkle@deployment-mediawiki14:~$ echo "delete krinkle0" | nc deployment-memc11 11211 NOT_FOUND ^C krinkle@deployment-mediawiki14:~$ echo "delete krinkle0" | nc deplo... [22:53:18] 10Beta-Cluster-Infrastructure: Login broken by memcached local firewall - https://phabricator.wikimedia.org/T399349#10998614 (10bd808) 05Open→03Resolved a:03bd808 I think the moral of the story here is that "make puppet run" is a necessary, but not always sufficient action when getting Beta to work wit... [22:54:13] 10Beta-Cluster-Infrastructure: Login broken by memcached ferm rules being bypassed by hiera configuration - https://phabricator.wikimedia.org/T399349#10998620 (10bd808) [23:15:42] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#10998626 (10bd808) GitHub Actions are going to be using the same IP blocks as Microsoft Azure. As long as we are blocking in Bet... [23:17:28] bd808: I suppose it goes without saying, a web server can handle a lot more load with a working Memcached stack. [23:17:47] * Krinkle looks at Grafana for mw14 [23:23:27] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Unable to generate family for wpbeta:zh with github action (ClientError: (403) Request forbidden) - https://phabricator.wikimedia.org/T399415#10998627 (10bd808) [23:23:28] 10Beta-Cluster-Infrastructure: 2025-07-11 traffic overload - https://phabricator.wikimedia.org/T399329#10998628 (10bd808) [23:23:30] 10Beta-Cluster-Infrastructure, 07Epic, 13Patch-For-Review: 2025 tracking task for Beta Cluster (deployment-prep) traffic overload protection (blocking unwanted crawlers) - https://phabricator.wikimedia.org/T393487#10998629 (10bd808) [23:24:47] 10Beta-Cluster-Infrastructure: 2025-07-11 traffic overload - https://phabricator.wikimedia.org/T399329#10998630 (10bd808) [23:24:49] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests, 07Upstream: CI tests fails with TimeoutError when userinfo is retrieved - https://phabricator.wikimedia.org/T399367#10998631 (10bd808) [23:32:23] 10Beta-Cluster-Infrastructure, 10Pywikibot, 07Pywikibot-tests: Unable to generate family for wpbeta:zh with github action (ClientError: (403) Request forbidden) - https://phabricator.wikimedia.org/T399415#10998648 (10bd808) The blocks made for {T399329} have caught some of the Microsoft Azure network range w... [23:39:30] 10Beta-Cluster-Infrastructure: 2025-07-11 traffic overload - https://phabricator.wikimedia.org/T399329#10998654 (10bd808) We found in {T399349} that memcached was firewalled off from everything which probably made this all much. much worse than it otherwise would have been. MediaWiki really needs caching to work... [23:47:40] 10Beta-Cluster-Infrastructure: 2025-07-11 traffic overload - https://phabricator.wikimedia.org/T399329#10998663 (10Krinkle) Yep :) [WMCS Grafana: deployment-prep](https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?var-project=deployment-prep&orgId=1&from=2025-07-13T21:00:00.000Z&to=2025-07-13T23:45...