[00:02:42] 10Tool-containers: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups - https://phabricator.wikimedia.org/T414836#11560647 (10bd808) I thought a bit about how to adjust envvar lookups so that multiple instances of the reverse proxy job could run in parallel. I think that could... [00:32:56] (03update) 10raymond-ndibe: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [00:49:54] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [00:50:07] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] (refactor_job_payload) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [00:51:01] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] (refactor_job_payload) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [01:17:12] (03update) 10raymond-ndibe: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [01:18:10] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] (refactor_job_payload) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [01:20:39] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] (refactor_job_payload) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [01:27:09] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T348755) [01:31:43] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T348755) [02:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:06:24] (03update) 10raymond-ndibe: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [04:11:29] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] (refactor_job_payload) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [04:21:13] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (publish_continuous_job_to_internet) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T348755) [04:24:18] (03update) 10raymond-ndibe: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [04:34:28] (03update) 10raymond-ndibe: support exposing continuous jobs to the internet [repos/cloud/toolforge/jobs-api] (replace_job_images_with_web_images) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/262 (https://phabricator.wikimedia.org/T388092) [04:37:49] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] (refactor_job_payload) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [04:38:06] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] (refactor_job_payload) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [04:58:11] (03update) 10raymond-ndibe: support publishing continuous jobs to the internet [repos/cloud/toolforge/jobs-cli] (refactor_job_payload) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/142 (https://phabricator.wikimedia.org/T388092) [05:01:08] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [05:01:52] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [05:02:51] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [05:04:05] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [05:27:21] (03update) 10raymond-ndibe: jobs-api: test continuous job publish [repos/cloud/toolforge/toolforge-deploy] (test_for_image_argument_handling_in_jobs) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1116 (https://phabricator.wikimedia.org/T388092) [06:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:50:58] 06cloud-services-team, 10Toolforge: [components-api] Redeploying same version shows diff in "timeout" - https://phabricator.wikimedia.org/T415280#11561338 (10Aklapper) [10:00:43] (03Abandoned) 10Arendpieter: Use IDP for authentication [labs/striker] - 10https://gerrit.wikimedia.org/r/1189915 (https://phabricator.wikimedia.org/T359554) (owner: 10Arendpieter) [10:38:56] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:18:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:35:24] 06cloud-services-team, 10Cloud-VPS: Add check for cloud-wide root keys to the offboarding script - https://phabricator.wikimedia.org/T398214#11561848 (10taavi) a:03taavi [13:38:47] (03open) 10swayamagrahari: Code refractoring and Ui fixes [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/62 [14:11:13] (03merge) 10swayamagrahari: Code refractoring and Ui fixes [toolforge-repos/isa] - 10https://gitlab.wikimedia.org/toolforge-repos/isa/-/merge_requests/62 [14:35:47] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T410948#11562521 (10Xqt) Pywikibot on Toolforge is **five releases behind** the current upstream stable version 10.7.5. Is there an issue preventing regular updates? [15:19:45] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T410948#11562718 (10fnegri) 05Open→03In progress @Xqt these upgrades are usually handled as part of WMCS [Clinic duties](https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Se... [15:25:54] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Tool-admin: Import tool-admin jobs to Git repo - https://phabricator.wikimedia.org/T415383#11562725 (10taavi) Tool accounts are cheap. [15:27:09] (03merge) 10taavi: server: Allowlist permitted annotations [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/35 (https://phabricator.wikimedia.org/T415192) [15:33:05] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: ingress-admission: bump to 0.0.77-20260128152724-d2dfd2a6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1117 (https://phabricator.wikimedia.org/T415192) [15:33:37] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [15:33:44] 06cloud-services-team, 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T410948#11562745 (10fnegri) Hmm something seems to be broken: `lang=shell-session tools.pywikibot@tools-bastion-15:~$ toolforge jobs run --image tool-pywikibot/pywikibot-scripts-... [15:34:01] 10Tool-redminbot, 10Phabricator: Request to create a workboard for #tool-redminbot - https://phabricator.wikimedia.org/T415793 (10Redmin) 03NEW [15:34:04] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T410948#11562758 (10fnegri) [15:34:15] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T410948#11562772 (10fnegri) a:03fnegri [15:37:12] 10Tool-redminbot, 10Phabricator: Request to create a workboard for #tool-redminbot - https://phabricator.wikimedia.org/T415793#11562784 (10Redmin) [15:42:55] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [15:43:16] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission [15:50:57] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T410948#11562841 (10fnegri) I think this could be the issue: `lang=diff +.. versionchanged:: 10.7.1 + Default ``PASS_BASENAME`` was changed from ``user-pass... [15:51:58] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission [15:58:34] (03merge) 10taavi: ingress-admission: bump to 0.0.77-20260128152724-d2dfd2a6 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1117 (https://phabricator.wikimedia.org/T415192) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:07:45] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: Move ingress annotations to an allowlist model - https://phabricator.wikimedia.org/T415192#11562902 (10taavi) 05Open→03Resolved [16:19:56] (03update) 10kbach: Draft: Introduce support for passing OAD URL in a parameter [toolforge-repos/api-spec-reader] - 10https://gitlab.wikimedia.org/toolforge-repos/api-spec-reader/-/merge_requests/2 [16:46:39] 10Tool-redminbot, 10Phabricator: Request to create a workboard for #tool-redminbot - https://phabricator.wikimedia.org/T415793#11563029 (10Pppery) 05Open→03Invalid Added you to #trusted-contributors so you should be able to do this yourself. [16:50:39] (03approved) 10wikigit: feat(T410402): request validation [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/10 (owner: 10aghirelli) [16:50:42] (03merge) 10wikigit: feat(T410402): request validation [toolforge-repos/wmf-openapi-linter] - 10https://gitlab.wikimedia.org/toolforge-repos/wmf-openapi-linter/-/merge_requests/10 (owner: 10aghirelli) [17:18:20] 10Tool-redminbot, 10Phabricator: Request to create a workboard for #tool-redminbot - https://phabricator.wikimedia.org/T415793#11563206 (10Redmin) Thanks! [17:56:45] 10Tool-redminbot: Query wiki replicas to fetch list of pages - https://phabricator.wikimedia.org/T415811 (10Redmin) 03NEW [17:57:18] 10Tool-redminbot: Query wiki replicas to fetch list of pages - https://phabricator.wikimedia.org/T415811#11563437 (10Redmin) p:05Triage→03Medium [18:05:52] 10Tool-redminbot: Use argparse to parse CLI arguments - https://phabricator.wikimedia.org/T415812 (10Redmin) 03NEW [18:15:15] 10Tool-redminbot: Fix cases of misnested tags - https://phabricator.wikimedia.org/T415814 (10Redmin) 03NEW [18:16:10] 10Tool-redminbot: Use argparse to parse CLI arguments - https://phabricator.wikimedia.org/T415812#11563499 (10Redmin) p:05Triage→03Low [18:19:24] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T410948#11563515 (10fnegri) Found the issue: I was running my tests as the `pywikibot` user in toolforge, but they should be run as `wikitech-double-redirect-b... [18:36:15] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25), 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T410948#11563571 (10fnegri) 05In progress→03Resolved [18:44:36] 06cloud-services-team, 10Toolforge, 10Pywikibot: In the Toolforge pywikibot image, missing env vars cause obscure error messages - https://phabricator.wikimedia.org/T415818 (10fnegri) 03NEW [18:44:49] 06cloud-services-team, 10Toolforge, 10Pywikibot: In the Toolforge pywikibot image, missing env vars cause obscure error messages - https://phabricator.wikimedia.org/T415818#11563638 (10fnegri) p:05Triage→03Low [18:46:00] 06cloud-services-team, 10Data-Services: Deprecate /data/scratch volume - https://phabricator.wikimedia.org/T415819 (10taavi) 03NEW [18:46:43] 06cloud-services-team, 10Toolforge: In the Toolforge pywikibot image, missing env vars cause obscure error messages - https://phabricator.wikimedia.org/T415818#11563656 (10taavi) Untagging #Pywikibot as the envvar config system is specific to the Toolforge image. [18:47:10] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25), 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T410948#11563659 (10fnegri) > Found the issue: I was running my tests as the pywikibot user in toolforge Raised {T415818} to impr... [18:50:14] 06cloud-services-team, 10Toolforge: In the Toolforge pywikibot image, missing env vars cause obscure error messages - https://phabricator.wikimedia.org/T415818#11563663 (10bd808) Making the missing envvar error more obvious might be as easy as turning things like `os.environ.get('PWB_USERNAME')` into `os.envir... [19:10:38] 10Tool-redminbot: Handle all cases of obsolete tags - https://phabricator.wikimedia.org/T415821 (10Redmin) 03NEW [19:11:03] 10Tool-redminbot: Handle all cases of obsolete tags - https://phabricator.wikimedia.org/T415821#11563741 (10Redmin) p:05Triage→03High [19:13:42] 10Tool-redminbot: Handle all cases of obsolete tags - https://phabricator.wikimedia.org/T415821#11563747 (10Redmin) [20:41:18] 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T415828#11563946 (10LibUp-bot) [20:41:19] 10Toolforge, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T415829#11563948 (10LibUp-bot)