[01:57:10] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11523748 (10Raymond_Ndibe) >>! In T359649#11518078, @fnegri wrote: > Is this something that we still want to do? Yes. It's pr... [02:17:18] (03update) 10raymond-ndibe: runtimes: k8s: Simplify checking for quota errors [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/258 (https://phabricator.wikimedia.org/T414229) (owner: 10taavi) [02:17:20] (03approved) 10raymond-ndibe: runtimes: k8s: Simplify checking for quota errors [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/258 (https://phabricator.wikimedia.org/T414229) (owner: 10taavi) [02:17:22] (03update) 10raymond-ndibe: runtimes: k8s: Simplify checking for quota errors [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/258 (https://phabricator.wikimedia.org/T414229) (owner: 10taavi) [02:35:51] (03update) 10raymond-ndibe: jobs-api: add jobs version migration script and docs [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/746 (https://phabricator.wikimedia.org/T359649) (owner: 10dcaro) [02:43:26] (03open) 10raymond-ndibe: runtime::diff_with_running_job: temp conditional to force job version upgrade from v1 -> v2 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/259 (https://phabricator.wikimedia.org/T359649) [03:01:15] (03update) 10raymond-ndibe: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [05:22:34] (03open) 10raymond-ndibe: Draft: images::from_url_or_name: match variants of the ssame image [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/260 [05:23:27] (03update) 10raymond-ndibe: Draft: images::from_url_or_name: match variants of the ssame image [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/260 [05:24:33] (03update) 10raymond-ndibe: Draft: images::from_url_or_name: match variants of the ssame image [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/260 [08:07:19] (03close) 10mercy-o: Added filtering feature of both the tasks and topics categories [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/1 [08:28:51] (03merge) 10taavi: runtimes: k8s: Simplify checking for quota errors [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/258 (https://phabricator.wikimedia.org/T414229) [08:31:39] (03update) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.460-20260115082907-d1d0699e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1111 (https://phabricator.wikimedia.org/T414229) [08:31:47] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-api: bump to 0.0.460-20260115082907-d1d0699e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1111 (https://phabricator.wikimedia.org/T414229) [08:31:57] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:43:10] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [08:45:00] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [08:58:23] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [08:58:59] (03merge) 10taavi: jobs-api: bump to 0.0.460-20260115082907-d1d0699e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1111 (https://phabricator.wikimedia.org/T414229) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [08:59:46] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: jobs-api does not properly handle quota errors when restarting a job - https://phabricator.wikimedia.org/T414229#11524120 (10taavi) 05Open→03Resolved I believe this is fixed now. [10:34:40] (03PS1) 10Elukey: Add fake kerberos keytabs for the Puppetserver hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1227290 (https://phabricator.wikimedia.org/T402512) [10:35:01] (03CR) 10Elukey: [V:03+2 C:03+2] Add fake kerberos keytabs for the Puppetserver hosts [labs/private] - 10https://gerrit.wikimedia.org/r/1227290 (https://phabricator.wikimedia.org/T402512) (owner: 10Elukey) [11:03:17] (03update) 10mercy-o: Add recommended tasks table display and multi-filtering of topics, geography, and tasks [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/2 [11:15:58] (03update) 10mercy-o: Add recommended tasks table display and multi-filtering of topics, geography, and tasks [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/2 [11:36:13] 06cloud-services-team: Alerts with changing values in `summary` will spam tasks - https://phabricator.wikimedia.org/T414669 (10fgiunchedi) 03NEW [11:38:01] (03update) 10mercy-o: Add recommended tasks table display and multi-filtering of topics, geography, and tasks [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/2 [11:42:25] (03update) 10mercy-o: Add recommended tasks table display and multi-filtering of topics, geography, and tasks [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/2 [11:49:37] (03update) 10mercy-o: Add recommended tasks table display and multi-filtering of topics, geography, and tasks [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/2 [11:59:57] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS: Add monitoring for Metadata service - https://phabricator.wikimedia.org/T395797#11524557 (10fgiunchedi) [12:00:33] 06cloud-services-team, 10Cloud-VPS: openstack magnum (or heat) resource leak - https://phabricator.wikimedia.org/T392031#11524562 (10fgiunchedi) Is this still a problem? [12:22:48] 10VPS-project-Codesearch, 07Upstream: codesearch is not searching package-lock.json - https://phabricator.wikimedia.org/T241033#11524645 (10A_smart_kitten) Pasting the exclusion message here for searchability: ` Trigram ratio too high (0.11), probably not text ` Also noting that `operations/puppet`'s `modules/... [12:38:46] (03update) 10mercy-o: Add recommended tasks table display and multi-filtering of topics, geography, and tasks [toolforge-repos/microtask-generator] - 10https://gitlab.wikimedia.org/toolforge-repos/microtask-generator/-/merge_requests/2 [12:42:08] 06cloud-services-team, 10Toolforge: Replace ingress-nginx before upstream EOL date - https://phabricator.wikimedia.org/T392356#11524697 (10taavi) p:05Medium→03High [12:44:14] 06cloud-services-team, 10Toolforge: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674 (10taavi) 03NEW [12:46:30] 06cloud-services-team, 10Toolforge: Remove remaining uses of ingress-nginx specific annotations - https://phabricator.wikimedia.org/T414674#11524714 (10taavi) The current list of tools relying on this functionality is at P87548. Of those, a couple are simple redirects that can be replaced with https://wikitech... [12:51:51] 10VPS-project-Codesearch, 07Upstream: codesearch is not searching package-lock.json - https://phabricator.wikimedia.org/T241033#11524725 (10A_smart_kitten) Skimming through & Ctrl+F-ing for the word `probably`, it seems like a large majority of exclude... [13:26:00] 10VPS-project-Codesearch, 07Upstream: codesearch is not searching package-lock.json - https://phabricator.wikimedia.org/T241033#11524778 (10Ladsgroup) I have a counter-proposal. I don't think scanning package-lock is useful or better said, it's a X/Y problem. We need proper SBOMs produced from our software (an... [13:29:29] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Cloud-VPS, 06Infrastructure-Foundations, 10netops: cloud: edge network suffers downtime if one cloudsw is down - https://phabricator.wikimedia.org/T375259#11524793 (10fgiunchedi) [13:37:05] 06cloud-services-team: WMCS hardware services: 3-node HA redundancy model - https://phabricator.wikimedia.org/T377570#11524829 (10taavi) Closed as these hosts: * are non-stateful * have low-enough loads that a single server can handle usual traffic levels So a third node would have very little benefits. [13:45:44] 06cloud-services-team, 10Cloud-VPS: [wmcs-cookbooks] Improve cloudvirt.drain and cloudvirt.set_maintenance - https://phabricator.wikimedia.org/T375867#11524843 (10fgiunchedi) p:05High→03Medium Back to medium as this hasn't been an immediate problem AFAICT. `set_maintenance` cookbook can likely go as @fnegr... [13:58:49] 10VPS-project-Codesearch, 07Upstream: codesearch is not searching package-lock.json - https://phabricator.wikimedia.org/T241033#11524899 (10A_smart_kitten) I mean, I guess I don't have a firm immediate opinion on that (though I guess my immediate reply would probably be something like 'if folks don't find `pac... [14:09:51] (03PS1) 10Elukey: Revert "Add fake kerberos keytabs for the Puppetserver hosts" [labs/private] - 10https://gerrit.wikimedia.org/r/1227342 [14:09:56] (03CR) 10Elukey: [V:03+2 C:03+2] Revert "Add fake kerberos keytabs for the Puppetserver hosts" [labs/private] - 10https://gerrit.wikimedia.org/r/1227342 (owner: 10Elukey) [14:33:48] (03update) 10raymond-ndibe: runtime::diff_with_running_job: temp conditional to force job version upgrade from v1 -> v2 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/259 (https://phabricator.wikimedia.org/T359649) [14:44:55] 06cloud-services-team, 13Patch-For-Review: Alerts with changing values in `summary` will spam tasks - https://phabricator.wikimedia.org/T414669#11525089 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Done! [15:27:52] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11525375 (10fnegri) 05Open→03In progress a:03Raymond_Ndibe [15:32:27] 06cloud-services-team (FY2025/2026-Q3-Q4), 10Toolforge (Toolforge iteration 25), 13Patch-For-Review: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version - https://phabricator.wikimedia.org/T359649#11525404 (10fnegri) Discussed in the team meeting today, @taavi suggested we coul... [15:33:51] 06cloud-services-team: WMCS hardware services: 3-node HA redundancy model - https://phabricator.wikimedia.org/T377570#11525410 (10Andrew) ftr I can also good with sticking with 2-node setups. [15:37:13] 10Cloud-VPS (Quota-requests), 10Observability-Logging: Request to increase quotas for logging project - https://phabricator.wikimedia.org/T414648#11525421 (10fnegri) +1 [16:21:29] !log volans@cloudcumin1001 logging START - Cookbook wmcs.openstack.quota_increase by 8 cores, 80 gigabytes, 16384 ram, 2 volumes (T414648) [16:21:34] T414648: Request to increase quotas for logging project - https://phabricator.wikimedia.org/T414648 [16:21:37] !log volans@cloudcumin1001 logging END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 8 cores, 80 gigabytes, 16384 ram, 2 volumes (T414648) [16:29:15] 10Cloud-VPS (Quota-requests), 10Observability-Logging: Request to increase quotas for logging project - https://phabricator.wikimedia.org/T414648#11525629 (10Volans) 05Open→03Resolved p:05Triage→03Medium a:03Volans Limits increased. [16:32:54] 10Cloud-VPS (Quota-requests), 10Observability-Logging: Request to increase quotas for logging project - https://phabricator.wikimedia.org/T414648#11525653 (10colewhite) Thank you!! [17:07:32] (03open) 10fnegri: Don't chmod the home directory [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/304 [17:40:48] 06cloud-services-team, 10Cloud-VPS: openstack magnum (or heat) resource leak - https://phabricator.wikimedia.org/T392031#11525837 (10Andrew) The intermittent magnum failures are still a problem as far as I know. There's a major refactor upcoming that will likely replace this issue with other, different ones. [18:14:09] (03update) 10fnegri: runtime::diff_with_running_job: temp conditional to force job version upgrade from v1 -> v2 [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/259 (https://phabricator.wikimedia.org/T359649) (owner: 10raymond-ndibe) [18:14:41] (03update) 10fnegri: Use the image name as provided by builds-api [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T403322) (owner: 10damian) [18:15:00] (03update) 10fnegri: [jobs-api] save business models in a DB [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/114 (owner: 10raymond-ndibe) [22:19:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [22:24:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [23:53:46] 10Tool-Phabricator-bug-status: Usurp and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#11527238 (10bd808) >>! In T142237#10449475, @bd808 wrote: > It would be nice though, so I will try not to loose track of this for years again. It was only one year this... [23:56:32] 10Tool-Phabricator-bug-status: Usurp and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#11527241 (10bd808) 05Open→03Stalled p:05Triage→03Medium a:03bd808 [23:58:01] 10Tool-Phabricator-bug-status: BugStatusUpdate gadget can't work - https://phabricator.wikimedia.org/T329927#11527249 (10bd808) p:05Triage→03Medium [23:58:32] 10Tool-Phabricator-bug-status: Usurp and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#11527252 (10bd808) [23:58:36] 10Tool-Phabricator-bug-status: BugStatusUpdate gadget can't work - https://phabricator.wikimedia.org/T329927#11527253 (10bd808) [23:59:46] 10Tool-Phabricator-bug-status: Usurp and move phabricator-bug-status to the Toolforge Jobs Framework - https://phabricator.wikimedia.org/T142237#11527256 (10bd808)