[00:05:55] FIRING: MaxConntrack: Max conntrack at 84.29% on cloudvirt1067:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:55:55] RESOLVED: MaxConntrack: Max conntrack at 83.17% on cloudvirt1067:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [01:10:24] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1166814 (owner: 10L10n-bot) [02:05:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [02:25:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [03:12:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [03:27:45] (03PS1) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [03:30:56] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [04:22:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [05:03:00] (03open) 10marostegui: task_template_*_primary: Refine DNS step [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/12 [05:12:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [06:08:17] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team: Add the 'other assignee' field to the Phabricator test instance - https://phabricator.wikimedia.org/T398732#10981975 (10A_smart_kitten) Thanks @Dzahn! I've done that now - for the record, I've left the new value I've changed tha... [06:12:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [06:42:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance runner-1040 in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [07:07:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance runner-1040 in project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [08:08:02] 10PAWS: [2025-07-08] PAWS down - https://phabricator.wikimedia.org/T398912#10982258 (10A_smart_kitten) [08:32:53] (03approved) 10ladsgroup: task_template_*_primary: Refine DNS step [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/12 (owner: 10marostegui) [08:32:55] (03merge) 10ladsgroup: task_template_*_primary: Refine DNS step [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/12 (owner: 10marostegui) [09:28:45] (03PS1) 10Majavah: settings: Update UID ranges [labs/striker] - 10https://gerrit.wikimedia.org/r/1167155 (https://phabricator.wikimedia.org/T355663) [09:44:23] 06cloud-services-team, 10Bitu, 06Infrastructure-Foundations, 07LDAP, 13Patch-For-Review: Allocate more available UNIX UIDs for human users - https://phabricator.wikimedia.org/T355663#10982782 (10MoritzMuehlenhoff) 05Open→03Resolved I created a user manually with the folllowing LDIF: ` dn: cn=bu... [09:48:08] (03CR) 10Majavah: [C:03+2] settings: Update UID ranges [labs/striker] - 10https://gerrit.wikimedia.org/r/1167155 (https://phabricator.wikimedia.org/T355663) (owner: 10Majavah) [09:49:35] (03Merged) 10jenkins-bot: settings: Update UID ranges [labs/striker] - 10https://gerrit.wikimedia.org/r/1167155 (https://phabricator.wikimedia.org/T355663) (owner: 10Majavah) [10:05:00] (03open) 10taavi: logging: alloy: Deploy cluster-wide on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/871 (https://phabricator.wikimedia.org/T386480) [10:05:03] (03update) 10taavi: logging: alloy: Deploy cluster-wide on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/871 (https://phabricator.wikimedia.org/T386480) [10:05:25] (03update) 10taavi: logging: alloy: Deploy cluster-wide on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/871 (https://phabricator.wikimedia.org/T386480) [10:08:58] david-caro opened https://github.com/toolforge/paws/pull/490 [10:21:56] david-caro opened https://github.com/toolforge/paws/pull/491 [10:33:06] david-caro opened https://github.com/toolforge/paws/pull/492 [10:47:19] FIRING: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [10:52:19] RESOLVED: HighIOWaitStalling: High iowait detected on clouddumps1002:9100. - https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Shared_storage#Dumps - https://grafana.wikimedia.org/d/000000568/wmcs-dumps-general-view - https://alerts.wikimedia.org/?q=alertname%3DHighIOWaitStalling [11:41:07] 06cloud-services-team, 10Data-Services, 10ContentTranslation: Consider adding ContentTranslation (CX) tables to wiki replicas - https://phabricator.wikimedia.org/T196020#10983267 (10Nikerabbit) The publicity has been defined in {T386094} so this seems more of wiki replicas task now how to expose crosswiki ta... [11:42:43] 06cloud-services-team, 10Data-Services, 06Data-Persistence, 06Privacy Engineering, 07SecTeam-Processed: Add "wikishared" database to wiki replicas - https://phabricator.wikimedia.org/T395072#10983273 (10taavi) [11:42:44] 06cloud-services-team, 10Data-Services, 10ContentTranslation: Consider adding ContentTranslation (CX) tables to wiki replicas - https://phabricator.wikimedia.org/T196020#10983272 (10taavi) [11:57:18] 10Tool-drift-tracker, 06DBA: Wire tables catalog to drift tracker - https://phabricator.wikimedia.org/T398949 (10Ladsgroup) 03NEW [12:00:35] (03approved) 10dcaro: logging: alloy: Deploy cluster-wide on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/871 (https://phabricator.wikimedia.org/T386480) (owner: 10taavi) [12:01:47] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Data-Services: [wikireplicas] Gather usage stats - https://phabricator.wikimedia.org/T381587#10983378 (10fnegri) Another tool we might use is [querysampler.py](https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/wmcs/d... [12:15:42] 10PAWS: [2025-07-08] PAWS down - https://phabricator.wikimedia.org/T398912#10983422 (10dcaro) Fixing with the stack of https://github.com/toolforge/paws/pull/492 Updated also the docs https://wikitech.wikimedia.org/wiki/PAWS/Admin#Rotating_credentials [12:26:00] 10Tool-drift-tracker, 06DBA: Wire tables catalog to drift tracker - https://phabricator.wikimedia.org/T398949#10983501 (10Ladsgroup) p:05Triage→03Medium [12:28:03] (03update) 10raymond-ndibe: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [12:37:00] (03merge) 10taavi: logging: alloy: Deploy cluster-wide on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/871 (https://phabricator.wikimedia.org/T386480) [12:37:30] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component logging [12:39:38] (03update) 10raymond-ndibe: [jobs-cli] refactor job payload [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/98 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [12:39:55] (03update) 10raymond-ndibe: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) [12:40:36] 06cloud-services-team, 10PAWS: [2025-07-08] PAWS down - https://phabricator.wikimedia.org/T398912#10983587 (10taavi) [12:42:44] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component logging [13:01:58] 06cloud-services-team, 10PAWS: [2025-07-08] PAWS down - https://phabricator.wikimedia.org/T398912#10983630 (10dcaro) After some playing around with re-enabling accounts, paws is back up and running, will merge the PRs and declare this a win xd [13:03:28] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10983640 (10github-toolforge-bot) david-caro closed https://github.com/toolforge/paws/pull/491 [13:03:42] david-caro closed https://github.com/toolforge/paws/pull/491 [13:03:42] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: rename lan-flat-cloudinstances2b to VLAN/legacy - https://phabricator.wikimedia.org/T389942#10983642 (10github-toolforge-bot) david-caro closed https://github.com/toolforge/paws/pull/492 [13:04:01] david-caro closed https://github.com/toolforge/paws/pull/492 [13:05:35] (03PS2) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [13:07:15] david-caro opened https://github.com/toolforge/paws/pull/493 [13:09:12] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [13:10:30] david-caro closed https://github.com/toolforge/paws/pull/493 [13:11:58] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:12:03] david-caro opened https://github.com/toolforge/paws/pull/494 [13:12:57] david-caro closed https://github.com/toolforge/paws/pull/494 [13:13:03] 06cloud-services-team, 10PAWS: [2025-07-08] PAWS down - https://phabricator.wikimedia.org/T398912#10983694 (10dcaro) 05Open→03Resolved p:05Triage→03High a:03dcaro [13:13:38] 06cloud-services-team, 10PAWS: [2025-07-08] PAWS down - https://phabricator.wikimedia.org/T398912#10983698 (10dcaro) This also moved the cluster to the new vxlan dualstack network [13:16:26] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: [wmcs-cookbooks] create_instance_with_prefix should not use vlan/legacy - https://phabricator.wikimedia.org/T398625#10983708 (10taavi) p:05Triage→03Medium a:03taavi [13:19:55] (03PS1) 10Majavah: wmcs_libs: openstack: Avoid new use of legacy VLAN network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1167220 (https://phabricator.wikimedia.org/T398625) [13:24:05] (03CR) 10CI reject: [V:04-1] wmcs_libs: openstack: Avoid new use of legacy VLAN network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1167220 (https://phabricator.wikimedia.org/T398625) (owner: 10Majavah) [13:25:09] (03approved) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/26 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:25:13] (03merge) 10dcaro: build: Upgrade Poetry dependencies [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/26 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [13:26:28] (03PS2) 10Majavah: wmcs_libs: openstack: Avoid new use of legacy VLAN network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1167220 (https://phabricator.wikimedia.org/T398625) [13:27:28] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: jobs-emailer: bump to 0.0.62-20250708132525-c01c0ea7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/872 [13:35:02] (03PS3) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [13:36:41] (03CR) 10FNegri: [C:03+1] wmcs_libs: openstack: Avoid new use of legacy VLAN network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1167220 (https://phabricator.wikimedia.org/T398625) (owner: 10Majavah) [13:38:38] (03PS3) 10Majavah: wmcs_libs: openstack: Avoid new use of legacy VLAN network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1167220 (https://phabricator.wikimedia.org/T398625) [13:38:47] (03CR) 10Majavah: [C:03+2] wmcs_libs: openstack: Avoid new use of legacy VLAN network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1167220 (https://phabricator.wikimedia.org/T398625) (owner: 10Majavah) [13:39:20] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [13:39:56] 06cloud-services-team, 06Data-Platform-SRE: wikitech-static: resume daily dumps - https://phabricator.wikimedia.org/T398968 (10fnegri) 03NEW [13:40:24] 06cloud-services-team, 10wikitech.wikimedia.org, 06Data-Platform-SRE: wikitech-static: resume daily dumps - https://phabricator.wikimedia.org/T398968#10983919 (10taavi) [13:41:22] (03update) 10dcaro: [jobs-api] check services diff [repos/cloud/toolforge/jobs-api] (fix_diff_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/158 (https://phabricator.wikimedia.org/T392717) (owner: 10raymond-ndibe) [13:42:05] (03Merged) 10jenkins-bot: wmcs_libs: openstack: Avoid new use of legacy VLAN network [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1167220 (https://phabricator.wikimedia.org/T398625) (owner: 10Majavah) [13:42:38] 06cloud-services-team, 10Cloud-VPS, 10Toolforge, 13Patch-For-Review: [wmcs-cookbooks] create_instance_with_prefix should not use vlan/legacy - https://phabricator.wikimedia.org/T398625#10983957 (10taavi) 05Open→03Resolved [13:45:35] (03PS4) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [13:46:48] 06cloud-services-team, 10wikitech.wikimedia.org, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): wikitech-static: resume daily dumps - https://phabricator.wikimedia.org/T398968#10983961 (10BTullis) p:05Triage→03High a:03BTullis [13:50:30] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [13:51:37] (03PS5) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [13:53:30] 06cloud-services-team, 10wikitech.wikimedia.org, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): wikitech-static: resume daily dumps - https://phabricator.wikimedia.org/T398968#10984000 (10BTullis) Apologies, this was an oversight on my part, as I had assumed that the standard XML/SQL dumps for labswiki would... [13:55:25] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [13:57:04] 06cloud-services-team, 10wikitech.wikimedia.org, 10Data-Platform-SRE (2025.07.05 - 2025.07.25): wikitech-static: resume daily dumps - https://phabricator.wikimedia.org/T398968#10984049 (10BTullis) I have back-filled to clouddumps100[1-2] with commands like this. ` dumpsgen@dumpsdata1003:/data/otherdumps/wiki... [13:58:15] (03PS6) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [14:02:00] 10cloud-services-team (FY2024/2025-Q3-Q4): Create WMCS offboarding checklist - https://phabricator.wikimedia.org/T398972 (10fnegri) 03NEW [14:02:23] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [14:19:59] (03PS7) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [14:24:02] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [14:27:18] (03PS8) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [14:27:46] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudcephosd2003-dev.codfw.wmnet - https://phabricator.wikimedia.org/T397979#10984285 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [14:31:30] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [14:33:16] (03PS9) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [14:44:14] (03CR) 10Hnowlan: [C:03+2] Add fake hcaptcha proxy secrets. [labs/private] - 10https://gerrit.wikimedia.org/r/1155221 (https://phabricator.wikimedia.org/T397841) (owner: 10Kamila Součková) [14:44:22] (03CR) 10Hnowlan: [V:03+2 C:03+2] Add fake hcaptcha proxy secrets. [labs/private] - 10https://gerrit.wikimedia.org/r/1155221 (https://phabricator.wikimedia.org/T397841) (owner: 10Kamila Součková) [14:55:24] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [14:57:17] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [14:58:24] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.reactivate (exit_code=97) [14:58:41] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [14:58:48] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [14:59:15] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [14:59:48] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [14:59:59] 06cloud-services-team, 10Toolforge (Toolforge iteration 21): [components-api,beta] CI pipelines should wait until Toolforge deployment is 100% successful - https://phabricator.wikimedia.org/T398485#10984427 (10dcaro) a:03dcaro [15:00:02] 10Toolforge (Toolforge iteration 21): [components-api,api-gateway] allow getting a deployment status using the deployment token - https://phabricator.wikimedia.org/T398623#10984428 (10dcaro) a:03dcaro [15:00:07] 10Toolforge (Toolforge iteration 21): [components-api,api-gateway] allow getting a deployment status using the deployment token - https://phabricator.wikimedia.org/T398623#10984430 (10dcaro) 05Open→03In progress [15:00:25] 10Toolforge (Toolforge iteration 21): [components-cli] Allow reading tool configuration from stdin - https://phabricator.wikimedia.org/T398424#10984433 (10dcaro) p:05Triage→03Medium [15:01:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: `toolforge jobs dump` fails for tools.stewardsbot - https://phabricator.wikimedia.org/T396210#10984444 (10dcaro) I think this can be closed now [15:01:13] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: `toolforge jobs dump` fails for tools.stewardsbot - https://phabricator.wikimedia.org/T396210#10984448 (10dcaro) 05In progress→03Resolved [15:02:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [15:02:15] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [15:03:42] (03PS10) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [15:03:48] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [15:04:02] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [15:04:38] (03PS11) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [15:04:45] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [15:05:29] 10Toolforge (Toolforge iteration 21): [infra] 2025-06-21 Several correlated potentially network issues during the night - https://phabricator.wikimedia.org/T397566#10984468 (10dcaro) The query logs don't occupy much, so I'll leave them "for the time being": ` root@tools-prometheus-9:~# ls -lh /var/log/prometheus... [15:07:32] 10Toolforge (Toolforge iteration 21): [infra] 2025-06-21 Several correlated potentially network issues during the night - https://phabricator.wikimedia.org/T397566#10984477 (10dcaro) Also, no new incidents happened, so let's wait a couple of weeks [15:08:30] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [15:10:22] (03PS12) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [15:11:49] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [15:14:11] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [15:24:04] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer [15:24:43] (03update) 10dcaro: toolforge_get_versions: fix cli version detection [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/870 [15:25:00] (03merge) 10dcaro: toolforge_get_versions: fix cli version detection [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/870 [15:25:11] (03update) 10dcaro: [lima-kilo.toolforge] ensure use of amd64 arch [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/249 (https://phabricator.wikimedia.org/T398016) (owner: 10raymond-ndibe) [15:25:25] (03close) 10dcaro: [lima-kilo.toolforge] ensure use of amd64 arch [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/249 (https://phabricator.wikimedia.org/T398016) (owner: 10raymond-ndibe) [15:25:36] (03merge) 10dcaro: openapi: added several servers [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/107 [15:26:15] (03update) 10dcaro: registry: move to the new registry url [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/13 [15:26:38] (03update) 10dcaro: maintain-kubeusers: bump to 0.0.178-20250702084425-15f2dd20 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/860 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:26:59] (03merge) 10dcaro: maintain-kubeusers: bump to 0.0.178-20250702084425-15f2dd20 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/860 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [15:28:19] (03open) 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620: components-api: bump to 0.0.133-20250708152552-264f918e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/873 [15:41:24] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer [15:46:03] (03PS13) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [15:50:03] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [15:59:56] (03open) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [15:59:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [16:00:38] (03approved) 10dcaro: jobs-emailer: bump to 0.0.62-20250708132525-c01c0ea7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/872 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:00:50] (03update) 10dcaro: jobs-emailer: bump to 0.0.62-20250708132525-c01c0ea7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/872 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:01:08] (03merge) 10dcaro: jobs-emailer: bump to 0.0.62-20250708132525-c01c0ea7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/872 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [16:10:50] (03PS14) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [16:15:17] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [16:15:50] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [16:20:06] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [16:24:43] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [16:28:56] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team: Add the 'other assignee' field to the Phabricator test instance - https://phabricator.wikimedia.org/T398732#10984982 (10Dzahn) 05Open→03Resolved Thank you. I will call it resolved. The paste you linked to does not seem... [16:31:41] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [16:37:02] 10VPS-project-Phabricator, 06collaboration-services, 06Release-Engineering-Team: Add the 'other assignee' field to the Phabricator test instance - https://phabricator.wikimedia.org/T398732#10985108 (10Aklapper) >>! In T398732#10981512, @Dzahn wrote: > You can be bold with this one. If you just want to ad... [16:43:55] (03PS1) 10Bovimacoco: fix: page views data not loading for selected project [labs/tools/mostvisitedarticle] - 10https://gerrit.wikimedia.org/r/1167263 (https://phabricator.wikimedia.org/T398342) [16:48:17] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [16:48:32] (03update) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118) [16:49:15] (03update) 10raymond-ndibe: [jobs-cli] quota refactor [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T389118) [16:50:27] (03update) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118) [16:50:46] (03update) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118) [16:52:14] (03PS2) 10Bovimacoco: fix: page views data not loading for selected project [labs/tools/mostvisitedarticle] - 10https://gerrit.wikimedia.org/r/1167263 (https://phabricator.wikimedia.org/T398342) [16:58:37] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [17:02:28] (03update) 10dcaro: components-api: bump to 0.0.133-20250708152552-264f918e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/873 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:02:47] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:07:38] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [17:08:11] (03update) 10raymond-ndibe: [jobs-cli] quota refactor [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/97 (https://phabricator.wikimedia.org/T389118) [17:10:57] 10Tool-community-activity-alerts-system: Send emails/alerts to community when there are peaks/abnormalities - https://phabricator.wikimedia.org/T398995 (10Kalli_navya) 03NEW [17:13:32] 10Tool-community-activity-alerts-system: Adding Annotation layer on the dashboard - https://phabricator.wikimedia.org/T398996 (10Kalli_navya) 03NEW [17:13:59] (03update) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118) [17:14:33] 10Tool-community-activity-alerts-system: Pandas to Polars migration - https://phabricator.wikimedia.org/T398997 (10Kalli_navya) 03NEW [17:14:53] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [17:14:53] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [17:15:37] 10Tool-community-activity-alerts-system: UI Enhancements in phase 1 - https://phabricator.wikimedia.org/T398998 (10Kalli_navya) 03NEW [17:17:26] 10Tool-community-activity-alerts-system: Extending the dashboard to report abonormalities in editor counts - https://phabricator.wikimedia.org/T398999 (10Kalli_navya) 03NEW [17:18:40] 10Tool-community-activity-alerts-system: Experiment with alerting and anomaly detection techniques - https://phabricator.wikimedia.org/T399000 (10Kalli_navya) 03NEW [17:19:18] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [17:19:55] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [17:20:17] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [17:23:42] (03approved) 10dcaro: components-api: bump to 0.0.133-20250708152552-264f918e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/873 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:23:45] (03merge) 10dcaro: components-api: bump to 0.0.133-20250708152552-264f918e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/873 (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:26:54] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [17:27:26] (03update) 10dcaro: Draft: dcaro test [repos/cloud/toolforge/jobs-api] (fix_diff_bug) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/182 [17:28:14] 10cloud-services-team (FY2024/2025-Q3-Q4): Create WMCS offboarding checklist - https://phabricator.wikimedia.org/T398972#10985448 (10fnegri) I created a first draft at https://www.mediawiki.org/wiki/Wikimedia_Cloud_Services_team/Offboarding_template [17:29:44] 10cloud-services-team (FY2024/2025-Q3-Q4): Create WMCS offboarding checklist - https://phabricator.wikimedia.org/T398972#10985452 (10fnegri) 05Open→03In progress p:05Triage→03Medium [17:31:03] 10cloud-services-team (FY2024/2025-Q3-Q4): Create WMCS offboarding checklist - https://phabricator.wikimedia.org/T398972#10985461 (10fnegri) [17:32:32] (03update) 10raymond-ndibe: [jobs-api] refactor quota models [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/164 (https://phabricator.wikimedia.org/T389118) [17:43:07] (03close) 10raymond-ndibe: [quota] inc deprecated counter for quota requests [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/181 (https://phabricator.wikimedia.org/T389118) [17:44:40] 06cloud-services-team, 10Toolforge, 13Patch-For-Review: [o11y,logging,infra] Deploy Loki to store Toolforge tool log data - https://phabricator.wikimedia.org/T386480#10985541 (10taavi) 05Open→03Resolved [17:49:00] (03PS15) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [18:19:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:20:30] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [18:24:00] 06cloud-services-team, 10Toolforge (Toolforge iteration 21), 13Patch-For-Review: toolsbeta harbor disk full - https://phabricator.wikimedia.org/T398715#10985648 (10Raymond_Ndibe) [18:25:58] (03approved) 10raymond-ndibe: build: enable ci builds [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) (owner: 10dcaro) [18:25:59] (03update) 10raymond-ndibe: build: enable ci builds [repos/cloud/toolforge/misctools-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/misctools-cli/-/merge_requests/1 (https://phabricator.wikimedia.org/T398202) (owner: 10dcaro) [18:41:53] (03PS16) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [18:42:37] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:42:44] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [18:44:55] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:45:03] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [18:45:29] (03PS17) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [18:45:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:45:42] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [18:46:48] (03PS18) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [18:46:53] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:47:00] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [18:48:26] (03PS19) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [18:48:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:48:37] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [18:51:01] (03PS20) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [18:51:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:51:19] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [18:53:46] (03PS21) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [18:53:57] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:54:05] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [18:55:12] (03PS22) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [18:55:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:55:34] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [18:56:33] (03PS23) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [18:56:39] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:56:46] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:00:30] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [19:10:00] (03PS24) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:10:04] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:10:21] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:11:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:12:29] (03PS25) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:12:31] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:12:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:13:16] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:15:07] (03PS26) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:15:13] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:15:20] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:16:39] (03PS27) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:17:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:17:29] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:19:45] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:20:38] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:20:41] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:21:23] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:22:33] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:23:07] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:23:51] (03PS29) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:24:40] (03PS30) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:24:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:24:57] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:27:50] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [19:27:53] (03PS31) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:27:59] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:28:08] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:31:44] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [19:31:46] (03PS32) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:33:42] (03PS33) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:33:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:33:54] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:34:12] (03PS34) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:34:17] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:34:25] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:36:27] (03PS35) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:36:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:36:47] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:38:52] (03PS36) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:38:57] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:42:07] andrew@cloudcumin1001 reactivate (PID 1938412) is awaiting input [19:42:27] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [19:51:04] (03PS37) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:51:08] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:51:12] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:51:25] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:54:32] (03PS38) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [19:54:43] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:55:11] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:57:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:58:02] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [19:58:16] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [20:31:29] (03PS39) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [20:31:35] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:31:54] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [20:33:24] (03PS40) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [20:33:31] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:33:43] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [20:35:42] (03PS41) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [20:35:48] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:39:02] andrew@cloudcumin1001 reactivate (PID 1943907) is awaiting input [20:39:31] (03CR) 10CI reject: [V:04-1] Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 (owner: 10Andrew Bogott) [20:44:34] (03PS42) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [20:44:39] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [20:44:42] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:44:48] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [20:45:40] (03PS43) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [20:45:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:46:13] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [20:48:40] (03PS44) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [20:48:53] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:49:20] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [20:50:11] (03PS45) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [20:50:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:50:32] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [20:50:47] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:51:11] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [20:52:11] (03PS46) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [20:52:25] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:52:36] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [20:53:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:54:08] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [21:02:36] (03PS47) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [21:03:45] (03PS48) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [21:05:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [21:06:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [21:06:11] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [21:06:23] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [21:09:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [21:09:33] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [21:11:12] (03PS49) 10Andrew Bogott: Add 'reactivate' cookbook [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1166955 [21:11:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [21:11:26] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [21:11:46] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [21:11:54] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [21:29:38] 06cloud-services-team: MaxConntrack Max conntrack at 92.93% on cloudvirt1070:9100 - https://phabricator.wikimedia.org/T398139#10986426 (10Andrew) 05Open→03Resolved a:03Andrew [21:30:52] 06cloud-services-team: PuppetFailure Puppet has failed on cloudcontrol2005-dev:9100 - https://phabricator.wikimedia.org/T398642#10986430 (10Andrew) 05Open→03Resolved a:03Andrew [21:45:34] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10986464 (10Jclark-ctr) @dcaro @Andrew @cmooney @ayounsi I need some assistance. I need to open a block of 4x ports on cloudsw1-f4-eqiad. The least dis... [21:58:16] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[48-51] - https://phabricator.wikimedia.org/T394333#10986480 (10Jclark-ctr) [23:08:22] (03PS3) 10SomeRandomDeveloper: Replace divs with semantic elements [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1091830 (https://phabricator.wikimedia.org/T227631) [23:08:46] (03CR) 10CI reject: [V:04-1] Replace divs with semantic elements [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1091830 (https://phabricator.wikimedia.org/T227631) (owner: 10SomeRandomDeveloper) [23:10:20] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [23:11:01] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [23:11:30] (03PS4) 10SomeRandomDeveloper: Replace divs with semantic elements [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1091830 (https://phabricator.wikimedia.org/T227631) [23:11:54] (03CR) 10CI reject: [V:04-1] Replace divs with semantic elements [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1091830 (https://phabricator.wikimedia.org/T227631) (owner: 10SomeRandomDeveloper)