[09:54:56] !log admin icinga downtime cloudvirt1025 for 6 days (T269313) [09:54:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:54:59] T269313: cloudvirt1025 connection issues on primary nic - https://phabricator.wikimedia.org/T269313 [10:12:16] !log toolsbeta Trying to create a whole new etcd member from ansible (T267412) [10:12:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:12:21] T267412: Provisioning/maintenance scripts - https://phabricator.wikimedia.org/T267412 [10:23:41] !log admin setting expiration to 2020-12-03 to the oldest backy snapshot of every VM in cloudvirt1024 (T269419) [10:23:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:23:45] T269419: cloudvirt1024: /srv full 99% - https://phabricator.wikimedia.org/T269419 [10:28:47] !log admin manually running `wmcs-purge-backups` on cloudvirt1024 (T269419) [10:28:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:28:52] T269419: cloudvirt1024: /srv full 99% - https://phabricator.wikimedia.org/T269419 [11:17:26] !log toolsbeta Created a new 'standardized' security froup for k8s from ansible toolsbeta-k8s-full-connectivity (T267412) [11:17:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:17:30] T267412: Provisioning/maintenance scripts - https://phabricator.wikimedia.org/T267412 [11:24:34] !log admin icinga downtime cloudvirt1024 for 6 days, to avoid paging noises (T269313) [11:24:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:24:38] T269313: cloudvirt1025 connection issues on primary nic - https://phabricator.wikimedia.org/T269313 [11:25:36] !log admin last log line referencing cloudvirt1024 is a mistake (T269313) [11:25:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:25:39] !log admin icinga downtime cloudvirt1024 for 6 days, to avoid paging noises (T269419) [11:25:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:25:42] T269419: cloudvirt1024: /srv full 99% - https://phabricator.wikimedia.org/T269419 [12:12:49] !log admin manually running `wmcs-purge-backups` again on cloudvirt1024 (T269419) [12:12:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:12:53] T269419: cloudvirt1024: /srv full 99% - https://phabricator.wikimedia.org/T269419 [17:19:53] any one there? [20:48:30] !log putting cloudvirt1025 and 1026 back into service because I'm pretty sure they're fixed. T269313 [20:48:31] andrewbogott: Unknown project "putting" [20:48:31] T269313: cloudvirt10[25-30] connection issues on primary nic - https://phabricator.wikimedia.org/T269313 [21:06:54] !log admin putting cloudvirt1025 and 1026 back into service because I'm pretty sure they're fixed. T269313 [21:06:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:06:58] T269313: cloudvirt10[25-30] connection issues on primary nic - https://phabricator.wikimedia.org/T269313 [22:19:35] !log mwoffliner paysing mwoffliner3 for migration do a different hypervisor [22:19:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwoffliner/SAL [22:23:59] !log admin moving cloudvirt1023 out of the ceph aggregate and into maintenance for T269467 [22:24:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:24:02] T269467: Upgrade nic firmware on cloudvirt1023 - https://phabricator.wikimedia.org/T269467 [22:33:54] !log admin moving cloudvirt1023 back into the ceph aggregate; it doesn't need upgrades after all T269467 [22:33:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:33:58] T269467: Upgrade nic firmware on cloudvirt1023 - https://phabricator.wikimedia.org/T269467