[08:51:12] !log admin icinga downtime cloudvirt1025/1026 for reboots [08:51:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:00:17] !log admin rebooting cloudvirt1025 for kernel upgrade [09:00:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:03:07] !log paws paws-master-01/03 and a couple of other servers are down because hypervisor is rebooting [09:03:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [09:03:33] !log tools tools-sgebastion-08 is down because hypervisor is rebooting [09:03:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:09:37] !log admin cloudvirt1025 reboot ended OK [09:09:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:12:21] !log admin rebooting cloudvirt1026 for kernel upgrade [09:12:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:13:11] !log tools 9 tools-sgeexec nodes and 6 other related VMs are down because hypervisor is rebooting [09:13:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:23:06] !log admin cloudvirt1026 reboot ended OK [09:23:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:41:12] !log cloudinfra manually start mariadb in cloudinfra-db01 [09:41:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [09:41:30] !log cloudinfra the cloudinfra-db01 VM was rebooted bc the hypervisor rebooted [09:41:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [09:54:59] !log cloudinfra manually restart mariadb in cloudinfra-db02 to fix replication [09:55:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL [11:10:20] !log toolsbeta re-create VM `toolsbeta-test-k8s-haproxy-2` to test https://gerrit.wikimedia.org/r/545532 (T236074) [11:10:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:10:24] T236074: Toolforge: rebuild the new k8s toolsbeta deployment and write final docs - https://phabricator.wikimedia.org/T236074 [11:19:59] !log toolsbeta re-create VM `toolsbeta-test-k8s-haproxy-1` to use new puppet profile (T236074) [11:20:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:20:02] T236074: Toolforge: rebuild the new k8s toolsbeta deployment and write final docs - https://phabricator.wikimedia.org/T236074 [11:56:37] !log toolsbeta point FQDN `k8s.toolsbeta.eqiad1.wikimedia.cloud` to `toolsbeta-test-k8s-haproxy-1` (T236074) [11:56:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:56:41] T236074: Toolforge: rebuild the new k8s toolsbeta deployment and write final docs - https://phabricator.wikimedia.org/T236074 [12:04:07] !log toolsbeta created 2 new VMs `toolsbeta-test-k8s-worker-[1,2]` T236074 [12:04:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:04:10] T236074: Toolforge: rebuild the new k8s toolsbeta deployment and write final docs - https://phabricator.wikimedia.org/T236074 [12:09:24] !log tools Deployed toollabs-webservice 0.47 to buster-tools and stretch-tools (T233347) [12:09:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:09:28] T233347: Remove access.log generation from default lighttpd.conf generated by `webservice` - https://phabricator.wikimedia.org/T233347 [14:01:01] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:50:45] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @Lucas_WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [20:00:43] !log tools Rebuilding all jessie and stretch docker images to pick up toollabs-webservice 0.47 (T233347) [20:00:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:00:47] T233347: Remove access.log generation from default lighttpd.conf generated by `webservice` - https://phabricator.wikimedia.org/T233347 [21:11:35] !log tools.wd-image-positions deployed ceb0b0cf8f (Structured Data on Commons support, use cropperjs; also tried a general pip upgrade but that broke OAuth login so downgraded to previous versions, except pip itself) [21:11:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [23:35:07] !log tools.stashbot Deployed c8951e6 (sal.py: Escape single curly braces in template args) {{phabT|235825}} [23:35:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [23:35:51] hmmmm... that's not exactly what I expected :/ [23:36:38] my hack template arg sanitization isn't quite right [23:36:57] and I think it was broken before and I just never noticed