[00:58:19] !log tools.translation-server Restarted pod per https://en.wikipedia.org/wiki/User_talk:Krenair#Citation_bot - seemed to be stuck, though now it just appears to return its own 404 [00:58:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translation-server/SAL [01:10:19] !log tools.translation-server (I assumed it presented a UI, in fact this API appears to be working now.) [01:10:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translation-server/SAL [01:11:06] !log tools.translation-server ((But maybe should be returning HTTP 405 rather than 404 in this case?)) [01:11:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translation-server/SAL [04:41:27] !log tools Rebuilding all Docker images to pick up webservice-python-bootstrap changes [04:41:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [05:23:53] !log tools Building 6 new tools-k8s-worker instances for the 2020 Kubernetes cluster (take 2) [05:23:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:52:52] !log admin repooling cloudvirt1013 after HW got fixed (T241313) [12:52:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:52:55] T241313: cloudvirt1013: server down for no reason (power issue?) - https://phabricator.wikimedia.org/T241313 [15:10:53] !log admin remove icinga downtime for cloudvirt1013 T241313 [15:10:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:10:56] T241313: cloudvirt1013: server down for no reason (power issue?) - https://phabricator.wikimedia.org/T241313 [15:45:33] !log tools Rebuilding all Docker containers again because I failed to actually update the build server git clone properly last time I did this [15:45:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:57:33] Forgive me if this has already been answered, but what's up with replag on s4 (commonswiki)? Looks like it's 4 days and counting... [16:03:54] mdaniels5757: T243488 -- TL;DR there was table compression happening for wikibase related tables on commonswiki. That finished and things are catching up now. [16:03:54] T243488: Wiki replicas replication lag for the s4 slice (commonswiki) is over 80 hours - https://phabricator.wikimedia.org/T243488 [16:04:15] OK, thanks so much! [19:29:15] !log openstack upgrade cloudcontrol100[34] to the latest python3 openstack clients in stretch [19:29:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [20:58:45] !log tools Built tools-k8s-worker-21 to test out build script following openstack client upgrade [20:58:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:15:46] !log deployment-prep deploying ores 283f627 [21:15:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [21:23:21] !log tools.refill-api Added resources limit range to the celery container per T243465 [21:23:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.refill-api/SAL [21:23:24] T243465: refill-api memory usage is somewhat dangerous to the Kubernetes cluster on Toolforge - https://phabricator.wikimedia.org/T243465 [23:08:08] |log tools.indic-wscontest Starting webservice (T241537) [23:08:10] T241537: Create: indic-wscontest tool - https://phabricator.wikimedia.org/T241537 [23:13:08] !log tools.indic-wscontest Starting webservice (T241537) [23:13:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.indic-wscontest/SAL [23:13:10] T241537: Create: indic-wscontest tool - https://phabricator.wikimedia.org/T241537