[02:39:36] !log lewton-test deleted project per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge [02:39:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Lewton-test/SAL [02:41:47] !log mcr-dev deleting all VMs and projects as per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge [02:41:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mcr-dev/SAL [02:45:06] !log queryrapi deleting project per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge [02:45:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Queryrapi/SAL [02:50:45] !log structurednavigation deleting project per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge [02:50:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Structurednavigation/SAL [02:52:16] !log test-twemproxy deleting VMs and project per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge [02:52:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Test-twemproxy/SAL [02:55:31] !log wikifactmine project per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge [02:55:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikifactmine/SAL [02:58:04] !log wpx deleting VMs and project per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2019_Purge [02:58:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wpx/SAL [13:34:38] !log admin [¢odfw1dev] prevent neutron from allocating floating IPs from the wrong subnet by doing `neutron subnet-update --allocation-pool start=208.80.153.190,end=208.80.153.190 cloud-instances-transport1-b-codfw` (T242594) [13:34:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:34:41] T242594: Unable to access codfw1-dev VMs via ssh - https://phabricator.wikimedia.org/T242594 [14:47:54] !log openstack finished testing ceph on cloudvirt1022, revert to local disk puppet role and recreate canary instance T225320 [14:47:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [14:47:56] T225320: Ceph Proof of Concept Build and Testing - https://phabricator.wikimedia.org/T225320 [16:26:20] !log tools Cordoned and fixed puppet on tools-k8s-worker-9. Rebooting now. T242559 [16:26:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:26:24] T242559: Partialy setup tools-k8s-worker instances created by novaadmin causing problems - https://phabricator.wikimedia.org/T242559 [16:31:03] !log tools Cordoned and fixed puppet on tools-k8s-worker-10. Rebooting now. T242559 [16:31:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:33:49] !log tools Cordoned and fixed puppet on tools-k8s-worker-11. Rebooting now. T242559 [16:33:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:33:53] T242559: Partialy setup tools-k8s-worker instances created by novaadmin causing problems - https://phabricator.wikimedia.org/T242559 [16:41:20] !log toolsbeta There was a filesystem unclean and other problems on the "old cluster" worker node 1001. Rebooting it in case that helps. [16:41:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:42:27] !log tools Cordoned and fixed puppet on tools-k8s-worker-12. Rebooting now. T242559 [16:42:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:42:30] T242559: Partialy setup tools-k8s-worker instances created by novaadmin causing problems - https://phabricator.wikimedia.org/T242559 [17:48:57] !log tools Running `puppet ca destroy` for each unsigned cert on tools-puppetmaster-01 (T242642) [17:49:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:49:00] T242642: Cleanup unsigned puppet client certs on tools-puppetmaster-01 - https://phabricator.wikimedia.org/T242642 [18:12:22] !log tools.versions `kubectl delete pod versions-57b986d64f-7fdmd` to test latest php7.3 image [18:12:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.versions/SAL [18:54:48] !log tools.quarry Moved redirect only webservice from grid engine to 2020 Kubernetes cluster using technique from https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Make_a_tool_redirect_to_another_tool_WITHOUT_running_a_webservice (T242538) [18:54:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quarry/SAL [18:54:51] T242538: Many grid engine backend webservices not registered at tools-proxy redis following depool restarts - https://phabricator.wikimedia.org/T242538 [20:29:50] !log tools.ato Restarted webservice to fix broken registration with the front proxy (T242538) [20:29:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ato/SAL [20:29:53] T242538: Many grid engine backend webservices not registered at tools-proxy redis following depool restarts - https://phabricator.wikimedia.org/T242538 [20:40:20] !log tools.blockyquery Restarted webservice to fix broken registration with the front proxy (T242538) [20:40:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.blockyquery/SAL [20:40:23] T242538: Many grid engine backend webservices not registered at tools-proxy redis following depool restarts - https://phabricator.wikimedia.org/T242538 [20:47:17] !log tools.botriconferme Restarted webservice to fix broken registration with the front proxy (T242538) [20:47:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.botriconferme/SAL [20:47:20] T242538: Many grid engine backend webservices not registered at tools-proxy redis following depool restarts - https://phabricator.wikimedia.org/T242538 [20:50:43] !log tools.bd808-test Testing a fancy !log trick [20:50:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bd808-test/SAL [20:52:43] * bd808 tries to script a thing he has never scripted before [20:52:49] !log tools.catgraph Restarted webservice to fix broken registration with the front proxy (T242538) [20:52:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.catgraph/SAL [20:53:19] !log tools.cgstat Restarted webservice to fix broken registration with the front proxy (T242538) [20:53:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cgstat/SAL [20:53:48] !log tools.cluebotng Restarted webservice to fix broken registration with the front proxy (T242538) [20:53:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [20:54:20] !log tools.convert Restarted webservice to fix broken registration with the front proxy (T242538) [20:54:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.convert/SAL [20:54:50] !log tools.deadlinks Restarted webservice to fix broken registration with the front proxy (T242538) [20:54:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.deadlinks/SAL [20:55:19] !log tools.derivative Restarted webservice to fix broken registration with the front proxy (T242538) [20:55:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.derivative/SAL [20:55:24] zhuyifei1999_: FYI I moved the tool for quarry from the grid to the 2020 Kubernetes cluster and replaced the redirect functionality with a bare ingress -- https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Make_a_tool_redirect_to_another_tool_WITHOUT_running_a_webservice [20:55:51] !log tools.dewikinews-rss Restarted webservice to fix broken registration with the front proxy (T242538) [20:55:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dewikinews-rss/SAL [20:56:19] !log tools.dispenser Restarted webservice to fix broken registration with the front proxy (T242538) [20:56:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dispenser/SAL [20:56:49] !log tools.dow Restarted webservice to fix broken registration with the front proxy (T242538) [20:56:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dow/SAL [20:57:22] !log tools.fountain-test Restarted webservice to fix broken registration with the front proxy (T242538) [20:57:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fountain-test/SAL [20:57:48] !log tools.freddy2001 Restarted webservice to fix broken registration with the front proxy (T242538) [20:57:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.freddy2001/SAL [20:58:04] !log tools.gerakitools Restarted webservice to fix broken registration with the front proxy (T242538) [20:58:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.gerakitools/SAL [20:58:34] !log tools.germancontributioncounts Restarted webservice to fix broken registration with the front proxy (T242538) [20:58:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.germancontributioncounts/SAL [20:59:03] !log tools.grantmetrics Restarted webservice to fix broken registration with the front proxy (T242538) [20:59:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.grantmetrics/SAL [20:59:34] !log tools.gyan Restarted webservice to fix broken registration with the front proxy (T242538) [20:59:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.gyan/SAL [21:00:04] !log tools.igloo Restarted webservice to fix broken registration with the front proxy (T242538) [21:00:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.igloo/SAL [21:00:35] !log tools.inactiveadmins Restarted webservice to fix broken registration with the front proxy (T242538) [21:00:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.inactiveadmins/SAL [21:01:04] !log tools.jimmy Restarted webservice to fix broken registration with the front proxy (T242538) [21:01:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jimmy/SAL [21:01:33] !log tools.khanomalumat Restarted webservice to fix broken registration with the front proxy (T242538) [21:01:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.khanomalumat/SAL [21:02:04] !log tools.linedwell Restarted webservice to fix broken registration with the front proxy (T242538) [21:02:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.linedwell/SAL [21:02:38] !log tools.mediaviews Restarted webservice to fix broken registration with the front proxy (T242538) [21:02:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mediaviews/SAL [21:03:04] !log tools.metaviews Restarted webservice to fix broken registration with the front proxy (T242538) [21:03:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.metaviews/SAL [21:03:34] !log tools.mostlinkedmissing Restarted webservice to fix broken registration with the front proxy (T242538) [21:03:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mostlinkedmissing/SAL [21:04:04] !log tools.mrmetadata Restarted webservice to fix broken registration with the front proxy (T242538) [21:04:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mrmetadata/SAL [21:04:34] !log tools.musikanimal Restarted webservice to fix broken registration with the front proxy (T242538) [21:04:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.musikanimal/SAL [21:04:37] !log tools.omarghridabot Restarted webservice to fix broken registration with the front proxy (T242538) [21:04:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.omarghridabot/SAL [21:05:04] !log tools.osmlint Restarted webservice to fix broken registration with the front proxy (T242538) [21:05:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.osmlint/SAL [21:05:34] !log tools.patrolstats Restarted webservice to fix broken registration with the front proxy (T242538) [21:05:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.patrolstats/SAL [21:06:04] !log tools.poiimport Restarted webservice to fix broken registration with the front proxy (T242538) [21:06:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.poiimport/SAL [21:06:34] !log tools.portal Restarted webservice to fix broken registration with the front proxy (T242538) [21:06:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.portal/SAL [21:07:04] !log tools.ptwikis Restarted webservice to fix broken registration with the front proxy (T242538) [21:07:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ptwikis/SAL [21:07:33] !log tools.render-tests Restarted webservice to fix broken registration with the front proxy (T242538) [21:07:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.render-tests/SAL [21:08:07] !log tools.rotbot Restarted webservice to fix broken registration with the front proxy (T242538) [21:08:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.rotbot/SAL [21:08:34] !log tools.russbot Restarted webservice to fix broken registration with the front proxy (T242538) [21:08:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.russbot/SAL [21:09:03] !log tools.searchsbl Restarted webservice to fix broken registration with the front proxy (T242538) [21:09:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.searchsbl/SAL [21:09:33] !log tools.shrinitools Restarted webservice to fix broken registration with the front proxy (T242538) [21:09:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.shrinitools/SAL [21:09:49] !log tools.shuaib Restarted webservice to fix broken registration with the front proxy (T242538) [21:09:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.shuaib/SAL [21:10:19] !log tools.shuaib-bot Restarted webservice to fix broken registration with the front proxy (T242538) [21:10:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.shuaib-bot/SAL [21:10:49] !log tools.sign-language-browser Restarted webservice to fix broken registration with the front proxy (T242538) [21:10:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sign-language-browser/SAL [21:11:19] !log tools.slumpartikel Restarted webservice to fix broken registration with the front proxy (T242538) [21:11:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.slumpartikel/SAL [21:11:25] such bot logging. much wow [21:11:50] !log tools.soweego Restarted webservice to fix broken registration with the front proxy (T242538) [21:11:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.soweego/SAL [21:12:19] !log tools.stockholm-mania Restarted webservice to fix broken registration with the front proxy (T242538) [21:12:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stockholm-mania/SAL [21:12:49] !log tools.tessdata Restarted webservice to fix broken registration with the front proxy (T242538) [21:12:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tessdata/SAL [21:13:22] !log tools.timerelengteam Restarted webservice to fix broken registration with the front proxy (T242538) [21:13:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.timerelengteam/SAL [21:13:49] !log tools.title-search Restarted webservice to fix broken registration with the front proxy (T242538) [21:13:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.title-search/SAL [21:14:18] !log tools.toolhub Restarted webservice to fix broken registration with the front proxy (T242538) [21:14:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolhub/SAL [21:14:48] !log tools.toolschecker-ge-ws Restarted webservice to fix broken registration with the front proxy (T242538) [21:14:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolschecker-ge-ws/SAL [21:15:18] !log tools.tulsibot Restarted webservice to fix broken registration with the front proxy (T242538) [21:15:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tulsibot/SAL [21:15:50] !log tools.validator Restarted webservice to fix broken registration with the front proxy (T242538) [21:15:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.validator/SAL [21:16:19] !log tools.vvoters Restarted webservice to fix broken registration with the front proxy (T242538) [21:16:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.vvoters/SAL [21:16:48] !log tools.wahldiagramm Restarted webservice to fix broken registration with the front proxy (T242538) [21:16:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wahldiagramm/SAL [21:17:18] !log tools.wdmap Restarted webservice to fix broken registration with the front proxy (T242538) [21:17:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmap/SAL [21:17:48] !log tools.wikidata-timeline Restarted webservice to fix broken registration with the front proxy (T242538) [21:17:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikidata-timeline/SAL [21:18:20] !log tools.wikiedudashboard-test Restarted webservice to fix broken registration with the front proxy (T242538) [21:18:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiedudashboard-test/SAL [21:18:52] !log tools.wikilinkbot Restarted webservice to fix broken registration with the front proxy (T242538) [21:18:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikilinkbot/SAL [21:19:18] !log tools.wptestblog2 Restarted webservice to fix broken registration with the front proxy (T242538) [21:19:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wptestblog2/SAL [21:19:50] !log tools.wscontest Restarted webservice to fix broken registration with the front proxy (T242538) [21:19:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wscontest/SAL [21:20:18] !log tools.yemen Restarted webservice to fix broken registration with the front proxy (T242538) [21:20:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.yemen/SAL [21:20:49] !log tools.zhdeletionpedia Restarted webservice to fix broken registration with the front proxy (T242538) [21:20:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zhdeletionpedia/SAL [21:21:19] !log tools.zhwiki-qualifications-check Restarted webservice to fix broken registration with the front proxy (T242538) [21:21:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zhwiki-qualifications-check/SAL [21:21:44] victory! And only 2 of them are still obviously busted [22:02:14] log tools.convert Cleaned up old log files as part of debugging webservice 503 response after restart [22:02:52] missed the !log :) [22:04:12] !log tools.convert Cleaned up old log files as part of debugging webservice 503 response after restart [22:04:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.convert/SAL [22:04:17] thanks paladox :) [22:04:23] yw :) [22:09:43] !log tools.dewikinews-rss Deleted old access and error logs as part of debugging webservice 503 response after restart [22:09:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dewikinews-rss/SAL [22:39:14] !log tools.dewikinews-rss Hot patched deployment with fix proposed at https://github.com/dapete42/dewikinews-rss/pull/2 [22:39:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dewikinews-rss/SAL [22:39:24] blerg [22:39:27] !log tools.dewikinews-rss Hot patched deployment with fix proposed at https://github.com/dapete42/dewikinews-rss/pull/2 (T242680) [22:39:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dewikinews-rss/SAL [22:39:29] T242680: dewikinews-rss: Huge $HOME/error.log output for each page load - https://phabricator.wikimedia.org/T242680 [22:49:21] !log tools.omarghridabot Stopped "replace.py" job stuck in 'qw' status due to missing queue name [22:49:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.omarghridabot/SAL