[00:00:24] !log tools.wikifactmine-api Migrated to 2020 Kubernetes cluster [00:00:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikifactmine-api/SAL [00:01:18] !log tools.wikifile-transfer Migrated to 2020 Kubernetes cluster [00:01:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikifile-transfer/SAL [00:02:10] !log tools.wikihistory Migrated to 2020 Kubernetes cluster [00:02:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikihistory/SAL [00:03:06] !log tools.wikiinfo Migrated to 2020 Kubernetes cluster [00:03:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiinfo/SAL [00:04:27] !log tools.wikiloop Migrated to 2020 Kubernetes cluster [00:04:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloop/SAL [00:05:24] !log tools.wikiloves Migrated to 2020 Kubernetes cluster [00:05:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL [00:06:02] !log tools.wikilovesdownloads Migrated to 2020 Kubernetes cluster [00:06:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikilovesdownloads/SAL [00:07:00] !log tools.wikimap Migrated to 2020 Kubernetes cluster [00:07:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimap/SAL [00:07:57] !log tools.wikinity-test Migrated to 2020 Kubernetes cluster [00:07:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikinity-test/SAL [00:09:26] !log tools.wikipathways2wiki Migrated to 2020 Kubernetes cluster [00:09:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikipathways2wiki/SAL [00:10:25] !log tools.wikipedia-fetch-content Migrated to 2020 Kubernetes cluster [00:10:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikipedia-fetch-content/SAL [00:11:03] !log tools.wikipedia-readability Migrated to 2020 Kubernetes cluster [00:11:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikipedia-readability/SAL [00:11:51] !log tools.wikiprovenance Migrated to 2020 Kubernetes cluster [00:11:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiprovenance/SAL [00:12:48] !log tools.wikiradio Migrated to 2020 Kubernetes cluster [00:12:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiradio/SAL [00:13:45] !log tools.wikishield Migrated to 2020 Kubernetes cluster [00:13:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikishield/SAL [00:15:01] !log tools.wikisource-penguin-classics Migrated to 2020 Kubernetes cluster [00:15:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikisource-penguin-classics/SAL [00:23:03] !log tools.wikisource-penguin-classics Stopped PHP 5.6 webservice due to application crashes (T246556) [00:23:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikisource-penguin-classics/SAL [00:24:10] !log tools.wikistats Migrated to 2020 Kubernetes cluster [00:24:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikistats/SAL [00:44:55] !log tools.wikitei Migrated to 2020 Kubernetes cluster [00:44:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikitei/SAL [00:46:50] !log tools.wikitools Migrated to 2020 Kubernetes cluster [00:46:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikitools/SAL [00:47:34] !log tools.wikitweets Migrated to 2020 Kubernetes cluster [00:47:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikitweets/SAL [00:48:10] !log tools.wikivoyage Migrated to 2020 Kubernetes cluster [00:48:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikivoyage/SAL [00:48:52] !log tools.wikiwhatsappbot Migrated to 2020 Kubernetes cluster [00:48:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiwhatsappbot/SAL [00:50:27] !log tools.wiktioutils Migrated to 2020 Kubernetes cluster [00:50:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wiktioutils/SAL [00:51:15] !log tools.wikyrillomat Migrated to 2020 Kubernetes cluster [00:51:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikyrillomat/SAL [00:51:33] * bd808 wonders why so many tools in the "w" batch were skipped by the automation script [00:52:19] !log tools.winter Migrated to 2020 Kubernetes cluster [00:52:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.winter/SAL [00:54:01] !log tools.wiwosm Migrated to 2020 Kubernetes cluster [00:54:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wiwosm/SAL [00:54:48] !log tools.wle Migrated to 2020 Kubernetes cluster [00:54:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wle/SAL [00:55:20] !log tools.wlm-maps Migrated to 2020 Kubernetes cluster [00:55:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wlm-maps/SAL [00:55:47] !log tools.wlm-stats Migrated to 2020 Kubernetes cluster [00:55:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wlm-stats/SAL [00:56:23] !log tools.wlm-us Migrated to 2020 Kubernetes cluster [00:56:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wlm-us/SAL [00:56:51] !log tools.wlmtrafo Migrated to 2020 Kubernetes cluster [00:56:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wlmtrafo/SAL [00:57:32] !log tools.wmcharts Migrated to 2020 Kubernetes cluster [00:57:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmcharts/SAL [00:58:09] !log tools.wmcounter Migrated to 2020 Kubernetes cluster [00:58:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmcounter/SAL [00:58:42] !log tools.wmcz Migrated to 2020 Kubernetes cluster [00:58:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmcz/SAL [00:59:38] !log tools.wmde-access Migrated to 2020 Kubernetes cluster [00:59:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmde-access/SAL [01:00:17] !log tools.wmde-reference-previews Migrated to 2020 Kubernetes cluster [01:00:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmde-reference-previews/SAL [01:00:48] !log tools.wmf-task-samtar Migrated to 2020 Kubernetes cluster [01:00:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmf-task-samtar/SAL [01:01:35] !log tools.wmpt Migrated to 2020 Kubernetes cluster [01:01:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmpt/SAL [01:02:13] !log tools.wmukevents Migrated to 2020 Kubernetes cluster [01:02:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmukevents/SAL [01:03:03] !log tools.worklist-tool Migrated to 2020 Kubernetes cluster [01:03:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.worklist-tool/SAL [01:03:38] !log tools.wp-world Migrated to 2020 Kubernetes cluster [01:03:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wp-world/SAL [01:04:12] !log tools.wpcleaner Migrated to 2020 Kubernetes cluster [01:04:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wpcleaner/SAL [01:04:38] !log tools.wprequests Migrated to 2020 Kubernetes cluster [01:04:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wprequests/SAL [01:05:06] !log tools.wptestblog Migrated to 2020 Kubernetes cluster [01:05:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wptestblog/SAL [01:05:57] !log tools.wpv Migrated to 2020 Kubernetes cluster [01:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wpv/SAL [01:06:34] !log tools.wrcp Migrated to 2020 Kubernetes cluster [01:06:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wrcp/SAL [01:07:01] !log tools.ws-google-ocr Migrated to 2020 Kubernetes cluster [01:07:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ws-google-ocr/SAL [01:07:43] !log tools.ws2wd Migrated to 2020 Kubernetes cluster [01:07:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ws2wd/SAL [01:08:12] !log tools.www-portal-builder Migrated to 2020 Kubernetes cluster [01:08:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.www-portal-builder/SAL [01:10:40] !log tools.zhaofeng-test Migrated to 2020 Kubernetes cluster [01:10:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zhaofeng-test/SAL [01:12:53] !log tools.zhaofeng-test Migrated custom /data/project/zhaofeng-test/refill-kube.yaml deployment to 2020 Kubernetes cluster [01:12:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zhaofeng-test/SAL [01:15:35] !log tools.zppixbot Migrated custom /data/project/zppixbot/k8s/sopelbot.yaml deployment to 2020 Kubernetes cluster [01:15:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [01:19:10] There is only one tool left running on the legacy Kubernetes cluster!!!1! [01:19:28] Y'all really can't know how happy I am right now :) [01:27:02] !log tools running the force-migrate command to make sure any new kubernetes deployments are on the new cluster. [01:27:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [01:39:17] !log tools.paws-public added a local version of kubectl 1.4.12 for talking to the old cluster in ~/bin/ [01:39:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [01:42:32] !log tools.paws-public set the current context to 'default' to make migration easier for now using ~/bin/kubectl [01:42:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [01:48:44] !log tools old version of kubectl removed. Anyone who needs it can download it with `curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.4.12/bin/linux/amd64/kubectl` [01:48:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:49:22] are there documents about how can i fix CrashLoopBackoff pods on toolforge? [03:52:08] it depends on why they're crash looping [03:53:25] have you looked at `kubectl get pods` `kubectl logs ` `kubectl describe pods ` [05:02:59] WhitePhosphorus: A pod in CrashLoopBackOff means that the docker container inside the pod has crashed repeatedly and Kubernetes has decided that it should not retry that process any more. As An.tiComposite said the cause is highly variable, but nearly always is either a logic bug in the application or the tool using more memory than it has been allocated. [05:03:46] WhitePhosphorus: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-pod-replication-controller/#debugging-pods -- there may be some tips there that can help you. Looking at the $HOME/error.log or $HOME/uwsgi.log files may be useful as well. [05:05:00] I opened T246559 a few hours ago to remind me to check on the Pods that are in the crash state [05:05:00] T246559: Investigate CrashLoopBackoff Pods on 2020 Kubernetes cluster - https://phabricator.wikimedia.org/T246559 [05:14:25] !log tools.blubber Hard restart of webservice to fix startup crash caused by a webservice v0.63 (T246559) [05:14:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.blubber/SAL [05:29:52] ty guys i've managed to fix my tools [05:34:38] WhitePhosphorus: excellent! Do you mind if I ask what the problem turned out to be? [05:35:12] bd808 ImportError: No module named 'encodings' [05:35:36] Corrupt venv or just missed the pip install for that? [05:35:36] i removed the original virtual env and created a new one [05:35:42] *nod* [05:37:30] !log tools.catscan2 Editied /data/project/catscan2/.lighttpd.conf to remove fast-cgi config section that was preventing lighttpd from starting. (T246559) [05:37:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.catscan2/SAL [05:43:07] !log tools.cobot Restarted as a python2 webservice to match the existing www/python/venv. (T246559) [05:43:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cobot/SAL [05:46:15] !log tools.corhist Hard restart of webservice to fix startup crash caused by a webservice v0.63 bug (T246559) [05:46:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.corhist/SAL [06:15:36] !log tools.dtz Hard restart of webservice to fix startup crash caused by a webservice v0.63 bug (T246559) [06:15:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dtz/SAL [06:18:27] !log tools.farhangestan Restarted webservice as 'python' type (py3.4) rather than 'python3.5' to match existing www/python/venv (T246559) [06:18:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.farhangestan/SAL [06:23:14] !log tools.hroest2 Tool stuck in CrashLoopBackOff because python 2.7 venv did not contain the flask library. (T246559) [06:23:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.hroest2/SAL [06:23:52] !log tools.hroest2 Stopped trivial 'hello world' tool after fixing it's venv. [06:23:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.hroest2/SAL [06:26:47] !log tools.machtsinn Restarted with python3.5 runtime to match existing www/python/venv version (T246559) [06:26:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.machtsinn/SAL [06:28:34] !log tools.matthobot Stopped python webservice because there are no python application files in www/python/src. (T246559) [06:28:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.matthobot/SAL [07:32:00] !log tools.mw2sparql Hard restart of webservice to fix startup crash caused by a webservice v0.63 bug (T246559) [07:32:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mw2sparql/SAL [07:35:59] !log tools.quick-intersection Edited /data/project/quick-intersection/.lighttpd.conf to remove fast-cgi config section that was preventing lighttpd from starting. (T246559) [07:36:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quick-intersection/SAL [07:39:35] !log tools.shextranslator Restarted with python3.7 runtime to match existing www/python/venv version (T246559) [07:39:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.shextranslator/SAL [07:51:04] !log tools.wdumps Stopped tool because it is missing the required www/python/src/app.py entry point. This is causing uwsgi to crash. (T246559, T246562) [07:51:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdumps/SAL [08:03:17] !log tools.webarchivebot Deleted webarchivebot-backend deployment because code is not running properly on the 2020 Kubernetes cluster. (T246559, T246563) [08:03:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.webarchivebot/SAL [08:10:14] !log tools.wikicite-dashboard Edited /data/project/wikicite-dashboard/.lighttpd.conf to remove fast-cgi config section that was preventing lighttpd from starting. (T246559) [08:10:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikicite-dashboard/SAL [15:55:07] !log tools.wikibugs Updated channels.yaml to: 7fc88d6228d00ad5e773c8c4b0065e2471240a03 Add User-revi to the channels list [15:55:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [16:02:51] Anything known that could cause https://phabricator.wikimedia.org/T246577#5930625 on the cloud services side? [16:11:38] ^ recovered [19:55:48] ][[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[ [20:45:52] !log tools.teg increased services quota to 2 for k8s T246553 [20:45:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.teg/SAL [20:45:55] T246553: "teg" tool needs a higher Services quota to migrate to 2020 Kubernetes cluster - https://phabricator.wikimedia.org/T246553 [20:48:38] !log tools.teg running kubectl apply -f backend.yml T246553 [20:48:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.teg/SAL [20:49:37] !log tools.teg starting php7.2 webservice T246553 [20:49:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.teg/SAL [20:58:35] !log tools.teg set namespace resourcequota for cpu to 2.5 T246553 [20:58:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.teg/SAL [20:58:38] T246553: "teg" tool needs a higher Services quota to migrate to 2020 Kubernetes cluster - https://phabricator.wikimedia.org/T246553 [23:19:46] hey all! quick question, if my toolforge k8s app isn't coming up, how do I see the output of the failed process start? [23:20:22] right now service.log is just "exited with status 1"