[00:00:09] <wm-bot>	 !log tools.digitaltmuseum <root> Migrated to 2020 Kubernetes cluster
[00:03:10] <wm-bot>	 !log tools.everythingisconnected <root> Deleted orphan webservice Pod running on legacy Kubernetes cluster
[00:04:38] <wm-bot>	 !log tools.externalitemsuggester <root> Migrated to 2020 Kubernetes cluster
[00:05:28] <wm-bot>	 !log tools.farhangestan <root> Migrated to 2020 Kubernetes cluster
[00:06:39] <wm-bot>	 !log tools.forrestbot <root> Migrated to 2020 Kubernetes cluster
[00:07:04] <wm-bot>	 !log tools.gerrit-newcomer-bot <root> Migrated to 2020 Kubernetes cluster
[00:10:06] <wm-bot>	 !log tools.gerrit-patch-uploader <root> Migrated to 2020 Kubernetes cluster
[00:21:46] <wm-bot>	 !log tools.gerrit-reviewer-bot <root> Deleted orphan ReplicaSet and Pod on legacy Kubernetes cluster
[00:26:31] <wm-bot>	 !log tools.giraffe <root> Shutdown orphan webservice running on legacy Kubernetes cluster
[00:29:41] <wm-bot>	 !log tools.globalcsd <root> Migrated to 2020 Kubernetes cluster
[00:31:54] <wm-bot>	 !log tools.gtirloni-sandbox <root> Deleted manually created ReplicaSet running on legacy Kubernetes cluster
[00:38:42] <wm-bot>	 !log tools.kmlexport <root> Deleted orphan ReplicaSet running on legacy Kubernetes cluster
[00:38:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.kmlexport/SAL
[00:41:11] <wm-bot>	 !log tools.lziad <root> Migrated to 2020 Kubernetes cluster
[00:41:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lziad/SAL
[01:42:41] <wm-bot>	 !log tools.lziad-cvnbot <root> Migrated to 2020 Kubernetes cluster
[01:42:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lziad-cvnbot/SAL
[01:44:44] <wm-bot>	 !log tools.most-wanted <root> Deleted orphan ReplicaSet running on the legacy Kubernetes cluster
[01:44:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.most-wanted/SAL
[01:46:22] <wm-bot>	 !log tools.mrmetadata <root> Deleted orphan ReplicaSet running on the legacy Kubernetes cluster
[01:46:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mrmetadata/SAL
[01:47:47] <wm-bot>	 !log tools.my-first-flask-oauth-tool <root> Deleted orphan ReplicaSet running on the legacy Kubernetes cluster
[01:47:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.my-first-flask-oauth-tool/SAL
[01:51:47] <wm-bot>	 !log tools.neechalbot <root> Deleted orphan webservice running on legacy Kubernetes cluster
[01:51:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.neechalbot/SAL
[01:54:28] <wm-bot>	 !log tools.npp-lv <root> Tool is missing .kube configuration directory needed for use of Kubernetes cluster
[01:54:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.npp-lv/SAL
[02:00:53] <wm-bot>	 !log tools.ores-demos <root> Migrated to 2020 Kubernetes cluster
[02:00:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ores-demos/SAL
[02:07:41] <wm-bot>	 !log tools.pathoschild-contrib <root> Migrated to 2020 Kubernetes cluster
[02:07:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pathoschild-contrib/SAL
[04:00:46] <AmandaNP>	 andrewbogott: re. T208414 - project files would be in /project right?
[04:00:47] <stashbot>	 T208414: Check whether utrs project requires NFS or not - https://phabricator.wikimedia.org/T208414
[04:04:56] <andrewbogott>	 AmandaNP: I think /data/project
[04:05:26] <andrewbogott>	 you can look in /etc/fstab to see where various nfs things are mounted
[04:10:02] <AmandaNP>	 andrewbogott: fstab doesn't work (unless I have to install it) but there is nothing in /data/project, so I think we are good to take everything off but home
[04:11:04] <andrewbogott>	 AmandaNP: /etc/fstab is a file, you'd have to open it in nano or something
[04:11:08] <andrewbogott>	 but, ok!
[04:11:15] <andrewbogott>	 Can you comment that on the task?  I'm on the go right now
[04:12:15] <AmandaNP>	 ahh. and ya sure
[04:12:58] <andrewbogott>	 thanks!
[12:11:31] <fnielsen>	 I had a problem Wednesday after upgrading my Python-based Toolforge webservice with Python3.7 and K8S. Thursday it seemed resolved (after the scheduled NFS maintenance and work of bd808 and Bstorm), but now it is gone again and I get very little information from the log file. I have put up a phabicket at https://phabricator.wikimedia.org/ a question is whether I should try to resolve it myself or there is any log that the WMF 
[12:11:31] <fnielsen>	 staff can see that can give a hint. I wonder if it is a memory problem. The interaction with bd808/bstorm Wednesday did not seem to suggest to me that it was a memory problem.
[12:12:10] <fnielsen>	 The ticket is here: https://phabricator.wikimedia.org/T246523
[12:39:39] <fnielsen>	 I am wondering if there is any Grafana knowledgable person a can say from https://grafana-labs.wikimedia.org/d/toolforge-k8s-namespace-resources/kubernetes-namespace-resources?orgId=1&var-namespace=tool-scholia&refresh=5m whether there is anything wrong?
[16:32:09] <andrewbogott>	 !log utrs removed scratch, dumps, project mounts from /etc/fstab on all instances
[16:32:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL
[16:32:12] <bstorm_>	 !log admin downtimed the smart alert on cloudvirt1009 until Monday since apparently predictive failures flap T244986
[16:32:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[16:32:14] <stashbot>	 T244986: cloudvirt1009: Device not healthy -SMART- - https://phabricator.wikimedia.org/T244986
[16:32:43] <bstorm_>	 fnielsen: I'm looking into it.  
[16:40:56] <bstorm_>	 !log tools.scholia cleaned up orphaned replicaset on the old cluster.  
[16:40:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.scholia/SAL
[16:41:24] <bstorm_>	 !log tools.scholia stopping and starting the app on the new cluster 
[16:41:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.scholia/SAL
[20:47:59] <wm-bot>	 !log tools.phetools <root> Deleted orphan ReplicaSet running on the legacy Kubernetes cluster
[20:48:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phetools/SAL
[20:54:41] <wm-bot>	 !log tools.piagetbot <root> Migrated sopelbot.yaml to 2020 Kubernetes cluster
[20:54:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.piagetbot/SAL
[21:40:36] <wm-bot>	 !log tools.recoin <root> Migrated to 2020 Kubernetes cluster
[21:40:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.recoin/SAL
[21:46:31] <wm-bot>	 !log tools.schubota <root> Stopped orphan Kubernetes webservice on legacy cluster
[21:46:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.schubota/SAL
[21:47:03] <wm-bot>	 !log tools.schubota <root> Stopped grid engine webservice which was serving an empty /data/project/schubota/public_html
[21:47:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.schubota/SAL
[21:55:42] <wm-bot>	 !log tools.tabletop <root> Deleted orphan ReplicSet running on the legacy Kubernetes cluster
[21:55:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tabletop/SAL
[22:32:23] <wm-bot>	 !log tools.teg <root> Stopped php7.2 webservice stuck in CrashLoopBackOff due to a syntaxically invalid /data/project/teg/.lighttpd.conf file (T246553)
[22:32:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.teg/SAL
[22:34:01] <wm-bot>	 !log tools.teg <root> Deleted partially applied /data/project/teg/backend.yml Kubernetes deployment on 2020 Kubernetes cluster (T246553)
[22:34:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.teg/SAL
[22:35:26] <wm-bot>	 !log tools.teg <root> Deleted /data/project/teg/backend.yml Kubernetes deployment on legacy cluster (T246553)
[22:35:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.teg/SAL
[22:36:47] <wm-bot>	 !log tools.topviews <root> Migrated to 2020 Kubernetes cluster
[22:36:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.topviews/SAL
[22:39:18] <wm-bot>	 !log tools.urbanecm-test-1 <root> Deleted orphan webservice running on legacy Kubernetes cluster
[22:39:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.urbanecm-test-1/SAL
[22:43:08] <wm-bot>	 !log tools.vltools <root> Deleted broken custom deployment from /data/project/vltools/etc/test.yaml running on the legacy Kubernetes cluster
[22:43:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.vltools/SAL
[22:47:21] <wm-bot>	 !log tools.w-slackbot <root> Migrated to 2020 Kubernetes cluster
[22:47:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.w-slackbot/SAL
[22:49:37] <wm-bot>	 !log tools.wdumps <root> Migrated to 2020 Kubernetes cluster
[22:49:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdumps/SAL
[22:50:52] <wm-bot>	 !log tools.webarchivebot <root> Migrated to 2020 Kubernetes cluster
[22:51:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.webarchivebot/SAL
[22:53:37] <wm-bot>	 !log tools.webarchivebot <root> Migrated /data/project/webarchivebot/bin/deployment.yaml to 2020 Kubernetes cluster
[22:53:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.webarchivebot/SAL
[22:56:32] <wm-bot>	 !log tools.whichsub <root> Migrated to 2020 Kubernetes cluster
[22:56:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.whichsub/SAL
[23:03:03] <wm-bot>	 !log tools.whois <root> Deleted orphan webservice running on legacy Kuberntes cluster
[23:03:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.whois/SAL
[23:04:02] <wm-bot>	 !log tools.wiki-topic <root> Migrated to 2020 Kubernetes cluster
[23:04:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wiki-topic/SAL
[23:04:45] <wm-bot>	 !log tools.wikibugs <root> Migrated to 2020 Kubernetes cluster
[23:04:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[23:35:26] <wm-bot>	 !log tools.wikidata-exports Migrated to 2020 Kubernetes cluster
[23:35:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikidata-exports/SAL
[23:54:28] <wm-bot>	 !log tools.wikidata-janitor <root> Migrated to 2020 Kubernetes cluster
[23:54:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikidata-janitor/SAL
[23:55:28] <wm-bot>	 !log tools.wikidata-reconcile <root> Migrated to 2020 Kubernetes cluster
[23:55:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikidata-reconcile/SAL
[23:57:35] <wm-bot>	 !log tools.wikidata-reconcile <root> Deleted 2 orphan ReplicaSets found running on the legacy Kubernetes cluster
[23:57:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikidata-reconcile/SAL
[23:58:38] <wm-bot>	 !log tools.wikidata-redirects-conflicts-reports <root> Migrated to 2020 Kubernetes cluster
[23:58:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikidata-redirects-conflicts-reports/SAL
[23:59:32] <wm-bot>	 !log tools.wikidipendenza <root> Migrated to 2020 Kubernetes cluster
[23:59:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikidipendenza/SAL