[02:16:03] so is everything up and running properly on the new k cluster now? [02:17:24] yep [02:17:46] well [02:17:54] there's like 2 paws-public pods on the old one still [02:18:27] but I think that's the only straggler [02:19:05] meanwhile on the new cluster: [02:19:05] krenair@tools-k8s-control-1:~$ sudo -i kubectl get -A pod -o json | jq '.items | length' [02:19:06] 940 [02:47:13] hey again all! quick question. has wikimedia considered hosting a sentry for wmflabs and others? [02:47:29] tgr you seem like the type to know :) ^ [02:48:26] what exactly do you mean by that [02:49:23] well, sentry.io has a centrally hosted one, but sentry itself is an open-source application, so maybe WMF could host one for tool developer use? [02:51:09] again, that doesn't really answer the question - what is it [02:51:36] https://github.com/getsentry/sentry [02:51:52] visit sentry.io to learn more [02:53:55] i guess there's this https://phabricator.wikimedia.org/tag/sentry/ [02:56:32] there's a task, somewhere, about doing better monitoring for toolforge tools [02:56:34] unless a tool is severely impacting the service for others, there's really not much point to have a central error reporting service [02:57:19] well, given that labs is somewhat limited-access for us humble tool developers, it can be very useful for debugging user-impacting issues [02:57:24] it's one of the few things the grid does better than k8s at the moment: it's really easy to have the grid send me an email when something breaks [02:57:38] (in a bot, webservices are different) [03:24:43] Framawiki tried to set up Sentry for labs tools, not sure where that's at [03:27:03] https://wikitech.wikimedia.org/wiki/Nova_Resource:Sentry says it got purged end of last year [03:27:44] unless that's something else [03:29:56] no, that was intended for use for both that and as a testbed for containerizing Sentry (which didn't work out) [03:31:14] https://phabricator.wikimedia.org/project/profile/2348/ was the task/project I was thinking about earlier [08:01:43] [Global Notice] Hello, freenode will be undergoing maintenance 3 hours from now. Please disconnect from this network, connect to irc.quakenet.org, and join #Quakenet for updates, otherwise, you may be k-lined permanently from freenode. Thank you. -freenode staff [08:01:44] [Global Notice] Hello, freenode will be undergoing maintenance 3 hours from now. Please disconnect from this network, connect to irc.quakenet.org, and join #Quakenet for updates, otherwise, you may be k-lined permanently from freenode. Thank you. -freenode staff [08:01:45] [Global Notice] Hello, freenode will be undergoing maintenance 3 hours from now. Please disconnect from this network, connect to irc.quakenet.org, and join #Quakenet for updates, otherwise, you may be k-lined permanently from freenode. Thank you. -freenode staff [08:04:14] [Global Notice] Hello, freenode will be undergoing maintenance 3 hours from now. Please disconnect from this network, connect to irc.quakenet.org, and join #Quakenet for updates, otherwise, you may be k-lined permanently from freenode. Thank you. -freenode staff [08:04:16] [Global Notice] Hello, freenode will be undergoing maintenance 3 hours from now. Please disconnect from this network, connect to irc.quakenet.org, and join #Quakenet for updates, otherwise, you may be k-lined permanently from freenode. Thank you. -freenode staff [08:04:16] [Global Notice] Hello, freenode will be undergoing maintenance 3 hours from now. Please disconnect from this network, connect to irc.quakenet.org, and join #Quakenet for updates, otherwise, you may be k-lined permanently from freenode. Thank you. -freenode staff [09:44:37] !log tools.wikiloves Deploy latest from Git master: f02938a (T246696) [09:44:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL [09:49:42] !log tools.wikiloves webservice restart for new edition (africa2020) to be displayed (T246696) [09:49:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL [10:51:04] !log tools cordoned/drained all legacy k8s worker nodes except 1001/1002 (T246689) [10:51:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:51:07] T246689: Toolforge: cleanup legacy kubernetes cluster - https://phabricator.wikimedia.org/T246689 [10:54:53] !log tools deleted VMs `tools-worker-[1003-1020]` (legacy k8s cluster) (T246689) [10:54:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:28:06] arturo: thanks for working on openstack packaging! I'm going to do some more testing but things look good so far [14:39:17] andrewbogott: great! [15:12:16] hm [15:12:27] arturo, on my test VM I don't get this warning but I do on cloudservices2002-dev [15:12:36] GPG error: http://mirrors.wikimedia.org/osbpo stretch-queens-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 56056AB2FEE4EECB [15:12:43] (for the -nochanges repo too) [15:37:39] ok will take a look [16:38:14] !log tools.paws-public increased the services quota to 2 in the 2020 k8s cluster T246519 [16:38:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [16:38:17] T246519: "paws-public" tool running 2 custom pods on legacy Kubernetes cluster - https://phabricator.wikimedia.org/T246519 [16:47:55] !log tools.paws-public deploy services and deployments on the 2020 k8s cluster via kubectl apply -f deploy.yaml T246519 [16:47:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [16:47:59] T246519: "paws-public" tool running 2 custom pods on legacy Kubernetes cluster - https://phabricator.wikimedia.org/T246519 [17:01:22] everything about paws has to be unique :/ [17:11:45] It's working in some ways. I think the old one used a replica controller for an unfortunate reason [17:12:27] So far, this isn't THAT unique, but if there's a perms issue, this might do it. [17:13:08] It might just be 404 because of domain name and rewriting...since I'm using the service address [17:18:38] It's not perms or the renderer couldn't do it's thing, I think... [17:31:14] !log tools create a OpenStack virtual ip address for the new elasticsearch cluster T236606 [17:31:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:31:18] T236606: Rebuild Toolforge elasticsearch cluster with Stretch or Buster - https://phabricator.wikimedia.org/T236606 [18:02:49] !log tools create OpenStack DNS record for elasticsearch.svc.tools.eqiad.wikimedia.cloud T236606 [18:02:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:02:53] T236606: Rebuild Toolforge elasticsearch cluster with Stretch or Buster - https://phabricator.wikimedia.org/T236606 [18:16:30] !log tools create OpenStack DNS record for elasticsearch.svc.tools.eqiad1.wikimedia.cloud (eqiad1 subdomain change) T236606 [18:16:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:16:34] T236606: Rebuild Toolforge elasticsearch cluster with Stretch or Buster - https://phabricator.wikimedia.org/T236606 [21:55:49] !log tools.paws-public launched ingress on the new cluster, removing the service object on the old cluster T246519 [21:55:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [21:55:52] T246519: "paws-public" tool running 2 custom pods on legacy Kubernetes cluster - https://phabricator.wikimedia.org/T246519 [21:58:07] !log tools.paws-public recreated the service on the old cluster because it didn't work right away? T246519 [21:58:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [22:07:34] !log tools.paws-public removing the service object on the old cluster again T246519 [22:07:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [22:07:37] T246519: "paws-public" tool running 2 custom pods on legacy Kubernetes cluster - https://phabricator.wikimedia.org/T246519 [22:20:13] !log tools.paws-public created ingress for paws-public.wmflabs.org and deleted old service object T246519 [22:20:14] !log tools.paws-public Added bstorm_, chicocvenancio, and bd808 as co-maintainers [22:20:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [22:20:16] T246519: "paws-public" tool running 2 custom pods on legacy Kubernetes cluster - https://phabricator.wikimedia.org/T246519 [22:20:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [22:23:08] !log tools.paws-public deleted all resources on the old cluster T246519 [22:23:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws-public/SAL [22:24:29] bstorm_: OMG!!1! The legacy Kubernetes cluster is empty now! [22:24:32] <3 [22:27:41] 🎉 [22:28:55] !log tools.zppixbot add wikibugs to ignore list and restart bot to deploy [22:28:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [22:30:09] 🎉 [22:49:16] bd808, bstorm_: Congratulations. [22:51:06] I love it when a plan comes together. -- bd808 & Colonel John “Hannibal” Smith [22:51:48] super cut style -- https://www.youtube.com/watch?v=NsUFBm1uENs [23:13:25] !log tools.bash Updated to f249e23c (Update for ElasticSearch 7 backend) and switched to  http://elasticsearch.svc.tools.eqiad1.wikimedia.cloud backend [23:13:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bash/SAL