[07:46:05] !log deployment-prep enable syntax highlighting line numbering on all namespaces (T267911) [07:46:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [07:46:13] T267911: Add line numbering to syntax highlighting (CodeMirror extension) - https://phabricator.wikimedia.org/T267911 [10:25:22] !log dplbot force-cleanup job 9999850 (T279914) [10:25:24] arturo: Unknown project "dplbot" [10:25:24] arturo: Did you mean to say "tools.dplbot" instead? [10:25:24] T279914: Job won't die - https://phabricator.wikimedia.org/T279914 [10:25:34] !log tools.dplbot force-cleanup job 9999850 (T279914) [10:25:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dplbot/SAL [10:43:59] !log admin enabled ceph upmap balancer on codfw (T274573,T274573) [10:44:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:44:03] T274573: [ceph] Use upmap PG balancer - https://phabricator.wikimedia.org/T274573 [11:17:19] !log tools deleted shutoff VMs tools-static-12,13 (T278539) [11:17:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:17:24] T278539: Toolforge: migrate static servers to Debian Buster - https://phabricator.wikimedia.org/T278539 [11:18:48] !log tools deleted shutoff VM tools-mail-02 (T278538) [11:19:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:19:01] T278538: Toolforge: migrate mail server to Debian Buster - https://phabricator.wikimedia.org/T278538 [11:20:18] !log tools deleted shutoff VM tools-docker-registry-03,04 (T278303) [11:20:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:20:23] T278303: Toolforge: migrate docker registry to Debian Buster - https://phabricator.wikimedia.org/T278303 [11:21:35] !log tools deleted shutoff VM tools-sge-services-03,04 (T278354) [11:21:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:21:40] T278354: Toolforge: migrate services node to Debian Buster - https://phabricator.wikimedia.org/T278354 [11:23:01] !log tools deleted shutoff VM tools-package-builder-02 (T275864) [11:23:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:23:07] T275864: Toolforge: migrate to Debian Buster - https://phabricator.wikimedia.org/T275864 [13:11:06] !log admin upgrading eqiad1 designate to version Victoria, T261137 [13:11:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:11:11] T261137: upgrade cloud-vps openstack to Openstack version 'Victoria' - https://phabricator.wikimedia.org/T261137 [13:26:01] !log tools upgrade puppet and python-wmflib on tools-prometheus-03 [13:26:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:36:46] !log admin upgrading codfw1dev to version Victoria, T261137 [14:36:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:36:51] T261137: upgrade cloud-vps openstack to Openstack version 'Victoria' - https://phabricator.wikimedia.org/T261137 [14:43:07] !log admin enabling ceph upmap pg balancer on equiad (T274573) [14:43:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:43:11] T274573: [ceph] Use upmap PG balancer - https://phabricator.wikimedia.org/T274573 [14:49:37] !log admin Running the first_pass balancing plan on ceph eqiad, current eval 0.030622 (T274573) [14:49:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:49:41] T274573: [ceph] Use upmap PG balancer - https://phabricator.wikimedia.org/T274573 [14:57:25] hi all, I am mentoring a student on a research project. I would like to give him access to a VPS instance on Tool Labs [14:57:46] i have just sent him the instructions to create a Wikitech developer account and get shell access [14:58:08] how can I give him access to the VPS and him to the project (horizon) [14:58:35] should I open a new issue on Phabricator, re-use the one where we asked for the creation of the instance or none of the above? [14:59:02] CristianCantoro: are you an administrator on that project? [14:59:44] you can check on open stack, for example https://openstack-browser.toolforge.org/project/video [15:02:05] joakino: I am [15:02:05] https://openstack-browser.toolforge.org/project/wikicommunityhealth [15:02:16] !log admin First pass finished, improved eval to 0.030075 (T274573) [15:02:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:02:20] T274573: [ceph] Use upmap PG balancer - https://phabricator.wikimedia.org/T274573 [15:02:32] "I would like to give him access to a VPS instance on Tool Labs" -- "Tool Labs" was renamed to Toolforge 4 years ago. "Cloud VPS" is what you meant. [15:03:48] !log admin Executing a second pass, there's still movements to improve the eval of 0.030075 (T274573) [15:03:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:04:16] CristianCantoro: I believe you can give them access from horizon.wikimedia.org https://wikitech.wikimedia.org/wiki/Help:Horizon_FAQ [15:04:22] bd808: I always get confused with those names ;-), I hope the next will be "Tool Cloud" so we will reach maximum entropy [15:04:44] switch to your project, and look under Access -> Project members [15:04:49] or maybe "Cloudforge" [15:05:20] cloudforge sounds cool 🙂 [15:05:29] renames are very hard tho [15:05:39] https://en.wikipedia.org/wiki/CloudForge [15:06:47] I'm surprised that's not an AWS service [15:06:54] as it sounds like one [15:07:26] Cool Barge [15:07:42] I agree "CloudForge" sounds really cool [15:08:32] yuvipanda! [15:08:42] * bd808 does the happy dance [15:08:43] !log admin Activating continuous upmap balancer, keeping a close eye (T274573) [15:08:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:08:49] T274573: [ceph] Use upmap PG balancer - https://phabricator.wikimedia.org/T274573 [15:08:52] hi bd808 :) [15:31:34] !log toolsbeta live-hacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/678043/ (T277653) [15:31:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:31:38] T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653 [15:36:47] !log toolsbeta created VM toolsbeta-sgeexec-0903 (buster) (T277653) [15:36:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:36:51] T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653 [15:38:57] i thought the 09 prefix was the debian version and buster ones should be 10xx? [15:39:40] Majavah: in theory that would be correct, yes. [15:40:14] arturo: ^ losing the naming connection to the distro will probably make it harder to track what is what [15:40:26] ack [15:44:40] !log toolsbeta delete VMs toolsbeta-sgeexec-0903 and toolsbeta-buster-sgeexec-01 (no longer useful) [15:44:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:50:34] hi! the ~/www/ directory in my toolforge tool account is owned by root - is there a way for me to undo that? [15:53:42] inductiveload: you can try `take ~/www/` [15:54:13] the `take` command is a local invention that wraps `sudo chown` with some additional checks [16:12:41] joakino: bd808, thank you, I was able to add him to the project and he could access the VPS [16:41:51] !log toolsbeta create VM toolsbeta-sgeexec-1002 (T277653) [16:41:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:41:56] T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653 [16:42:54] !log admin Ceph balancer got the cluster to eval 0.014916, that is 88-77% usage for compute pool, and 28-19% usage for the cinder one \o/ (T274573) [16:42:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:42:57] T274573: [ceph] Use upmap PG balancer - https://phabricator.wikimedia.org/T274573 [16:50:54] bd808, woop that works, thank you very much [16:57:30] !log deployment-prep deploying ores f08a3cb [16:57:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [16:59:58] Well. That didn't work. [17:00:30] !log deployment-prep failed deploy to ORES (connection to host failed) [17:00:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [17:01:00] * Majavah worries that they have broken something [17:01:10] halfak: what exactly is failing for you? [17:01:20] https://phabricator.wikimedia.org/T278723#6995430 [17:01:31] Probably not you. We haven't done an ORES deployment in a while. [17:01:53] But could be. Looks like the ssh handshake is failing with our deployment host. [17:02:11] I was just about to post about it in -releng [17:04:31] Posted in #wikimedia-releng. :) [17:12:57] halfak: yeah I don't think this was caused by my work on beta and I don't have time to look further today [20:07:10] !log toolhub Updated deployed demo to container built from bec6c35 [20:07:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolhub/SAL [20:36:39] the performance of cinder filesystems and role::labs::lvm is going to be the same right? [20:37:10] legoktm: mostly, yes. all storage is backed by ceph now [20:37:55] gotcha, that's what I thought [20:38:05] the pools may be tuned slightly differently. Like there are some custom flavors that give higher IOPS limits and I don't think we have a concept for that in cinder volumes yet [20:38:24] I should be switching to a new codesearch vm backed by cinder later today [20:38:34] hmm, pretty sure we were just using the defaults [20:39:24] I don't think many projects have the iops tweak available. maybe toolforge and integration? [20:39:49] the toolforge etcd nodes needed it if I'm remembering correctly [20:53:38] !log codesearch switched codesearch.wmcloud.org and codesearch-beta.wmcloud.org to point to codesearch8 (T279763) [20:53:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [20:53:44] T279763: Move codesearch to cinder - https://phabricator.wikimedia.org/T279763 [20:55:06] one thing I do see is that we're using way less inodes on cinder [20:55:37] er, no wait, cinder has way more inodes than our previous instance [20:56:20] 15M inodes (I allocated a 60GB volume) compared to our old /srv of 3.9M inodes / 60GB [20:57:57] I wonder if we forgot the `sudo mkfs.ext4 -T news` step last time