[06:49:58] !log tools remove scfc kubernetes admin access after bd808 removed tools.admin membership to avoid maintain-kubeusers crashes when it expires [06:50:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:21:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lingua-libre-bot/SAL [07:42:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lingua-libre-bot/SAL [08:56:45] Is there a new ram limit on labs? [08:56:56] my convert jobs are getting killed since one month [08:57:17] i am running the bot for approx. 5 year and never had that kind of problems :/ [09:06:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lingua-libre-bot/SAL [10:25:42] Steinsplitter: check this out: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Allocating_additional_memory [10:36:09] !log tools rebased labs/private.git after merge conflict [10:36:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:37:40] hi all. I'm trying to resize an instance but I get this error: "'nova.exception.FlavorDiskSmallerThanMinDisk". The problem is that I need more ram in an old instance but new flavors have smaller HD. The old flavor says 40G, (although that checking df -h says 20G) , and I don't need the 40G, 20 is ok, but the resize functions fails. [10:37:51] This is the instance: covid-data.wmf-research-tools.eqiad1.wikimedia.cloud [10:38:07] mmmm [10:38:27] hey arturo [10:38:42] (ups, this Libera keeps changing my nick) [10:38:51] o/ dsaez [10:39:26] dsaez: does this instance have valuable data inside? [10:39:32] yes! [10:39:43] ok! [10:39:47] so here is my proposal: [10:40:31] 1) detach data from VM, by using a cinder volume [10:40:31] 2) once 1) is completed, delete/create instances as desired (data will stay at the cinder volume) [10:40:59] I can guide you in the process, should be really simple [10:41:29] let's go (this sound easier that reconfig all the services and packages I have configured there) [10:42:40] ok, are you in horizon? [10:43:27] for the record, we will follow the docs here: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_instances [10:44:47] yes I'm there [10:44:55] let me check the doc [10:45:48] basically, we will click a few buttons on horizon to create the volume [10:45:53] then, mount it on the VM [10:46:12] then, mv the data from the VM filesystem to the mounted volume [10:46:40] if you have enough storage quota, you can even snapshot the volume at that point [10:47:08] I've tried to create the snapshot but got an error [10:47:21] ? [10:47:27] let me check [10:47:58] Policy doesn't allow os_compute_api:servers:create_image to be performed. [10:49:42] did you try snapshoting the virtual machine instance or the cinder volume? [10:50:11] this is just what I get if click on "Create Snapshot" on the horizon interface [10:50:30] you should have a 'Volumes' menu entry on the left [10:50:50] like, part of the main menu of things: Compute, Volumes, Network... [10:51:13] yes [10:51:47] did you create the cinder volume in that menu first? [10:52:46] !log wmf-research-tools briefly add myself as projectadmin to help with cinder data migration [10:52:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmf-research-tools/SAL [10:53:13] not I haven't [10:53:25] dsaez: ok, let's do it then! [10:53:55] ok, should we create an empty volumen? [10:54:14] yes, you can think of a cinder volume like a pendrive [10:54:37] is just storage that you can plug to virtual machines and store things on it [10:56:24] ooook. [10:56:37] how much data do you have? in GB? [10:56:42] can that also be used as swap-memory (or wont be efficent?) [10:57:06] thet instance I want to copy, is using now 16G [10:57:33] I've never tried the swap thing, but I bet it would be very very slow [10:58:00] https://0bin.net/paste/zIlmdRmc#2dpEjXnFgJL6gKdP7m9SiHHx-wl8wdk+YargW/t9vw8 [10:58:30] ok, try creating a 20GB cinder volume [10:59:45] done. [11:00:26] ok, now attach it to your VM [11:02:50] I do ~$ sudo wmcs-prepare-cinder-volume [11:02:51] and I get: [11:03:04] The only block device device available to mount is vdb. Selecting. [11:03:26] right [11:03:42] did it mount on /srv [11:03:49] the cinder volume you created is attached as /dev/vdb on covid-data [11:04:06] got it... [11:04:08] now, cp (or mv) the data you are interested in to the /srv [11:05:33] done. [11:08:13] question. If I want to create a snapshot for the full system (packages, web services configuration, etc ... ) can I do that? [11:08:28] I don't think that's something we support today [11:08:47] got it. [11:08:54] because we usually configure packages etc using puppet (so is reproducible anyway) [11:09:31] interesting. I need to learn about puppets. [11:09:59] Ok, now, I should create the new instance ... and reconfigure everything there ? [11:10:07] now your data is safely in the cinder volume you can do whaetever experiment with the VM without fear of losing the data [11:10:14] got it [11:11:10] dsaez: better, if you no longer need any particular storage in the VM (because the data is in the external volume) perhaps you can try the resize now to whatever new flavor [11:12:52] the problem with the resize, is that apparently it doesn't work to reduce HD size. [11:13:13] I see [11:13:18] Th´at least that is my understanding of the error [11:13:30] and makes sense if you think about it: it could mean data loss [11:13:37] true [11:14:06] let me see if I can enable an additional flavor [11:15:07] ooh, that would be amazing. [11:17:58] dsaez: how much ram do you need? [11:18:07] and CPU? [11:18:11] 8G [11:18:20] 2CPUS and 8G would be the ideal [11:19:42] dsaez: see if you can find `g3.cores2.ram80.disk40.private` in the resize instance menu [11:20:05] yeah! [11:20:32] !log admin created `g3.cores2.ram80.disk40.private` for the wmf-research-tools project, to allow resizing a 40G disk instance [11:20:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:21:20] dsaez: cool, try the resize! [11:21:40] funcionó! great! thanks I owe a beer [11:22:01] you saved me one day of work. [11:23:09] great! [11:24:04] dsaez: the thing is: given you are storing the data in the cinder volume, basically all storage in the VM is 'leaking' in the sense we are allocating 40G for that VM that is no use. I encourage you to let us know when you no longer need the VM, so we can drop the flavor I just created [11:26:42] got it. I'll work on refactoring my code. When I started this project, collecting data for COVID related pages, I never thought that we have ended having more than 10K related articles, and with active edting for so long [11:27:32] So, the code is not efficient, and that is why I need so much RAM ...but I need to rewrite all this. And then I'll start a new instance from 0 [11:28:52] arturo: do you saw my question above? should i open a ticket? i think i have a similar problem ... as dsaez [11:29:06] i had no RAM problems for almost a decade here on wikimedia.... [11:29:27] but sinc approx a mont (or moor) hm. strange. [11:29:34] convert... Killed. [11:38:59] Steinsplitter: check out this https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Allocating_additional_memory [11:39:41] thanks [11:48:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [14:08:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [14:12:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [15:22:38] is there a telegram channel bridge to this channel? [15:22:54] yes [15:22:57] * wm-bb looks it up [15:23:42] yuvipanda: https://t.me/wmcloudirc [15:24:07] yay! [15:24:18] ttyl, LimeChat [15:34:40] bd808: the description of that channel still has freenode in it, could you update it please? [15:37:12] majavah: nice spotting. It now says "Discussion of Wikimedia's Cloud VPS, Toolforge, and related services. | Telegram bridge for the #wikimedia-cloud Libera.Chat IRC channel. | Code of Conduct applies: https://www.mediawiki.org/wiki/CoC" [16:01:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [17:06:58] I'm not sure what changed, but since I last did it a few months ago, Rust compile times via the grid are much faster, so thanks :D [17:07:09] yw, legoktm [17:08:04] (I actually didn't do anything, i'll pass it on to whoever actually did work) [18:03:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [18:22:14] hmm hello! trying to do standalone puppet master and encountering an issue with apache2 [18:22:22] doing https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Step_1:_Setup_a_standalone_puppetmaster [18:22:34] and apache2 failing with [18:22:34] AH00534: apache2: Configuration error: No MPM loaded. [18:22:45] mpm confs look about right to me [18:25:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [18:54:52] bd808: sorry to bother... not having a lot of luck myself. any ideas about role::puppetmaster::standalone not able to start apache2? i think maybe i got past the 'No MPM loaded' error by hackily adding a LoadModule statement into apache2.conf...but now am having other issues 'no listening sockets available, shutting down' [18:55:17] please say "i don't know" if you don't have any ideas (and/or point me to the right person to ask) [18:55:20] :) [18:55:43] ottomata: are any other software listening on the port puppet wants [18:55:45] ottomata: is apache2.conf by any chance completely empty? [18:55:52] Zppix: no [18:55:54] checked that [18:56:12] nothin on 80, 443 or 8140 [18:56:16] That was one of the issues i had so just wanted to offer that [18:56:26] majavah: no it has [18:56:27] Include sites-local/* [18:56:34] oh [18:56:41] ottomata asking about role::puppetmaster::standalone feels like dejavu from many years ago 🙂 [18:56:42] which only has a blank dummy.conf in it!~ [18:57:00] haha .... [18:57:02] ah, this is some cloudvps puppet weirdness that I have seen a couple of times but never bothered to file a task or investigate properly [18:57:34] a hacky fix is to apt-get purge apache2, remove /etc/apache2 and apt-get install apache2 manually, and then run puppet again [18:57:53] ahh crazy. [18:57:55] ok... [18:58:11] there's some weirdness that tries to provide an apache2 config directory not managed by puppet for local apps but it sometimes fails like this [18:58:23] I wonder if the inital install of apache maybe gets borked and causes the blank conf [18:58:55] and file a task about it being empty please [18:59:15] ok [18:59:25] that seems to have worekd majavah thank you! [19:07:31] majavah: https://phabricator.wikimedia.org/T283531 [19:15:10] huh majavah yeah passenger problems, now [19:15:12] sudo a2enmod passenger [19:15:12] ERROR: Module passenger does not exist! [19:15:28] but the passenger .deb package seems installed infe [19:15:38] libapache2-mod-passenger is already the newest version (5.0.30-1.1). [19:17:18] did you run puppet after the apache2 fix? [19:17:52] yes...i think [19:17:55] also had to purge and install libapache2-mod-passenger [19:18:31] hmmm, in my experience puppet deals with everything else once you remove apace2 [19:18:47] yeah [19:18:49] Notice: /Stage[main]/Httpd/Httpd::Mod_conf[passenger]/Exec[ensure_present_mod_passenger]/returns: ERROR: Module passenger does not exist! [19:18:49] Error: '/usr/sbin/a2enmod passenger' returned 1 instead of one of [0] [19:18:51] Error: /Stage[main]/Httpd/Httpd::Mod_conf[passenger]/Exec[ensure_present_mod_passenger]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/a2enmod passenger' returned 1 instead of one of [0] (corrective) [19:18:56] you can try manually enabling it using a2enmod passenger [19:19:01] yeah tried that [19:19:01] except [19:19:06] it wasn't in mods-avilable [19:19:12] which was why I purged & reionstalled... [19:19:15] but now i'm having other issues! [19:19:15] :) [19:20:02] oh, maybe als need puppet-master-passenger resinstalled [19:20:51] ok! [19:20:52] yes [19:21:56] majavah: ok i tthink its all working now [19:22:01] updated ticket with full workarorund [21:17:53] bd808: I think so, it'll keep trying to rejoin but I don't think it'll actually cause operational issues. [21:18:00] if it does I'll just turn it off and move on [21:19:51] cool cool. I may try in a bit and see what shenanigans follow :) [22:01:56] bd808: wikibugs will infact cope [22:27:30] It's not crying at the moment [22:27:39] So it must be coping with me killing ##RhinosF1 [22:32:46] !log admin changing the default ttl for eqiad1.wikimedia.cloud. from 3600 to 60; this should help us avoid madness when re-using hostnames. [22:32:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:32:56] We banned it from #miraheze-libera on freenode [23:00:33] poor wikibugs