[07:26:05] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/424218 (owner: 10L10n-bot)
[12:53:09] <wikibugs>	 (03PS2) 10Rxy: SULWatcher: Target to only loginwiki [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/423713 (https://phabricator.wikimedia.org/T174287)
[12:54:20] <wikibugs>	 (03CR) 10Rxy: "Zhuyifei1999: thanks for your review." [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/423713 (https://phabricator.wikimedia.org/T174287) (owner: 10Rxy)
[13:23:41] <Lokal_Profil>	 !log wikispeech Deploy latest from Git master: f951d84, 98103e1 (T182861), 70cf552, c3bc53f (T177809), 1101d4b, b122d77, 9ae6b57, f961d2b, cf08625 (T183858), 958b2d1, c54aed9, 92e85fe, b4e54b7 (T133680)
[13:23:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL
[13:23:51] <stashbot>	 T177809: Fix multiple PHP class declarations in one file - https://phabricator.wikimedia.org/T177809
[13:23:51] <stashbot>	 T133680: [Task] Functionality: Play selected text (Wikispeech) - https://phabricator.wikimedia.org/T133680
[13:23:51] <stashbot>	 T182861: Break out lexicon tool code to a separate branch while under development - https://phabricator.wikimedia.org/T182861
[13:23:52] <stashbot>	 T183858: Update for SPDX License List 3.0 - https://phabricator.wikimedia.org/T183858
[16:02:19] <gehel>	 andrewbogott: Hey! I'm having trouble adding a hiera config to the deployment-maps puppet prefix on horizon. Half the config is saved, and the other half disappear...
[16:02:43] <gehel>	 running a linter on the yaml does not show any error, I'm out of idea...
[16:02:51] <gehel>	 does this sound like a known issue?
[16:02:55] <andrewbogott>	 gehel: if you're adding hiera settings for classes that are already included, maybe the settings are appearing up amongst the class definitions?
[16:03:28] * gehel is checking
[16:04:19] <gehel>	  /facepalm...
[16:04:40] <andrewbogott>	 It's not very obvious behavior but unfortunately the backend only allows us to store each key once
[16:04:44] <gehel>	 andrewbogott: thanks! I've spent 1h not understanding this!
[16:04:46] <andrewbogott>	 so, it's presumed to be part of the class def
[16:04:51] <andrewbogott>	 Hope it makes sense now!
[16:05:22] <gehel>	 Ok, so next question... how do I change them in the class definition? Do I need to remove the class and re-add it?
[16:26:40] * chicocvenancio is unsure, can check in 1 hour
[17:03:50] <chicocvenancio>	 gehel: yeah, that seems to be the way to change parameters
[17:05:46] <gehel>	 chicocvenancio: thanks! that's what I did and it worked...
[18:26:28] <jdlrobson>	 hey all i'm having disk space problems on reading-web-staging-3 instance
[18:27:52] <chicocvenancio>	 jdlrobson: what have you tried?
[18:28:03] <jdlrobson>	 i've cleaned tmp
[18:28:06] <jdlrobson>	 but still having  issues
[18:28:11] <jdlrobson>	 i don't know what's taking up the space
[18:29:25] <jdlrobson>	 The folders taking up the most space according to du are /srv/lxc and  /srv/mediawiki-vagrant
[18:29:34] <jdlrobson>	 and a lot of vagrant stuff
[18:29:40] <jdlrobson>	 im not sure where i can recover space
[18:30:02] <jdlrobson>	 any ideas?
[18:31:10] <jdlrobson>	 and http://reading-web-staging.wmflabs.org/ is now not working :/
[18:35:17] <chicocvenancio>	 jdlrobson: is this an instance with only 20gb of quota?
[18:36:05] <jdlrobson>	 lemme check chicocvenancio 
[18:36:05] <bd808>	 jdlrobson: It looks like you did not mount the additional quota for your instance. 
[18:37:22] <bd808>	 the instance is a m1.large which means that there is about 60G more disk quota that you are not actually using
[18:37:26] <jdlrobson>	 it's m1.large
[18:37:31] <jdlrobson>	 yeh.. how do i mount that?
[18:37:35] <jdlrobson>	 i'm in a bit of a pickle now
[18:37:47] <chicocvenancio>	 thanks bd808 
[18:38:13] <bd808>	 jdlrobson: its going to take a bit of a dance to move things, but it can be done
[18:38:22] * bd808 looks for any existing instructions
[18:38:43] <chicocvenancio>	 bd808: is there a way to mount it somewhere other than `/srv`and then moving it
[18:38:47] <chicocvenancio>	 ?
[18:39:24] <bd808>	 jdlrobson: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space
[18:39:41] <bd808>	 the extra trick for you is that you have a lot of data on the current /srv
[18:40:02] <bd808>	 so when you enable the role and then run puppet it will look like all that data is gone
[18:40:15] <bd808>	 its really not though, it will just be hidden by the newly mounted disk
[18:40:34] <jdlrobson>	 bd808: is there anything that can go wrong following these instructions that i should be wary of?
[18:41:10] <bd808>	 so you'll have to disable puppet, unmount /srv, remount it somewhere like /tmp, move the current /srv data over, unmount again, and finally remount at /srv
[18:41:32] <bd808>	 adding the role won't hurt anything, but it will hide the current /srv data
[18:41:36] <chicocvenancio>	 jdlrobson: I can help with that
[18:42:35] <jdlrobson>	 chicocvenancio: bd808 i'm terrible at all this sysadmining. I've been having awful trouble with basic stuff like logrotate this week :/
[18:43:57] <jdlrobson>	 the puppet role has been applied
[18:44:55] <bd808>	 jdlrobson: you should probably `vagrant halt` before you force the puppet run to provision the disk
[18:45:15] <bd808>	 so cd /srv/mediawiki-vagrant; vagrant halt
[18:45:18] <jdlrobson>	 running that now
[18:45:25] <bd808>	 then puppet agent -tv
[18:46:01] <bd808>	 after that's done, `mount` should show a mostly empty disk at /srv
[18:46:19] <chicocvenancio>	 !log tools killed wget that was hogging io 
[18:46:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:50:18] <jdlrobson>	 chicocvenancio: bd808 is it normal for halting to take so long?
[18:50:27] <jdlrobson>	 been at least 5 minutes
[18:51:37] <bd808>	 jdlrobson: at 100% disk usage it might not stop as hoped :/
[18:51:56] <jdlrobson>	 :(
[18:52:10] <jdlrobson>	 bd808: any ideas?
[18:53:52] <bd808>	 jdlrobson: I can jump in an help in a few minutes.
[18:54:14] <bd808>	 There are probably mediawiki logs that can be cleaned up to free some space.
[18:56:43] <jdlrobson>	 thanks bd808 
[18:59:44] <bd808>	 jdlrobson: ok, I've got a window of time to help. Where are you at in the process?
[18:59:50] <jdlrobson>	 bd808: looks like vagrant went down
[18:59:50] <jdlrobson>	 i think
[18:59:55] <jdlrobson>	 given 502 on http://reading-web-staging.wmflabs.org/
[19:00:06] <jdlrobson>	 role::labs::lvm::srv has been enabled
[19:00:09] <jdlrobson>	 but that's all i've done
[19:00:18] <jdlrobson>	 i guess running puppet now is the next stage?
[19:00:35] <bd808>	 yes: sudo puppet agent -tv
[19:01:34] * bd808 sees: "/dev/mapper/vd-second--local--disk on /srv type ext4 (rw)" from moun
[19:01:38] <bd808>	 *mount
[19:02:24] <bd808>	 jdlrobson: is the puppet run finished?
[19:03:23] <bd808>	 !log reading-web-staging Adding role::labs::lvm::srv to reading-web-staging-3 with jdlrobson
[19:03:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Reading-web-staging/SAL
[19:07:49] <bd808>	 !log reading-web-staging Moving old /srv to new disk on reading-web-staging-3
[19:07:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Reading-web-staging/SAL
[19:08:36] <bd808>	 I'll document how I'm doing this on https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space too to help the next person
[19:11:44] <jdlrobson>	 bd808: thanks
[19:11:45] <jdlrobson>	 !!
[19:20:09] <bd808>	 "Notice: Currently this role is only supported on Debian Jessie." -- gah. I can't remember why we dropped support for Trusty from Role::Labs::Mediawiki_vagrant
[19:26:06] <bd808>	 why does it keep asking me for mwvgrant's password? ://
[19:26:53] <bd808>	 oh!! crap because the sudoers file has been cleaned up I bet
[19:33:44] <bd808>	 !log reading-web-staging Rebooting reading-web-staging-3
[19:33:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Reading-web-staging/SAL
[19:46:38] <bd808>	 jdlrobson: this is not going great, but I haven't given up yet
[19:47:23] <jdlrobson>	 bd808: good to know. Got some testing we need to do there before the train rolls out today :/
[19:47:40] <jdlrobson>	 im gonna have to break for lunch soon though!
[19:47:44] <jdlrobson>	 stomach grumbling
[19:47:53] <jdlrobson>	 happy to pair when i get back if that helps
[19:48:07] <bd808>	 jdlrobson: yup. I'll ping when I'm done or giving up
[19:55:55] <bd808>	 !log reading-web-staging Disabled puppet on reading-web-staging-3
[19:55:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Reading-web-staging/SAL
[19:58:57] <chicocvenancio>	 bd808: anything I can do to help? Have a notebook and seemly steady connection now 
[20:00:02] <bd808>	 chicocvenancio: right now the problem is that something has changed about the Vagrant binary installed and it's having problems doing all the things to start the container.
[20:00:15] <bd808>	 I'm stepping through debugging it
[20:15:40] <jdlrobson>	 bd808: hey back
[20:16:27] <bd808>	 jdlrobson: the thing that is fighting me now is getting the LXC container that Vagrant manages back up and running.
[20:17:11] <bd808>	 there are a bunch of custom sudoer rules that make it work and for some reason the vagrant that is installed there is doing different things that are  not allowed by the current rules
[20:29:06] <bd808>	 jdlrobson: I think I got the wiki back up and running there with some local config hacks
[20:29:12] <jdlrobson>	 \o/
[20:29:20] <jdlrobson>	 bd808: i seem to remember having to run some kind of source command to get vagrant working correctly
[20:29:22] <bd808>	 puppet is currently disabled
[20:30:02] <jdlrobson>	 http://reading-web-staging.wmflabs.org/ bd808 is progress :)
[20:52:44] <bd808>	 jdlrobson: I'm done messing with things now. I will put up a puppet patch to fix the things I had to hack manually.
[20:53:31] <bd808>	 jdlrobson: I had to reboot the server at one point, so there may be manual stuff that someone needs to start up again. I think there were some screens running there before
[20:54:45] <jdlrobson>	 looks like i need to do a git update bd808 
[20:54:49] <jdlrobson>	 will take it from here :)
[21:03:42] <jdlrobson>	 bd808: Eeep no more LocalSettings ? http://reading-web-staging.wmflabs.org/
[21:04:33] <bd808>	 jdlrobson: there is a ton of stuff in /srv/mediawiki-vagrant/settings.d
[21:05:04] <bd808>	 I'm not sure what's going on for you there
[21:05:54] <jdlrobson>	 it looks like there is no LocalSettings.php any more inside /srv/mediawiki-vagrant/mediawiki
[21:06:50] <bd808>	 jdlrobson: make it a symlink to /srv/mediawiki-vagrant/settings.d/wikis/wiki/LocalSettings.php
[21:07:49] <jdlrobson>	 bd808: hmmm didnt help
[21:08:21] <jdlrobson>	 should i copy it instead of symlink?
[21:08:24] <bd808>	 jdlrobson: :/ I'm in a meeting now
[21:08:40] <jdlrobson>	 no worries ill work it out
[21:11:03] <chicocvenancio>	 jdlrobson: can I help? Are you having trouble inside vagrant? 
[21:11:24] <jdlrobson>	 chicocvenancio: it looks like im back in bussiness thanks to bd808 
[21:13:11] <chicocvenancio>	 Cool