[00:05:22] Krenair: great, deleting [00:45:06] andrewbogott, done Hiera:Deployment-prep [00:45:27] that was less disruptive than I was expecting [00:54:53] Krenair: awesome, thank you! [00:55:10] I was thinking about doing https://phabricator.wikimedia.org/T235218#5629788 [00:55:13] tools is the other scary one but I probably have to do that one myself [00:55:47] I could just do something like "rsync -ar root@cloud-puppetmaster-01:/var/lib/puppet/server/ssl cloud-puppetmaster-frontend-ssl" and then back up to the new host [00:56:02] of course this keeps copies of signed certs etc. which would get out of sync with the currently-live puppetmaster's set [00:56:26] but maybe we can sync again when we're ready to switch over [00:56:33] Yeah, we'll have to re-do the certs at the last minute [00:57:04] oh, and I haven't put my key in the list for root@ access in that project [00:57:51] so the other issue there involves geoipupdate right? [00:57:59] Is that package itself actually non-free? [00:58:11] yeah we decided to just ditch that one I think [00:58:18] oh great :) [00:58:33] IIRC the problem with it is that its only use is to get non-free data [00:58:42] 'k [00:58:58] I'm going to go cook dinner but ping me if you get blocked by anything and I'll check back in after [00:59:00] i.e. the MaxMind DB that is used by, for example, varnish servers to do GeoIP lookups in prod [00:59:15] I'm going to sleep, I probably won't do anything more today [00:59:18] Yeah, seems like we can live without that as long as we can make puppet happy [00:59:23] oh, ok! Have a good night, in that case :) [11:59:38] !log admin icinga downtime for 1h cloudcontrol1004, cloudnet1003, cloudvirt1017/1020/1022 for PDU operations in the rack T227542 [11:59:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:59:41] T227542: b7-eqiad pdu refresh (Tuesday 11/5 @12pm UTC) - https://phabricator.wikimedia.org/T227542 [13:55:28] !log tools created 3 new VMs: `tools-k8s-etcd-[4,5,6]` T236826 [13:55:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:55:34] T236826: Toolforge: new k8s: initial build of the new kubernetes cluster - https://phabricator.wikimedia.org/T236826 [14:27:53] paladox, looks like the hiera for the 'git' project is mostly your doing — would you migrate those settings to Horizon as well? [14:28:00] (And, are there other projects that are mostly yours?) [15:01:29] andrewbogott: I can though currently mobile, so will need to do that later. [15:01:41] paladox: ok! Just let me know when you finish, thanks [15:01:52] I think I’m in the planet project but doint think I have sudo for that one [15:01:56] Ok! [16:16:42] Hi, I guess it’s know that the *.wmflabs.org cert expires in 10 days? [16:16:43] I just got a warning about that [16:19:52] cteam, we have a ticket for ^ don't we? I can't find it :/ [16:20:29] andrewbogott: yeah, cert is here just needs to be deployed. Let me find the ticket [16:20:51] bd808: I don't need a link as long as someone is in charge of it and that someone isn't me :) [16:20:56] T237066 [16:20:57] T237066: Push renewed *.wmflabs.org certificate and new private key to cluster (expires 2019-11-16) - https://phabricator.wikimedia.org/T237066 [16:21:26] now why did my search string not match that... [16:22:30] I can handle that if nobody wants to :-P [16:22:47] yes please! [16:24:02] Thanks bd808 andrewbogott and arturo [16:27:37] will do tomorrow in my morning [16:44:56] !log tools restarted lighttpd based webservice pods on tools-worker-100[1-9] (T233347) [16:45:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:45:00] T233347: Remove access.log generation from default lighttpd.conf generated by `webservice` - https://phabricator.wikimedia.org/T233347 [17:01:42] !log openstack rebuild debian stretch image with latest bootstrapvz config [17:01:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [17:04:47] Hieu Pham !log tools restarted lighttpd based webservice pods on tools-worker-101[0-9] (T233347) [17:04:48] T233347: Remove access.log generation from default lighttpd.conf generated by `webservice` - https://phabricator.wikimedia.org/T233347 [17:06:22] phamhi: I think you need to !log that again without your name, if it was supposed to be a real log message [17:06:33] oh ..thanks.. [17:06:43] !log tools restarted lighttpd based webservice pods on tools-worker-101[0-9] (T233347) [17:06:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:34:23] !log tools restarted lighttpd based webservice pods on tools-worker-102[0-9] (T233347) [17:34:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:34:28] T233347: Remove access.log generation from default lighttpd.conf generated by `webservice` - https://phabricator.wikimedia.org/T233347 [17:38:15] !log tools restarted lighttpd based webservice pods on tools-worker-103x and 1040 (T233347) [17:38:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:36:31] @bd808 @bstorm_ Hi folks! I would need admin access to dashiki project https://tools.wmflabs.org/openstack-browser/project/dashiki to create a new instance for it, right? Should I create a Phab task requesting for permissions? This is for deploying the visualization dashboard https://phabricator.wikimedia.org/T226663. [18:37:04] * arturo waves srish_aka_tux [18:37:15] Hey! [18:37:33] That can be granted by the project admins, I think (which seems to be the analytics team) [18:37:45] srish_aka_tux: do you actually need a new instance? It looks like all the other dashboards are just a proxy pointing to the existing instance [18:37:57] We can do it by force as well, of course :) [18:38:08] nicer not to though [18:40:03] srish_aka_tux: I guess the TL;DR answer is that yes you would need to be a project admin to make a new instance or to setup a new proxy to an existing instance. And that the first folks to ask are the current admins for the dashiki project. [18:40:20] OoO right! I just need a domain then :-/ What would be the process for setting that up? For deploying the dashboard to one of the existing instances, I would need admin rights, maybe? [18:40:21] If they are not responsive then you can escalate to us for help [18:40:48] I'm responsive! Sorry, we've just had some data issues we've been working on [18:40:54] srish_aka_tux: I'll add you now [18:40:55] we do not generally force new admins into existing projects [18:42:35] srish_aka_tux: what's your username on the instances? [18:43:08] @milimetric it is srishakatux [18:43:38] ok, srish_aka_tux you're an admin [18:46:58] thanks @milimetric :) [19:08:15] @andrewbogott Hi! I just saw that you deleted this page https://wikitech.wikimedia.org/wiki/Hiera:Dashiki/host/dashiki-staging-01 and the one for prod in Wikitech. This means we no more need to add the hostname on Wikitech as we specify it while setting up the proxy in Horizon? [19:13:56] srish_aka_tux: I'm afk but will catch up in 30 minutes or so [19:14:25] @andrewbogott no problem :) [19:23:40] andrewbogott https://wikitech.wikimedia.org/wiki/Hiera:Git can be deleted now [19:35:53] srish_aka_tux: I don't entirely know what's happening in that project, but that config data now lives in Horizon. That same VM has an instance puppet tab there. [19:36:09] I can't link to it because links don't work like that in Horizon but if you log in you can probably see what I mean [19:36:12] paladox: thank you! [19:40:36] @andrewbogott Okay! thanks.. [19:41:07] srish_aka_tux: we're removing those pages from wikitech as part of the (very gradual!) move away from having wikitech know things about openstack [19:41:28] yw :) [19:45:22] @andrewbogott Yeah, I asked as they were linked from deployment instructions in here https://github.com/wikimedia/analytics-dashiki :) It seems like the on-wiki steps are needed, and the docs are outdated, but I will know more when I will do the deployment steps. [19:45:59] srish_aka_tux: web proxy management is also in horizon now [20:48:50] Hi, I have questions related to Cloud VPS? [20:48:52] !help [20:48:52] Zoranzoki21: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [20:48:57] Where I can access to panel? [20:49:16] Zoranzoki21: I'm not sure I know what you mean by panel, can you tell me more? [20:49:25] managing VPS [20:49:28] I'm guessing https://horizon.wikimedia.org/ [20:50:02] It requests 2FA [20:50:04] https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS#Cloud_VPS_instance_administration_and_configurations [20:50:34] Zoranzoki21: correct, 2FA on your Developer account via Wikitech is required to use Horizon [20:53:29] Will enable [20:56:05] Enabled, cool! [20:57:10] Why I see tools project? [20:57:44] Zoranzoki21: because you are a member of that project [20:58:02] you are not an admin there, so you should not be able to change anything [20:58:11] if you can... file a security bug please :) [20:58:14] Yes, it is right, but I was confused when I logged on horizon [20:58:22] But I can't change anything [20:58:39] Horizon is cool, I can't wait to get my VPS created :) [21:00:18] But I believe to I will get it soon, task is assigned to bstorm_ [21:01:20] Yup! [21:03:54] andrewbogott: I'm not sure where in Horizon that hiera config is [21:04:29] was looking but didn't seem obvious [21:04:35] milimetric: for a project or for a single VM? [21:05:05] uh... I believe these are per-vm, I don't know much about it other than they generated apache config sections [21:05:46] milimetric: So when you say "that hiera config" I assume you're referring to a previous conversation or something? [21:06:10] from above, srish_aka_tux was showing you some Hiera:Dashiki pages that you deleted [21:06:10] Project-wide puppet config is in the 'puppet' link in the sidebar in horizon [21:06:15] oh, ok [21:06:34] ah, found it [21:06:42] it's in the Puppet Config tab on the instance [21:06:43] thanks! [21:06:50] ah, yep [21:07:00] I was getting there but couldn't remember the names of the whole tab tree to get there [21:07:18] milimetric: if you want to just enter raw hiera you can scroll to the bottom and switch to 'yaml mode' [21:07:54] ah! cool, well, I'm not opposed to the little edit things, it's a simple enough change [21:08:01] will just have to update the docs, thanks! [22:12:12] !log tools pushed docker-registry.tools.wmflabs.org/maintain-kubeusers:beta to the registry to deploy in toolsbeta [22:12:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:23:44] Zoranzoki21: 👋🏻 [22:23:59] How long has it been since you stood up your VM? I can take a look at it [22:24:10] bstorm_: I connected now :D [22:24:14] Greaty [22:24:15] With ssh -J zoranzoki21@primary.bastion.wmflabs.org zoranzoki21@srwiki-dev1.srwiki-dev.eqiad.wmflabs [22:24:17] *great! [22:24:21] How I can make it easily [22:25:14] Generally, you can put some of that into a config file on your system in `.ssh/config` [22:25:22] Let me see if we have some info on that on the wiki [22:26:06] I tried to make .config file [22:26:09] But this happens: [22:26:10] ssh: Could not resolve hostname srwiki-dev1.srwiki-dev.eqiad.wmflabs: Name or service not known [22:26:15] Ah ok [22:27:14] So the part you want is here: https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyJump_ssh_option_(recommended) [22:27:38] https://www.irccloud.com/pastebin/Ii1DctwM/ [22:28:03] If you have the right things swapped in for then that should fix the problem [22:28:08] Content of file: https://phabricator.wikimedia.org/F31016690 [22:28:32] Are you on Windows? [22:28:40] Yes [22:28:43] It should be `.ssh/config` [22:28:47] not .config [22:29:05] That could be it [22:29:15] Yep, works [22:29:24] But it should be updated in documentation [22:29:24] Great! :) [22:29:50] I'll see if I can find that typo [22:30:13] https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_public_and_private_Cloud_VPS_instances shows .config for all [22:30:40] I see .ssh/config so far. Maybe I'm not finding it [22:30:42] Oh I am wrong [22:30:47] You are right [22:30:52] I am sorry :/ [22:31:04] How to install some software? [22:31:10] I can use apt-get or? [22:32:50] Zoranzoki21, you want to run mediawiki there don't you? [22:33:08] the usual way that happens is with https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS [22:33:50] Yes, right [22:35:18] !log tools upgraded libpython3.4 libpython3.4-dbg libpython3.4-minimal libpython3.4-stdlib python3.4 python3.4-dbg python3.4-minimal to fix an old broken patch T237468 [22:35:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:35:22] T237468: tools-k8s-master-01 (Kubernetes API server for toolforge) has failing puppet staleness cron - https://phabricator.wikimedia.org/T237468 [22:40:59] -bash: cd: /srv/mediawiki-vagrant: No such file or directory [22:41:13] I enabled role on horizon [22:44:28] Ok, works.. It takes some time to create [22:49:54] !log tools Disassociated floating IP 185.15.56.60 from tools-static-13, traffic to this host goes via the project-proxy now. DNS was already changed a few days ago. T236952 [22:49:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:49:59] T236952: Move tools-static.wmflabs.org behind project-proxy - https://phabricator.wikimedia.org/T236952 [22:50:56] !log toolsbeta deployed the new maintain-kubeusers to toolsbeta T215531 T228499 [22:50:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [22:51:00] T215531: Deploy upgraded Kubernetes to toolsbeta - https://phabricator.wikimedia.org/T215531 [22:51:00] T228499: Toolforge: changes to maintain-kubeusers - https://phabricator.wikimedia.org/T228499 [23:08:11] !log tools Dropped 59a77a3, 3830802, and 83df61f from tools-puppetmaster-01:/var/lib/git/labs/private cherry-picks as these are no longer required T206235 [23:08:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:36:38] Zoranzoki21: b.storm_ told me you are working through some issues getting MediaWiki-Vagrant working on your new instance? [23:36:45] Anything I can help with? [23:38:48] http://srwiki-dev.wmflabs.org shows me 504. I made a web proxy, bstorm_ told me how to create web security group I done it [23:39:09] Too I made cat /srv/mediawiki-vagrant/puppet/hieradata/local.yaml [23:39:11] with content: [23:39:16] role::mediawiki::hostname: srwiki-dev.wmflabs.org [23:39:45] It's doing that timeout thing, bd808...we just set up a sec group wide open on port 80 and 8080, which should fix that part of it [23:39:58] Not sure about the proxy setup myself (not sure I've done one) [23:40:28] hmmm... yeah I see it spinning in my browser. Let me see if I can spot anything strange about the proxy config [23:41:14] OH, I added web security group with option edit instance in https://horizon.wikimedia.org/project/instances/ [23:42:22] that's better :) [23:42:45] Doh! [23:42:46] now it looks like you need to run puppet inside the vagrant managed LXC container again [23:43:08] cd /srv/mediawiki-vagrant; vagrant provision [23:43:15] That's what it was, the security group needed to be applied to the instance. Good thinking, [23:43:39] bd808: LXC container... You think on vagrant ssh? [23:44:58] I run vagrant provision [23:45:34] Zoranzoki21: when you run `vagrant ...` from the /srv/mediawiki-vagrant directory on srwiki-dev1.srwiki-dev.eqiad.wmflabs, it works in basically the same way as running vagrant on your laptop. The main difference (for most people) is that instead of the wiki running inside of a Virutalbox VM, it runs inside an LXC container. [23:46:41] Ok.. Output of vagrant provision: https://pastebin.com/XsYJQ8TE [23:49:44] That looks positive [23:50:19] hmmmm. [23:50:23] Yes, right [23:51:18] I tried the full wikifarm setup for you too -- https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS#Run_a_wikifarm -- and I'm still seeing the "No wiki found" page which means the vhost name is not actually being used as expected [23:52:58] 4th puppet run was the fix? -- https://srwiki-dev.wmflabs.org/wiki/Main_Page [23:53:54] Lo [23:53:55] Lol [23:53:59] Thank you very much! [23:55:58] How to log in> [23:59:33] I made account via createAndPromote.php everything works, thank you for help very much!