[08:54:57] <leloiandudu>	 hi. not sure if it's the right place to ask, but what's wrong with the API? I intermittently started getting "Unrecognized value for parameter "meta": userinfo" since around 11 hours ago
[08:56:38] <arturo>	 !log admin [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code (T261724)
[08:56:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[08:56:42] <stashbot>	 T261724: cloudgw: evaluate / validate setup in codfw1dev - https://phabricator.wikimedia.org/T261724
[08:57:09] <arturo>	 leloiandudu: try in the #wikimedia-operations IRC channel or send an email to the cloud@l.w.o list
[09:00:39] <leloiandudu>	 arturo thank you!
[09:56:54] <mmecor>	 Hi. I’d like to ask a question on subdomains.
[09:56:54] <mmecor>	 Long time ago I created a tool on the subdomain wcdo.wmflabs.org and ChicoVenancio helped me to configure the addressing. I’d like to change the subdomain to wdo.wmflabs.org and put some redirections. Could anyone point me how to do it?
[09:56:54] <mmecor>	 Thank you.
[10:31:22] <arturo>	 !log admin [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code (T261724)
[10:31:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:31:26] <stashbot>	 T261724: cloudgw: evaluate / validate setup in codfw1dev - https://phabricator.wikimedia.org/T261724
[10:31:37] <arturo>	 mmecor: yes, I can take a look!
[10:33:21] <mmecor>	 thank you.
[10:33:22] <mmecor>	 i have an nginx running on it. flask/dash on top of it too. i'd do the changes in the url in the dash code, but the general configuration NAT i have no idea how to do it.
[10:34:12] <arturo>	 NAT?
[10:34:31] <arturo>	 by the way, could you please open a phabricator task about this? so we have some papertrail...
[10:36:58] <arturo>	 oh, what you seem to have right now is a simple web proxy
[10:38:05] <arturo>	 mmecor: it should be pretty simple, check this: https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet#Migrate_from_a_*.wmflabs.org_proxy_to_a_*.wmcloud.org_proxy
[11:29:22] <mmecor>	 Thanks. I'll take a look at it.
[14:36:47] <andrewbogott>	 !log admin running apt-get update && apt-get install -y facter on all cloud-vps instances
[14:36:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:10:46] <andalousie>	 hello, i have some problem on toolforge
[15:11:32] <andalousie>	 I managed to start a webservice with grid engine
[15:11:49] <andalousie>	 but the service isn't started
[15:13:33] <andalousie>	 in service.log 2020-10-21T15:11:03.173879 Throttled for 3 restarts in last 3600 seconds
[15:13:51] <andalousie>	 but I don't know where I can see the errors
[15:14:25] <arturo>	 andalousie: try looking for log files in the tool home directory
[15:15:34] <andalousie>	 the error.log is empty
[15:15:56] <andalousie>	 and the service.log only shows throttled for 3 restarts ...
[15:16:10] <arturo>	 mmm that's weird
[15:16:24] <arturo>	 sorry I'm on a meeting and I can't pay full attention to this
[15:17:00] <andalousie>	 ok
[15:17:46] <andalousie>	 my service should start from a bash script
[15:17:58] <andalousie>	 but I followed the help on wikitech
[15:18:56] <bd808>	 andalousie: what is the tool name? We had some issues a week or so ago with grid engine webservices that may be causing you strange problems.
[15:19:14] <andalousie>	 it's alex-wiki
[15:19:58] <andalousie>	 I compiled a OpenResty and it could be launched with my start_server.sh script
[15:20:31] <andalousie>	 but I can't run it as a webservice so that it could be visited outside
[15:20:54] <bd808>	 andalousie: oh, so you are not using any of our normal tooling?
[15:21:23] <andalousie>	 yeah, I didn't use normal tooling
[15:22:22] <andalousie>	 but I think a script could be launched by generic type of grid engine
[15:24:03] <bd808>	 andalousie: does your custom nginx config pick up the $PORT environment variable as the port for it to run on? That will be necessary for this to work on the job grid. Each process started by `webservice` gets a unique port that is dynamically assigned at the time the process starts.
[15:24:49] <andalousie>	 I used envsubst to generate a nginx.conf that will pick up $PORT
[15:25:27] <andalousie>	 I tested with shell environment, but I don't know if it could work on grid engine
[15:26:13] <andalousie>	 #!/bin/bash
[15:26:14] <andalousie>	 PATH=/data/project/alex-wiki/openresty/nginx/sbin:$PATH
[15:26:14] <andalousie>	 export PATH
[15:26:15] <andalousie>	 cd /data/project/alex-wiki
[15:26:15] <andalousie>	 envsubst < conf/nginx.template > conf/nginx.conf
[15:26:15] <andalousie>	 nginx -p `pwd`/ -c conf/nginx.conf
[15:26:23] <andalousie>	 it's like this
[15:26:52] <bd808>	 it should be possible to make it work, but honestly I do not have the time to work with you to debug a unique use of Toolforge today.
[15:27:46] <andalousie>	 umm
[15:27:48] <andalousie>	 thanks
[15:28:28] <andalousie>	 if there's some solution please inform me. I'm User:Alexander_Misel on Wikipedia
[15:29:48] <andalousie>	 bye
[17:58:31] <legoktm>	 !log tools pushed toolforge-buster0-{build,run}:latest images to docker registry
[17:58:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:19:22] <andrewbogott>	 !log grantreview rebooting grantreview-04; it's OOM
[20:19:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Grantreview/SAL
[20:20:29] <andrewbogott>	 !log mediawiki-vagrant rebooting mwv-builder-03; it's oom
[20:20:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-vagrant/SAL
[20:22:41] <andrewbogott>	 !log reading-web-staging nehpets; it's oom
[20:22:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Reading-web-staging/SAL
[21:14:02] <andrewbogott>	 !log cloudinfra switching secondary puppetmaster from cloud-puppetmaster-04 to cloud-puppetmaster-05
[21:14:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL
[21:17:48] <andrewbogott>	 !log cloudinfra deleting broken cloud-puppetmaster-04
[21:17:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL
[21:28:49] <Skynet>	 What's going on with cyberbot-db-01?
[21:28:55] <Skynet>	 It suddenly went down.
[21:29:19] <Skynet>	 Or more specifically I'm getting "No route to host errors"
[21:29:45] <Skynet>	 andrewbogott: ^
[21:29:50] <bd808>	 Skynet: from what origin, using what hostname?
[21:30:02] <Skynet>	 Everywhere
[21:30:04] <andrewbogott>	 the db host is in migration
[21:30:10] <andrewbogott>	 it's huge so will take a while
[21:30:17] <Skynet>	 :/
[21:30:23] <Skynet>	 It's in migration again?
[21:30:53] <andrewbogott>	 this is a different host
[21:31:07] <andrewbogott>	 https://lists.wikimedia.org/pipermail/cloud-announce/2020-October/000326.html
[21:31:28] <Skynet>	 andrewbogott: I meant my DB VM seems to be undergoing migration quite often.
[21:31:57] <Skynet>	 Last migration was during Wikimania Stockholm because the VM was suffering from insufficient resources.
[21:32:15] <bd808>	 so "quite often" is every 13 months?
[21:32:38] <Skynet>	 I guess when you you put it that way, wow time flies
[21:32:58] <bd808>	 Skynet: I guess you missed https://lists.wikimedia.org/pipermail/cloud-announce/2020-October/000326.html
[21:33:18] <bd808>	 which does show that instance being scheduled to move today
[21:34:47] <Skynet>	 I guess I did miss it.  I was in the middle of system testing.
[21:36:13] <bd808>	 Skynet: on the plus side, this migration is moving your vm to the ceph storage backend which should make future movements of the instance from one hypervisor host to another nearly instant as far as interruptions to the running system are concerned
[21:37:03] <andrewbogott>	 yeah — I can't promise there will never be downtime again but it will make moves like this a lot easier
[21:37:50] <Skynet>	 :-)
[21:38:14] <Skynet>	 Ping me when my VM goes up again, so I can resume my work
[21:38:33] <Skynet>	 andrewbogott: ^
[21:38:47] <andrewbogott>	 ok