[03:51:39] <kart_>	 Which proxy now used for Labs instead of bastion.eqiad.wmflabs.org ?
[04:08:01] <kart_>	 Got it. ProxyJump is much better.
[10:48:38] <Lucas_WMDE>	 is it expected or a known issue that tools would slow down after moving from Trusty to Stretch?
[10:49:06] <Lucas_WMDE>	 in my case the tool was already using the Kubernetes backend, so I’m not sure if starting that from the Trusty or Stretch bastion even makes a difference, but still, there seems to be a slowdown
[10:49:18] <Lucas_WMDE>	 (I’ll submit a task with more details if no one replies)
[11:03:14] <wm-bot>	 I will let you know when I see Lucas_WMDE and I will deliver that message to them
[11:03:14] <arturo>	 @notify Lucas_WMDE not expected, please file a phab task
[11:04:26] <godog>	 hi! I'm trying to list all instances for a given wmcs project, as performed by ./modules/prometheus/files/usr/local/bin/prometheus-labs-targets
[11:05:05] <godog>	 though now 'niregion' for wikitech api is hardcoded, how can I get a similar list or detect the region an instance is running in to then pass that to niregion?
[11:06:54] <arturo>	 godog: checking
[11:07:04] <godog>	 I tried eqiad1 and eqiad1-r neither work tho
[11:07:06] <godog>	 arturo: thanks!
[11:09:26] <arturo>	 godog: my first reaction is surprise bc the script is using mediawiki API rather than proper openstack API usage
[11:10:14] <arturo>	 to know project/instances, one usually interacts with keystone/nova using observerenv credentials (public API, public and read-only credentials)
[11:11:28] <arturo>	 the openstack API is what this tool uses: https://tools.wmflabs.org/openstack-browser/
[11:12:36] <arturo>	 godog: we could work together in figuring out which concrete keystone/nova API query we need to fetch instances in a given project
[11:15:13] <godog>	 arturo: thanks for the pointer! I'll take a look at the tool and ping you back with more questions in case :)
[11:15:43] <godog>	 arturo: do you know if wikitech api niregion argument is supposed to check the new eqiad region too?
[11:16:09] <godog>	 I'm asking to check if I can get unblocked now by using the right niregion
[11:16:58] <arturo>	 I doubt wikitech API can do that, it's deprecated, I'm sure none of that was upgraded to support our "modern" use cases in current cloudvps
[11:17:19] <arturo>	 godog: here is another simple example of using the nova API from python to fetch instances of a project: ./modules/openstack/files/monitor/spreadcheck.py
[11:17:42] <arturo>	     client = novaclient.Client("1.1", project_id=project, **creds)
[11:17:42] <arturo>	     servers = client.servers.list()
[11:17:58] <godog>	 nice, seems simple enough
[11:18:38] <godog>	 and I should be able to hit nova api with readonly public credentials ?
[11:18:46] <arturo>	 yes
[11:19:25] <godog>	 awesome, thanks arturo for your help!
[11:20:24] <arturo>	 that example script loads credentials from this file: ./modules/openstack/templates/monitor/spreadcheck-tools.yaml.erb
[11:21:09] <arturo>	 godog: you are welcome :-) if you need help please create a phab task and I will help you with the code :-)
[11:22:26] <godog>	 totally!
[12:54:00] <arturo>	 !log tools add webservice security group to tools-sge-services-03/04
[12:54:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[15:26:21] <godog>	 I can't seem to load http://tools.wmflabs.org/sal known?
[15:32:16] <zhuyifei1999_>	 (don't think so)
[15:32:44] <Lucas_WMDE>	 it was slow yesterday as well, and right as I asked about it in this channel it recovered
[15:36:37] <godog>	 *nod* yeah still not loading for me, not sure if it does for you?
[15:37:19] <zhuyifei1999_>	 not for me at all
[15:37:59] <zhuyifei1999_>	 wait it just loaded
[15:39:08] <zhuyifei1999_>	 now it's loading fine, no idea why
[15:39:29] <godog>	 same here! maybe it is like Lucas_WMDE said, works as soon as we mention it
[15:40:50] <Lucas_WMDE>	 lol
[15:50:48] <arturo>	 :-)
[15:51:12] <arturo>	 there should be some underlying issue in Toolforge, worth opening a phab task
[16:26:19] <zhuyifei1999_>	 godog: it's a Heisenbug :P
[16:27:20] <zhuyifei1999_>	 (I `sudo become sal`-ed abut it loaded about as soon as I typed that command and pressed enter)
[16:27:45] <zhuyifei1999_>	 wait is it not loading again?
[16:28:10] <zhuyifei1999_>	 taking forever to exit from k8s shell as well
[16:28:26] <arturo>	 zhuyifei1999_: WFM
[16:28:30] <gtirloni>	 !log tools.sal restarted tool
[16:28:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL
[16:28:43] <gtirloni>	 it's up now
[16:28:52] <Lucas_WMDE>	 WFM?
[16:28:58] <Lucas_WMDE>	 ah, urbandictionary says “works for me”
[16:29:03] <Lucas_WMDE>	 my guess was “wait five minutes” :D
[16:29:13] <zhuyifei1999_>	 lol
[16:29:29] <arturo>	 :-P
[16:29:40] <gtirloni>	 everything looked fine, pods running etc.. but with the changes we made in the last few days, I found it better to restart it anyway
[16:29:52] <arturo>	 thanks gtirloni 
[16:49:04] <andrewbogott>	 !log rcm migrating 'beryllium' VM to another virt host
[16:49:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL
[17:16:50] <arturo>	 !log tools T213421 shutdown tools-services-01/02. Will delete VMs after a grace period
[17:16:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:16:53] <stashbot>	 T213421: Toolforge: move services nodes from eqiad to eqiad1 - https://phabricator.wikimedia.org/T213421
[19:14:52] <Bubuloido>	 Hi there. The tool https://tools.wmflabs.org/osm4wiki/ is down ("502 bad gateway"). The tool is used on hundreds of thousands of pages (templates see https://www.wikidata.org/wiki/Q5845436). Who can fix it?
[19:47:52] <zhuyifei1999_>	 Bubuloido: looks good to me?
[19:48:07] <zhuyifei1999_>	 another Heisenbug?
[19:53:12] <Bubuloido>	 zhuyifei1999: Try https://tools.wmflabs.org/osm4wiki/cgi-bin/wiki/wiki-osm.pl?project=en&article=Johnston_Atoll or https://tools.wmflabs.org/osm4wiki/cgi-bin/wiki/wiki-osm.pl?project=Commons&article=Category%3ACities_in_New_York&l=2 . You see more than a blank world map and no "502" message on the left?
[20:14:07] <bd808>	 Bubuloido: maintainers are kolossos and plenz apparently -- https://tools.wmflabs.org/admin/tool/osm4wiki
[20:16:55] <bd808>	 the code is a 1300 line perl cgi script that I'm not going to even try to understand :/
[20:41:43] <bd808>	 !log tools Building tools-package-builder-02 to replace tools-package-builder-01
[20:41:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:26:25] <zhuyifei1999_>	 +1. looks don't look like a toolforge-wide issue
[21:26:43] <zhuyifei1999_>	 *doesn't look
[21:51:00] <bd808>	 !log tools Upgrading tools-manifest to 0.15 on tools-cron-01
[21:51:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:05:29] <bd808>	 !log tools Upgrading tools-manifest to 0.16 on tools-cron-01
[22:05:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:09:26] <bd808>	 !log tools Upgrading tools-manifest to 0.16 on tools-sgecron-01
[22:09:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:20:41] <bd808>	 !log tools.wsexport Increased webservice memory limit to 6G to see if that helps with OOM crashes
[22:20:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wsexport/SAL
[23:37:22] <bd808>	 !log tools Shutdown tools-package-builder-01. Use tools-package-builder-02 instead!
[23:37:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL