[09:20:59] <arturo>	 !log project-proxy disable to puppet for a controlled merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098 (T135046)
[09:21:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL
[09:21:02] <stashbot>	 T135046: Whitelist labs instances that need XFF header passed through the web proxy - https://phabricator.wikimedia.org/T135046
[12:04:41] <qgil>	 Test, please ignore.
[18:44:36] <jeh>	 !log admin create indexes and views for grwikimedia T245912
[18:44:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[18:44:39] <stashbot>	 T245912: Prepare and check storage layer for gr.wikimedia.org - https://phabricator.wikimedia.org/T245912
[19:44:26] <mdaniels5757>	 Is something up with login.tools.wmflabs.org? Can't SSH in (authenticates and shows the motd, but no ability to use the cli)...
[19:45:22] <jeh>	 mdaniels5757: I'll take a look
[19:45:43] <mdaniels5757>	 OK, thanks!
[19:48:37] <zhuyifei1999_>	 mad D sleeps
[19:49:05] <jeh>	 yeah, looks like some user processes are eating up all the shared resources
[19:49:35] * bd808 sighs
[19:49:54] <bd808>	 want to bet on it being pywikibot or a java process?
[19:50:09] <zhuyifei1999_>	 I bet the gzip one
[19:50:14] <jeh>	 ^
[19:50:35] <zhuyifei1999_>	 17046 pts/17   DN+    0:54 gzip -cd topicsForAllWikipediaPages.csv.gz
[19:51:40] <jeh>	 I've killed the gzip process and will reach out to the user
[19:52:26] <mdaniels5757>	 OK, thanks so much!
[20:09:08] <jeh>	 !log tools update default security group to allow prometheus01.metricsinfra.eqiad.wmflabs TCP 9100 T250206
[20:09:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:09:15] <stashbot>	 T250206: Deploy a proof of concept prometheus server in cloudvps to replace shinken - https://phabricator.wikimedia.org/T250206
[20:09:29] <musikanimal>	 ahh
[20:09:29] <musikanimal>	 arturo: looks like google-api-proxy needs to be whitelisted to have the XFF header, see the puppet role https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/role/manifests/labs/google_api_proxy.pp$37
[20:10:10] <jeh>	 !log cloudinfra update default security group to allow prometheus01.metricsinfra.eqiad.wmflabs TCP 9100 T250206
[20:10:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL
[20:16:23] <bd808>	 musikanimal: pretty late at night for a.rturo. I'll see if I can get that added
[20:17:38] <musikanimal>	 okay, thank you! more at https://phabricator.wikimedia.org/T135046#6060220. I'm wondering if we need the puppet role? Can't we just restrict incoming traffic via the security groups on Horizon?
[20:18:52] <bd808>	 in theory yes, a security group would work. But i trust Puppet more
[20:19:06] <musikanimal>	 okay, well I trust you more, so puppet it is! :)
[20:22:54] <bd808>	 !log project-proxy Added google-api-proxy.wmflabs.org & googlevision-api-proxy.wmflabs.org to profile::wmcs::novaproxy::xff_fqdns (T135046, T250312)
[20:22:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL
[20:22:58] <stashbot>	 T135046: Whitelist labs instances that need XFF header passed through the web proxy - https://phabricator.wikimedia.org/T135046
[20:22:58] <stashbot>	 T250312: HTTP 403 Error when using google-api-proxy on VPS - https://phabricator.wikimedia.org/T250312
[20:23:52] <bd808>	 !log project-proxy Forced puppet run on proxy-01
[20:23:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL
[20:24:58] <musikanimal>	 yay! back in business :) thanks for the prompt assistance!
[20:25:20] <bd808>	 yw. thank you for finding the problem :)
[20:25:35] <bd808>	 !log project-proxy Forced puppet run on proxy-02
[20:25:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL
[21:07:21] <halfak>	 !log wikilabels deploying wikilabels-wmflabs-deploy:e98aae6
[21:07:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[23:20:55] <bd808>	 !log tools Building ruby25-sssd/base and children (T141388, T250118)
[23:20:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:21:00] <stashbot>	 T141388: Add a easy way to run a ruby webservice on tools - https://phabricator.wikimedia.org/T141388
[23:21:00] <stashbot>	 T250118: Request creation of wlmitaly VPS project - https://phabricator.wikimedia.org/T250118