[00:03:55] <wikibugs>	 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3575183 (10Legoktm) I'm not fully aware of the technical considerations in this task, but is it possible to do <whatever>.eqiad.wmcloud in...
[00:42:47] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1412 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[01:22:47] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1412 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:28:52] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[01:49:23] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1419 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[02:08:54] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0]
[02:24:23] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0]
[02:47:40] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[03:22:39] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0]
[03:43:42] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1434 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[04:38:41] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1434 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:43:43] <wikibugs>	 10Toolforge, 10Tools: wsexport tool leaking files in /tmp - https://phabricator.wikimedia.org/T166337#3593621 (10Liuxinyu970226)
[10:54:19] <wikibugs>	 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that load resources from third party sites - https://phabricator.wikimedia.org/T172065#3503707 (10Liuxinyu970226) @zhuyifei1999  > https://tools.wmflabs.org/intuition/: https://translatewiki.net/w/i.php?title=Special:TranslationSta...
[14:54:53] <wikibugs>	 10PAWS: PAWS - Redirect loop detected - https://phabricator.wikimedia.org/T175454#3593792 (10Liridon)
[15:24:46] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[15:59:44] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:28:28] <musikanimal>	 enwiki.labsdb down?
[18:31:12] <musikanimal>	 !help
[18:31:12] <goat-bot>	 musikanimal: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[18:31:54] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[18:32:23] <musikanimal>	 guess it's time to switch to wikireplica-analytics.eqiad.wmnet :)
[18:36:05] <wikibugs>	 10cloud-services-team: Replicas are unresponsive - https://phabricator.wikimedia.org/T175457#3593898 (10MusikAnimal)
[18:36:15] <wikibugs>	 10cloud-services-team: Replicas are unresponsive - https://phabricator.wikimedia.org/T175457#3593910 (10MusikAnimal) p:05Triage>03High
[18:36:24] <musikanimal>	 I want to put unbreak now but I'm always afraid to do that
[18:37:37] <wikibugs>	 10cloud-services-team: Replicas are unresponsive - https://phabricator.wikimedia.org/T175457#3593912 (10MusikAnimal) p:05High>03Unbreak!
[18:37:49] <musikanimal>	 alright fine being bold and moving to UBN
[18:38:15] <musikanimal>	 hmm okay it's working now
[18:38:25] <musikanimal>	 see that's all I had to do, create a ticket for it to fix itself :)
[18:39:21] <wikibugs>	 10cloud-services-team: Replicas are unresponsive - https://phabricator.wikimedia.org/T175457#3593913 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Working now... closing!
[19:06:52] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:26:11] <shinken-wm>	 PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[20:01:09] <shinken-wm>	 RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0]
[21:59:49] <Zppix>	 bd808:  did the way to get into a vps change? my config is setup like on help:access section Accessing instances with ProxyCommand ssh option  but i still get ssh: Could not resolve hostname gerrit-mysql.git.eqiad.wmflabs: Name or service not known
[22:01:30] <zhuyifei1999_>	 Zppix: paste your ssh config
[22:02:06] <Zppix>	 Zhuyifei1999_ https://www.irccloud.com/pastebin/0y4uABHk/ssh%20config
[22:02:36] <zhuyifei1999_>	 it's in your ~/.ssh/config right?
[22:03:08] <Zppix>	 correct
[22:04:04] <zhuyifei1999_>	 can you ssh -vvv and see if 'Applying options for *.eqiad.wmflabs' appears in the first few debug lines?
[22:04:12] <Zppix>	 one sec
[22:04:40] <zhuyifei1999_>	 eg:
[22:04:41] <zhuyifei1999_>	 debug1: Reading configuration data /home/zhuyifei1999/.ssh/config
[22:04:42] <zhuyifei1999_>	 debug1: /home/zhuyifei1999/.ssh/config line 19: Applying options for *.eqiad.wmflabs
[22:05:05] <Zppix>	 https://www.irccloud.com/pastebin/CAt5sxN7/debug
[22:06:03] <zhuyifei1999_>	 hmm your ssh config is not read
[22:06:09] <Zppix>	 how do i fix?
[22:06:21] * zhuyifei1999_ googles
[22:06:45] <zhuyifei1999_>	 https://serverfault.com/questions/609334/ssh-is-no-longer-using-ssh-config
[22:07:02] <zhuyifei1999_>	 check the file permissions
[22:08:02] <Zppix>	 its normal read/write
[22:09:44] <Zppix>	 zhuyifei1999_:  im on windows fyi
[22:09:49] <zhuyifei1999_>	 uh
[22:10:59] * zhuyifei1999_ wonders if it will worth the time to read openssh source code and see how it handles windows acls
[22:11:20] <Zppix>	 brb
[22:17:10] <zhuyifei1999_>	 https://github.com/openssh/openssh-portable/blob/643c2ad82910691b2240551ea8b14472f60b5078/ssh.c#L473
[22:19:39] <zhuyifei1999_>	 _PATH_SSH_USER_CONFFILE is constant  '.ssh/config' https://github.com/openssh/openssh-portable/blob/7461a5bc571696273252df28a1f1578968cae506/pathnames.h#L85
[22:24:49] <zhuyifei1999_>	 https://github.com/openssh/openssh-portable/blob/22376d27a349f62c502fec3396dfe0fdcb2a40b7/readconf.c#L1717 <= if read_config_file is called it should fatal (when permission error) or be unreadable
[22:25:55] <zhuyifei1999_>	 so either openssh can't resolve your pw->pw_dir (which I can't explain), or your config is unreadable by openssh
[22:27:53] <zhuyifei1999_>	 fatal permission error looks like 'Bad owner or permissions on /home/zhuyifei1999/.ssh/config' which I reproduced by chmodding it 666
[22:32:04] <zhuyifei1999_>	 actually, https://serverfault.com/questions/609334/ssh-is-no-longer-using-ssh-config#comment725734_609337 is interesting. I'll check how ssh finds pw->pw_dir 
[22:33:23] <zhuyifei1999_>	 https://github.com/openssh/openssh-portable/blob/643c2ad82910691b2240551ea8b14472f60b5078/ssh.c#L575
[22:34:30] <zhuyifei1999_>	 so basically pw = getpwuid(getuid())
[22:36:40] <zhuyifei1999_>	 if you want you can compile a c code to check what it returns on windows.
[22:36:45] * zhuyifei1999_ has no idea
[23:09:42] <wikibugs>	 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3594082 (10bd808) >>! In T174860#3593331, @Legoktm wrote: > I'm not fully aware of the technical considerations in this task, but is it po...