[01:03:56] <notconfusing>	 Is anyone else having difficulty connecting to the database replicas? I get an unspecified error if I do `mysql --defaults-file=$HOME/replica.my.cnf -h wikidatawiki.analytics.db.svc.eqiad.wmflabs`
[01:12:31] <notconfusing>	 ok, it's just from my VPS, it works on tools-labs
[01:16:43] <Krenair>	 notconfusing, do you have a replica.my.cnf?
[01:18:21] <notconfusing>	 yes in the homedir
[01:19:07] <notconfusing>	 I get `ERROR 2026 (HY000): SSL connection error: unknown error number
[01:19:09] <notconfusing>	 `
[01:19:48] <notconfusing>	 It used to work, as of last checked about 4 days ago.
[01:21:44] <Krenair>	 notconfusing, what instance is this on?
[01:21:51] <notconfusing>	 wikidumpparse
[01:22:24] <notconfusing>	 oops that the project. whgi.eqiad.wmflabs
[01:24:04] <Krenair>	 notconfusing, and this replica.my.cnf only includes [config], a user= line and a password= line?
[01:24:31] <Krenair>	 what does `dpkg -S $(which mysql)` say?
[01:24:32] <notconfusing>	 scp'd directly over from toolforge
[01:25:31] <Krenair>	 have you tried --skip-ssl?
[01:25:54] <notconfusing>	 ah well I am use the mysql `mysql-community-client: /usr/bin/mysql`
[01:26:24] <notconfusing>	 it's not the mariadb version that's on tools labs. 
[01:26:56] <notconfusing>	 --skip-ssl works!
[01:27:03] <Krenair>	 uh
[01:27:06] <Krenair>	 what does `apt-cache policy mysql-community-client` say?
[01:28:06] <notconfusing>	 so, why has SSL suddenly recently changed? hrm. 
[01:28:57] <notconfusing>	 i really want to access the replicas with my python client, was just using myql-community-client to test why the connectivity had dropped
[01:30:27] <Krenair>	 what does it say?
[01:30:53] <notconfusing>	 https://paste.gnome.org/pt8wpnb5l
[01:32:17] <Krenair>	 http://repo.mysql.com/apt/debian
[01:32:18] <Krenair>	 what
[01:32:29] <Krenair>	 that is not a repository we add
[01:32:55] <notconfusing>	 no, installed a local mysql server so it must have come with that
[01:33:22] <Krenair>	 You should have mariadb-client-core-10.1 from debian.org
[01:34:28] <notconfusing>	 i understand that. im not trying to run cli queries anyway, i was just testing my real error which is:
[01:34:41] <notconfusing>	 via python: sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'enwiki.analytics.db.svc.eqiad.wmflabs' (timed out)"
[01:35:09] <notconfusing>	 it's good to know that i could connect with `--skip-ssl`. I'll come back to this tomorrow.
[01:35:30] <notconfusing>	 thanks for your assistance Krenair
[01:35:42] <Krenair>	 interesting
[01:36:05] <notconfusing>	 could an ssl cert have changed or something?
[01:36:37] <Krenair>	 possibly? I don't know what TLS certs are used on wikimedia DB servers
[01:36:58] <Krenair>	 I think that pymysql error is someting else though
[01:37:05] <Krenair>	 would be good to run that through strace and see what its really up to
[01:37:47] <notconfusing>	 thanks
[01:38:44] <Krenair>	 should have something like this:
[01:38:48] <Krenair>	 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
[01:38:49] <Krenair>	 connect(3, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("10.64.37.14")}, 16) = 0
[01:38:49] <Krenair>	 setsockopt(3, SOL_IP, IP_TOS, [8], 4)   = 0
[01:38:49] <Krenair>	 setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[01:38:49] <Krenair>	 setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[01:38:50] <Krenair>	 recvfrom(3, "Y\0\0\0\n5.5.5-10.1.38-MariaDB\0\362M\205\rZ"..., 16384, 0, NULL, NULL) = 93
[01:41:25] <Krenair>	 notconfusing, it's possible you have some security group rule or iptables rule that prohibits it?
[01:41:46] <Krenair>	 would be unusual as we don't normally restrict outbound traffic
[01:47:36] <Krenair>	 It looks like Wikimedia DB servers talking TLS base it on the private production Puppet CA
[01:47:46] <Krenair>	 which as a labs instance won't be trusted by default
[01:49:05] <Krenair>	 although.. hmm
[01:49:34] <Krenair>	 I'm not actually sure anything is set up to talk TLS
[11:05:56] <chicocvenancio>	 !log tools PAWS: change confimap to reference WMHACK 2019 as busiest time
[11:05:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[11:22:10] <chicocvenancio>	 !log tools PAWS: restart hub to get new configured announcement
[11:22:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:29:24] <Wurgl>	 zhuyifei1999_: Anything new with https://phabricator.wikimedia.org/T141100 ? Thanks
[15:31:35] <zhuyifei1999_>	 Wurgl: not really
[15:42:36] <Wurgl>	 :-(
[16:41:18] <Lucas_WMDE>	 is anyone around who can tell me the difference between the toolforge and toollabs things in operations/puppet?
[16:41:40] <Lucas_WMDE>	 they both seem to add largely the same scripts in /usr/local/bin/, and I want to add another one, so I’m confused where it should go
[16:42:03] <Lucas_WMDE>	 (if no one’s around I’ll just upload a patch that adds it to both places and it can be discussed on Gerrit I guess)
[18:07:35] <zhuyifei1999_>	 Lydia_WMDE: I think it should go to toolforge
[18:07:51] <zhuyifei1999_>	 bstorm_ knows more :)
[18:08:16] <bstorm_>	 Uh oh.  What do I know?
[18:08:19] * bstorm_ reads back
[18:08:48] <bstorm_>	 Ah in puppet: toollabs is mostly deprecated, but some things are still there because we haven't actually migrated *everything*(
[18:08:49] <mutante>	 that's the wrong user
[18:08:55] <mutante>	 Lucas isnt on the channel anynore
[18:09:11] <mutante>	 so you will find that on his gerrit change
[18:10:12] <mutante>	 it's better anyways. no need for realtime 
[18:10:17] <bstorm_>	 mutante: what change?
[18:10:32] <mutante>	 12:42 < Lucas_WMDE> (if no one’s around I’ll just upload a patch that adds it to both places and it can be discussed on Gerrit I guess)
[18:10:43] <bstorm_>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/510750?
[18:11:03] <mutante>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/510750 looks like it indeed
[18:11:10] <zhuyifei1999_>	 oh it's the wrong user... didn't see that :P
[18:11:21] <mutante>	 it's his latest one and he said he wants to add a new script
[18:11:35] <zhuyifei1999_>	 just tab-completed L ad found that one that ends with 'WMDE'
[18:21:14] <bstorm_>	 Commented and such.  Hopefully what I'm saying is helpful :)
[18:21:32] <mutante>	 :) i'm sure it is
[18:58:01] <wm-bot>	 !log tools.quickcategories <lucaswerkmeister> deployed f53ae8ac71 (remove commands help from index page)
[18:58:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[21:06:41] <zhuyifei1999_>	 !log video repool v2ccelery on encoding02, drain encoding01
[21:06:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL