[07:18:02] 10serviceops, 10IABot-Team: IABot 301 POST requests - https://phabricator.wikimedia.org/T274090 (10jijiki) [07:19:16] 10serviceops, 10IABot-Team: IABot 301 POST requests - https://phabricator.wikimedia.org/T274090 (10jijiki) [07:44:38] 10serviceops, 10Performance-Team, 10WikimediaDebug, 10Patch-For-Review: create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10MoritzMuehlenhoff) Why create new ones, this only creates needless churn by updating the Debug Extension twic... [07:52:05] 10serviceops, 10DBA, 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Joe) [07:59:09] 10serviceops, 10DBA, 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Marostegui) I have fixed this issue and mysql is now running. ` root@testreduce1001:/var/lib# mysql testreduce -e "show tables" +----------------------+ | Tables_in_testreduce | +------... [07:59:59] 10serviceops, 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Marostegui) Removing the #dba as we don't own this, the initial problem has been fixed. Mysql is now up - I will stay on the task though in case you need some more input. [08:26:10] 10serviceops, 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Joe) Thanks a ton @Marostegui! [09:00:41] 10serviceops, 10IABot-Team, 10InternetArchiveBot: IABot 301 POST requests - https://phabricator.wikimedia.org/T274090 (10Aklapper) [10:16:03] 10serviceops, 10Prod-Kubernetes: Kubernetes prod and staging share the same cluster key in hiera - https://phabricator.wikimedia.org/T273866 (10JMeybohm) [10:16:07] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10JMeybohm) [10:40:23] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10LSobanski) [11:26:33] 10serviceops, 10SRE, 10User-jijiki: ifup@eno1.service fails on mc* hosts after 4.19.171-2 upgrade - https://phabricator.wikimedia.org/T273918 (10jijiki) 05Open→03Resolved a:03jijiki [11:26:38] 10serviceops, 10SRE, 10User-jijiki: ifup@eno1.service fails on mc* hosts after 4.19.171-2 upgrade - https://phabricator.wikimedia.org/T273918 (10jijiki) thanks @elukey ! [14:27:40] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement switching of staging clusters - https://phabricator.wikimedia.org/T269835 (10akosiaris) >>! In T269835#6800424, @JMeybohm wrote: >>> When switching from staging-eqiad to staging-codfw (and vice versa) we would need to: >>> * Ensur... [14:29:29] I really want that to be Survive Ops [14:29:35] in the topic. [16:02:15] _joe_: ping [16:27:31] mutante, reg https://phabricator.wikimedia.org/T274034#6809737 .. is the /etc/my.cnf file on testreduce1001 puppetized or locally created? [16:28:19] i have a more generic qn to whoever: is there a way to know what files on a server come from puppet? [16:32:20] reimage it and see if they come back :D [16:32:47] looking at the puppet catalog yes, and if you don't have root you could check it with the puppet compiler in CI [16:32:56] but you have to extract them, a bit painful [16:37:58] if I want to know about a specific file I could remove it and rerun puppet... (assuming it's not going to break the service if gone) [16:38:47] most (but not all) the files managed by puppet have a header that says so too, fwiw [17:01:41] thanks. :) [17:08:40] 10serviceops, 10SRE, 10Epic: Track and remove jessie based container images from production - https://phabricator.wikimedia.org/T249724 (10hashar) [18:02:10] subbu: I will check in few minutes, but to the generic question, you can mv the file in question to /tmp, run puppet agent -tv, see if it gets recreated or not, if not, move it back from /tmp, or diff both versions [18:02:24] dont have to reimage for that [18:08:31] <_joe_> please don't do that with database files :) [18:08:34] jayme, _joe_: could one of you look at https://phabricator.wikimedia.org/T273521#6807963 and double check that my plan is sane? [18:08:43] <_joe_> sure lemme take a look [18:08:51] ack [18:12:08] subbu: the file is the default file that the puppet module mariadb installs if there is no custom config given [18:13:20] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team-TODO: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10Joe) >>! In T273521#6807963, @Legoktm wrote: > Also, is there a chance developers will ask to be able to pul... [18:13:25] method to find it was, i see some random string it, like in this case "is a default my.cnf template, used so that the module is auto-contained" and then grep -r "auto-contained" in the operations/puppet repo and it shows: modules/mariadb/templates/default.my.cnf.erb [18:13:29] <_joe_> legoktm: I would try to go the credentials way rather than the IP authn way [18:13:33] comment in it say "Do not use it, instead, use separate .cnf templates for each type of server" [18:13:49] <_joe_> anyways, going to log off for now [18:13:53] so the fix is probably to add a custom template [18:13:56] <_joe_> ttyl~ [18:13:56] mutante, ok ... so, to fix the config issue that manuel identified, i suppose it then needs a puppet patch with a custom config ? [18:14:04] mutante, ack [18:14:05] _joe_: thanks [18:14:10] subbu: yes, that, i think so [18:14:39] or the data needs to move to the place that is used in the default [18:14:46] but since the comment says to not use it... [18:15:04] we should add a custom config [18:15:43] back when I added that I just put the class on it with the defaults [18:15:47] ok [18:19:02] subbu: I can make the patch, thing is we just do this most simple case: [18:19:05] include ::profile::mariadb::generic_server [18:19:10] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Release-Engineering-Team-TODO: Create restricted docker-registry namespace for security patched images - https://phabricator.wikimedia.org/T273521 (10JMeybohm) I think we could make kubernetes use docker credentials distributed to the nodes by default, see h... [18:19:27] so that does not pass any parameters and everything is defaults [18:19:33] and I was hoping they are ok [18:19:57] but that is not good enough then for such a case [18:19:58] legoktm: have not tried, but should be straight forward to add credentials to k8s nodes [18:20:36] ok, I'll look into that route then [18:20:40] sounds good! [18:20:41] it seemed more straightforwarded to just put a generic db server on it for a test like that [18:20:51] thanks. [18:20:56] np [18:21:07] legoktm: we can do a test run with that in the new staging cluster to ensure we don't break image pulls that way [18:24:20] legoktm: I'm unsure if that (docker auth) works on sub paths of the registry, though [18:25:17] would it be an issue if we did authed pulls for non-private images? [18:25:29] I don' [18:25:39] * I don't think so. [18:26:03] At least it should not. Probably just adds a header to the requests. But in docker, you ever know :P [18:26:53] >.< [18:27:17] <_joe_> jayme: this is an nginx configuration though, hence the higher flexibility [18:28:45] yeah, I know. But there also is this weird trickery that docker does to check if it needs to authenticate (see wall-of-text in nginx conf) [18:36:30] 10serviceops, 10Excimer, 10Performance-Team, 10Sustainability (Incident Followup), 10User-jijiki: Verify Excimer's performance footprint - https://phabricator.wikimedia.org/T270152 (10Krinkle) 05Open→03Resolved a:03Joe [18:47:55] 10serviceops, 10WikimediaDebug, 10Patch-For-Review, 10Performance-Team (Radar): create mwdebug1004, mwdebug2003 and mwdebug2004 as mediawiki::canary_appserver - https://phabricator.wikimedia.org/T274023 (10Gilles) [18:52:20] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:40:18] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1391.eqiad.wmnet'] ` an... [19:55:31] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:56:19] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [19:59:48] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:00:08] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1389.eqiad.wmnet'] ` Of... [20:00:22] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:01:59] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [20:33:49] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Dzahn) >>! In T274034#6809737, @Marostegui wrote: > I noticed some issues issues there: > - /etc/my.cnf has: `datadir = /srv/sqldata` but there's nothing ther... [20:37:11] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1390.eqiad.wmnet'] ` an... [20:42:11] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1389.eqiad.wmnet'] ` an... [20:56:50] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by legoktm on cumin1001.eq... [21:09:47] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:10:41] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1305.eqiad.wmnet'] ` an... [21:15:38] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1304.eqiad.wmnet'] ` an... [21:17:04] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:21:48] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:26:54] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [21:31:25] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking), 10Patch-For-Review: mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Dzahn) https://puppet-compiler.wmflabs.org/compiler1003/27915/testreduce1001.eqiad.wmnet/index.html [21:35:10] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking), 10Patch-For-Review: mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Dzahn) deployed this change on testreduce1001, ran puppet. This changed the contents of `/etc/my.cnf]` like this: ` -datadir = /srv/s... [21:35:51] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking), 10Patch-For-Review: mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Dzahn) puppet did refresh the parsoid-rt service but please take a look as well [21:40:05] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking), 10Patch-For-Review: mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Dzahn) Please note there is now data in both /srv/sqldata and /var/lib/mysql, I applied the change to the config file as suggested by Manu... [21:40:44] subbu: https://phabricator.wikimedia.org/T274034#6813117 ff [21:41:33] the data dir of the running mariadb server matches what is in the /etc/my.cnf now, and puppet changed it [21:41:46] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking), 10Patch-For-Review: mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10ssastry) I am still seeing this error: ` ssastry@testreduce1001:~$ sudo service parsoid-rt status ... Feb 08 21:38:02 testreduce1001 nodej... [21:41:48] I did not manually restart the service but don't have to [21:42:41] still broken ... maybe scott or i will dig in there and see what is going on between nodejs & mariadb ... [21:42:59] but, curious why this broke after a server/vm reboot. [21:45:48] subbu: somehow when data was imported it went into /var/lib/mysql which was expected by a human or a script as the default. if we would have warned you to make sure it was /srv/sqldata as default like in prod.. it should not have happened I think [21:45:58] the reboot just restarted the service [21:46:15] now we matched config to reality though, so should be fine [21:46:22] and I can connect to mysql from commandline [21:46:50] i could connect to mysql from commandline even before your latest puppet patch. [21:46:56] and confirmed that the current process thinks /var/lib/mysql is the data dir. Manuel had fixed it already [21:47:03] just it would not have survived next reboot [21:47:05] now it will [21:48:43] ok. [21:49:11] looking at the status of parsoid-rt [21:49:16] puppet had told me it refreshed that [21:49:28] and not that it failed [21:50:05] yea, failed to connect to DB. looking at the credentials that tries to use [21:50:52] host: localhost, db: testreduce user: testreduce [21:51:14] trying that with mysql client manually [21:51:29] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1388.eqiad.wmnet'] ` an... [21:52:31] subbu: so this is making it weird, but.. i see the commandline used by parsoid-rt,, tells me the config file, it has db, user, pass, i use those to connect manually and it works just fine [21:52:39] right. [21:52:41] can I just restart it one more time? [21:52:54] i copy/pasted that password, it worked [21:53:01] i can repro the error with a simple test script in ~ssastry/test/foo.js [21:54:00] subbu: 'mysql -h localhost -u testreduce -p testreduce' and enter the password from /etc/testreduce/parsoid-rt.settings.js .. you get console..? i do [21:54:13] yes, on commandline it works properly. [21:54:31] maybe it is about resolving "localhost" [21:54:32] so, it is something with nodejs <-> mariadb probably.not sure why it broke now .. anyway, scott / arlo / i will poke at it. was the testreduce1001 rebooted after a buster upgrade? [21:54:59] localhost has IPv6 address ::1 [21:55:06] v6? [21:56:00] it is not impossible something changed there with the reboot.. hm [21:58:11] subbu: it is listening on tcp6 but not tcp it looks [21:58:50] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1397.eqiad.wmnet'] ` an... [22:02:28] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking), 10Patch-For-Review: mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Dzahn) `systemctl status parsoid-rt` does indeed show that `nodejs[16869]: Unable to connect to database` and that it is using credentials... [22:03:18] aha. ok. [22:06:57] subbu: yea, so next question is if we change nodejs or mysqld. interesting would be if it works if you just give it "2620:0:861:107:10:64:48:40" instead of "localhost", but i'll look at the mysqld config [22:09:45] so, i have a test script in ~ssastry/test/foo.js and if change localhost ot m5-master.eqiad.wmnet, that script works, but fails with localhost. so, my hunch is mysqld config needs tweaking. [22:10:42] and it didn't work with 2620:0:861:107:10:64:48:40 or ::1 but, i am mulit-tasking heavily .. so can also ask scott to investigate later if there is something on the nodejs end we can fix. [22:10:51] subbu: m5-master is a completely different story though [22:11:06] that was definitely not influenced by this reboot one way or another [22:11:24] ok [22:11:58] right, i just wanted to check that we could still connect to that one after reboot .. because that tells me it is specific to the db config on testreduce1001. [22:12:17] thanks for confirming that part did _not_ work, ack [22:16:44] it will be about bind-address and localhost, i am still on it [22:18:15] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1271.eqiad.wmnet', 'mw12... [22:26:52] subbu: i made it listen on v4 only instead of v6 only and restarted the parsoid-rt but .. it did NOT fix it.. where I totally expected that to be it [22:29:46] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Dzahn) I temp. added a `bind-address = 0.0.0.0` to the [mysqld] section of /etc/my.cnf and restarted mysqld. This made it listen on IPv4-only instead of IPv6-on... [22:31:24] hmm ... [22:34:20] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking): mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Dzahn) I wonder if nodejs could try to use the socket at `/run/mysqld/mysqld.sock` instead of TCP / localhost. [22:35:42] mutante, with socketPath option, it works. [22:36:21] so, i could switch it to that for now and you could invesigate (or not) what the problem with the localhost:port combo is separately with lower urgency. [22:37:00] subbu: yes, that was the idea to confirm if it is about resolving "localhost" or so which was my suspicion [22:37:07] that sounds good, probably better anyways [22:37:56] restarting mysqld one more time so that it's not in a temp hacked state [22:38:26] done. it is again listening on just IPv6 but the socket should work regardless [22:39:49] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1303.eqiad.wmnet'] ` an... [22:39:49] re-enabled puppet [22:40:17] ok .. i quickly verified with a local hack to the testreduce code and the config in /etc/testreduce and confirm socketPath works. [22:40:40] so, will push puppet patch for that ocnfig change and update the testreduce code to accept socketPath config value. [22:40:57] ok, great. this unbreaks it for you, probably wasnt a bad thing to do anyways and kind of confirms where the issue is [22:41:08] ack, thx [22:43:12] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw2245.codfw.wmnet'] ` an... [22:50:20] mutante, https://gerrit.wikimedia.org/r/c/operations/puppet/+/662789 [23:27:19] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:28:38] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:30:14] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:32:24] 10serviceops, 10SRE, 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10User-jijiki: Upgrade MediaWiki clusters to Debian Buster (debian 10) - https://phabricator.wikimedia.org/T245757 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqia... [23:56:24] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking), 10Patch-For-Review: mariadb failing on testreduce1001 - https://phabricator.wikimedia.org/T274034 (10Dzahn) Things work once nodejs is using the socket instead of connecting to localhost via TCP. Deployed the change above that switches to... [23:57:49] 10serviceops, 10Data-Persistence (Consulted), 10Parsoid (Tracking), 10Patch-For-Review: nodejs can't connect to mysqld via tcp/localhost any longer (was: mariadb failing on testreduce1001) - https://phabricator.wikimedia.org/T274034 (10Dzahn)