[00:57:34] Is Quarry down? [00:57:51] I'm getting "Can't connect to MySQL server on 'enwiki.analytics.db.svc.eqiad.wmflabs' ((2013, 'Lost connection to MySQL server during query')) [00:57:51] " when I try to run a query [01:00:03] chicocvenancio? zhuyifei1999_? [01:00:15] * zhuyifei1999_ looks [01:00:48] https://quarry.wmflabs.org/query/runs/all om my god [01:01:39] Actually, there's an error at the shell too [01:02:06] https://quarry.wmflabs.org/query/24721 looks to be alive to me [01:02:16] yeah works now IDK [01:02:41] https://i.imgur.com/X6NO9xo.png I was getting and error on the console there for a moment too [01:03:31] no idea [01:03:36] * SQL shrugs [11:36:36] !log Tools.stewardbots Add maintainer for revi [11:36:37] rxy: Unknown project "Tools.stewardbots" [11:37:05] !log stewardbots Add maintainer for revi [11:37:05] rxy: Unknown project "stewardbots" [11:37:05] rxy: Did you mean to say "tools.stewardbots" instead? [11:37:14] !log tools.stewardbots Add maintainer for revi [11:37:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [11:37:32] thanks [12:23:32] (03PS2) 10EddieGP: hiera: Kill some hiera paths [labs/private] - 10https://gerrit.wikimedia.org/r/423189 [12:23:33] (03PS2) 10EddieGP: Add note: most stuff here not used in cloud vps hiera [labs/private] - 10https://gerrit.wikimedia.org/r/423233 [17:19:17] Are the replication servers offline? [17:19:21] DB Error: Unable to connect to the Wikipedia DB. Using failover. please report if this message persist. [17:19:50] IABot is unable to access the enwiki database [17:20:16] andrewbogott: ^ [17:20:21] chasemp: ^ [17:22:19] Cyberpower678: !help will get the attention of whoever is oncall [17:22:53] Is that the error message? [17:22:53] chicocvenancio: thanks. :-) [17:23:21] chicocvenancio: that is the error message coming from tools.wmflabs.org/iabot [17:24:04] For performance it uses the replication DBs, but it's able to cope with server failure and use the failsafes. [17:25:17] What are the failsafes? [17:25:22] The API [17:25:36] But it will hit the API pretty hard. [17:26:08] Makes sense [17:26:15] Failing that it only partially loads the page. [17:26:35] The tool is still usable even if both fail but more difficult. [17:28:28] Quarry still works fine, it is unlikely to be the wiki replicas [17:32:58] Well the message isn't going away. [17:33:00] chicocvenancio: ^ [17:33:20] And the DB handler hasn't been modified lately. [17:34:57] chicocvenancio: AKAIK nothing has changed on my end in the last few days. [17:35:52] does it use labsdb1009 Cyberpower678 ? [17:36:01] paladox: no idea. [17:36:11] Cyberpower678 https://phabricator.wikimedia.org/T191149 [17:37:03] paladox: so the server is currently crashed? [17:37:11] labsdb1009 i think [17:37:20] Cyberpower678: which hostname are you connecting to? [17:37:32] enwiki.labsdb [17:37:34] * chicocvenancio was going to ask that [17:39:26] that resolves into dbproxy1010.eqiad.wmnet and connects to labsdb1010 for me [17:40:12] So it's not resolving to a crashed host. [17:40:53] I think your user is being denied [17:41:09] That's weird. Why is that? [17:41:33] https://phabricator.wikimedia.org/T190627 ? [17:42:14] Seems like it [17:42:34] Thanks zhuyifei1999_ [17:42:38] np [17:42:55] Hmm, why wasn't I informed of that ticket. [17:43:03] Oh wait, I am subscribed. [17:43:10] Cyberpower678: seems like the DBAs need you to change the way the tool is querying things [17:43:16] I never got any emails from that ticket. [17:43:50] zhuyifei1999_: ^ [17:43:52] Sorry about that [17:44:12] well, idk how phab handles emails [17:44:15] chicocvenancio: That's not related to the tool. That's a completely different bot task [17:44:34] But the user would be the same [17:44:45] If it is run under the same tool account [17:45:36] It actually runs from my VPS [17:46:00] Cyberpower678: https://phabricator.wikimedia.org/mail/ might give a hint on why a mail was not sent [17:46:53] zhuyifei1999_: Well I usually get all emails, but that one specific task was not emailing me. There might be others I'm not aware of. [17:47:10] Vps doesn't have db credentials, you're probably using the same credentials as the Toolforge one [17:47:17] chicocvenancio: yep. [17:47:34] The cyberbot tool is inactive, but I use it's credentials. [17:47:59] then you found the reason ;) [17:48:12] the ban is on the credentials [17:48:33] Cyberpower678: by the way [17:48:56] You're using the cyberbot credentials on iabot as well [17:49:14] The credentials for iabot are probably unaffected [17:49:47] chicocvenancio: I've disabled the offending bot job. [17:50:02] Any chance of unbanning cyberbot? [17:50:07] Please don't do the same troublesome query with it, but maybe use it for other jobs [17:50:33] iabot doesn't make offensive queries [17:50:37] Probably responding in the ticket will be enough, but I don't have the access to do it [17:51:42] Iabot is using cyberbot credentials, change that to iabot's own credentials [17:53:03] Ugh, all of the ticket emails went to my spam folder. [18:05:07] Cyberpower678: do you need help to change the credentials used? [19:28:09] chicocvenancio: nope. I'm not going to bother. I'm just going to wait for the unban to happen [20:40:32] chicocvenancio: thanks for your help earlier. [20:40:43] Didn't get a chance to thank you until now. :-) [21:36:16] You're welcome Cyberpower678