[01:10:02] !log toolsbeta upgrade toollabs-webservice to 0.66 for qa T249390 [01:10:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [01:10:06] T249390: Regression in webservice 0.65 in argument and config parsing - https://phabricator.wikimedia.org/T249390 [01:29:07] !log tools.sqid killed an update job because it was causing excessive load on the dumps nfs servers [01:29:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sqid/SAL [05:39:28] As far as email forwarding and the new ttoolforge domain, what's the current plan for the target email address format to be used? maintainer@_tool_.toolforge.org , or _tool_.maintainer@toolforge.org ? [08:42:21] hi, I'm in the process of upgrading grafana to its latest version (T244208), production is done just now and no known issues [08:42:22] T244208: Upgrade Grafana to 6.7 - https://phabricator.wikimedia.org/T244208 [08:42:26] ok to go ahead with cloudmetrics ? [09:16:34] godog: +1 [09:17:11] kk, thanks arturo ! [09:21:16] {{done}} [09:28:49] godog: LGTM [09:28:57] https://grafana-labs.wikimedia.org/d/R7BPaEbWk/toolforge-ingress [09:30:07] aye, I'm glad grafana upgrades tend to be smooth [14:02:43] addshore: could you comment on https://phabricator.wikimedia.org/T208416? I'm not sure if that's blocked on response from you or response from me explaining what the question is :) [14:54:17] !log admin `root@cloudcontrol1003:~# cp /etc/inputrc .inputrc` to solve some bash shortcut weirdness [14:54:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:58:58] DSquirrelGM: Literally nobody has thought about that particular problem yet. For now the mail stuff will still be tools.wmflabs.org. [15:00:39] just hadn't seen any mention about it at all, tried doing a few test emails but nothing went through via a toolforge address, figured I should at least ask [15:01:18] didn't know if it was an error in setup, or just not done yet [15:01:57] all of our mail stuff is not quite right -- T249237 [15:01:58] T249237: Fix Cloud VPS and Toolforge mail servers to work with the modern internet - https://phabricator.wikimedia.org/T249237 [15:04:42] referring specifically to the email forwarding in this case [15:04:57] not generating emails [15:07:18] anyway, thanks for letting me know about the status - or lack thereof - on that [15:27:46] probably wouldn't have even thought about it myself if it wasn't for remembering that email forwarding failure someone mentioned a little while back [19:20:26] !log admin rotated password and api token for pdns servers on cloudservices1003 and cloudservices1004 [19:20:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [20:56:18] hi, I keep getting "Lost connection to MySQL server during query" in my query at https://quarry.wmflabs.org/query/43732 ; is this perhaps fixable? [20:58:00] Usually means your query is taking too long to execute [20:58:20] Erutuon: hmmm... usually the query killer built into quarry fires before the backend query killer does, but that might be out of tune right now [20:58:51] but as Reedy says this is often a sign that the SQL query is asking for more than the database can allow [20:59:02] * bd808 looks at the query [21:00:00] there is no `use $dbname;` stanza. What db does Quarry use by default? enwiki? [21:00:09] I was wondering the same [21:00:51] seems to be, yeah [21:01:59] "Impossible WHERE noticed after reading const tables" [21:02:03] thanks mysql! [21:02:07] Erutuon: If you take your SQL and plug it into https://tools.wmflabs.org/sql-optimizer/ it will give you some hints about things you might try changing. The row count that this hits is really, really high and probably what causes the timeouts [21:02:54] Using where; Using index; Using temporary; Using filesort [21:02:59] we have a winner [21:03:07] Dropping the ORDER BY would be a big help [21:03:43] the combination of LIMIT and ORDER BY is almost never what you really want [21:04:11] the ORDER BY makes the backend server compute all of the results and then the LIMIT throws away most of them [21:06:04] the`AND EXISTS(...)` bit is also really another LEFT JOIN. I'm not sure if makign that explict would help MariaDB's solver do a better job or not [21:06:22] but I think I have convinced myself that the ORDER BY is the big enemy here [21:06:24] bd808 : The query didn't run long enough to hit the timeout [21:07:19] rerunning with "use enwiktionary_p", which I forgot [21:08:26] against enwiktionary_p it has a chance of succeeding [21:08:57] it will still make a list of 27,048 rows and then throw away 27,038 of them [21:09:15] okay, so left join [21:09:52] that's the ORDER BY + LIMIT. I'm not sure the join change will do much about that [21:11:08] removed ORDER BY [21:12:49] heh, just filtering Special:BrokenRedirects would be faster I suppose [21:13:19] musikanimal: I'm pretty impressed by sql-opitimizer telling folks about logging_logindex. You make cool tools :) [21:13:46] thanks! :) [21:14:04] I tried to get it to detect common mistakes [21:14:30] Erutuon: :) probably. SpecialPages and API calls are nearly always faster than direct SQL on the Wiki Replicas [21:43:13] b!log extdist Added Reedy as projectadmin (T249575) [21:43:13] T249575: Unable to do actions on Horizon - https://phabricator.wikimedia.org/T249575 [21:43:33] !log extdist Added Reedy as projectadmin (T249575) [21:43:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Extdist/SAL [23:26:03] !log tools.proneval-gsoc17 CrashLoopBackoff. Looks like memory constraints on the 2020 Kuberenetes cluster as cause. Restarting with --mem=2Gi [23:26:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.proneval-gsoc17/SAL [23:35:38] !log tools deploy toollabs-webservice v0.66 T154504 T234617 [23:35:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:35:42] T234617: Toolforge. introduce new domain toolforge.org - https://phabricator.wikimedia.org/T234617 [23:35:43] T154504: Make webservice backend default to kubernetes - https://phabricator.wikimedia.org/T154504 [23:49:21] !log tools Running rebuild_all for Docker images to pick up toollabs-webservice v0.66 (T154504, T234617) [23:49:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:49:26] T234617: Toolforge. introduce new domain toolforge.org - https://phabricator.wikimedia.org/T234617 [23:49:26] T154504: Make webservice backend default to kubernetes - https://phabricator.wikimedia.org/T154504 [23:57:51] since 23 March my pywikibot cronjob requests are getting ... pip install ... requests. Is that a fix that is needed in the pywikibot install, the grid or what? [23:58:20] ... Packages necessary for pywikibot are outdated.