[09:04:06] (03PS1) 10Volans: Add missing wmcs/monitoring dummy keys [labs/private] - 10https://gerrit.wikimedia.org/r/439856 [09:06:07] (03CR) 10Arturo Borrero Gonzalez: [C: 032] Add missing wmcs/monitoring dummy keys [labs/private] - 10https://gerrit.wikimedia.org/r/439856 (owner: 10Volans) [09:06:25] (03CR) 10Volans: [V: 032] Add missing wmcs/monitoring dummy keys [labs/private] - 10https://gerrit.wikimedia.org/r/439856 (owner: 10Volans) [13:10:19] did something change that means only project admins can edit Hiera for a project now? [13:10:50] andrewbogott hi, i carn't seem to edit hiera here https://horizon.wikimedia.org/project/instances/7cbcff06-d7a4-4b5c-bebf-5fda785b79fa/ (i am not a project admin but am a member) but i remember being able to edit that. [13:11:16] I think that is correct behavior [13:11:42] else everyone on toolforge would be able to edit tools hieara... [13:11:49] *hiera [13:13:04] ok [13:13:18] mutante will need to make me a project admin please :) [13:22:05] paladox: chicocvenancio is correct — in theory you need rights to delete a server in order to edit puppet for that server. [13:48:57] andrewbogott wondering could you add me as project admin of https://horizon.wikimedia.org/project/instances/7cbcff06-d7a4-4b5c-bebf-5fda785b79fa/?tab=instance_details__audit please? [13:51:26] paladox: horizon urls like that aren't really transferrable, they only work for you because of local state on your browser [13:51:34] oh [13:51:41] but in any case, I don't usually mess with project membership unless it's actually my project [13:51:44] andrewbogott ok, the planet project. [13:51:50] :) [13:55:16] paladox: let's just use the repo for Hiera, then we don't run into these issues anymore. [13:55:22] ok [13:57:09] i coudl swear it was different before [13:57:24] and you edited Hiera several times [13:57:31] * chicocvenancio hopes you're misrembering [13:58:01] i am also sure too [13:58:05] as i edited hiera [13:58:10] when not a project admin [14:05:25] paladox: try puppet now [14:05:30] thanks! [14:05:47] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::planet::meta_link in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/planet.pp:15:18 on node planet-hotdog.planet.eqiad.wmflabs [14:05:50] mutante new error now [14:05:55] just need to add the meta link [14:05:59] yes [14:06:48] mutante https://gerrit.wikimedia.org/r/c/operations/puppet/+/439914/ [14:08:36] paladox: done [14:08:42] mutante thanks! :) [14:09:53] mutante puppet works now :) [14:11:04] nice :) [15:19:41] musikanimal: can I explain https://quarry.wmflabs.org/query/27580 ? I'm getting `SHOW EXPLAIN failed: Target is not running an EXPLAINable command` with https://tools.wmflabs.org/sql-optimizer [15:20:26] looking [15:29:25] hmm, not sure why it's not working. That is definitely EXPLAINable! [15:30:16] running EXPLAIN manually, I see that it takes quite some time (over a second), that must be part of it [15:31:03] it could be https://phabricator.wikimedia.org/T195836 [15:31:22] or it could be a limitation on some mariadb versions where explain actually runs subqueries [15:31:38] which means some explains are very slow [15:33:23] no not that, I don't think [15:33:37] * zhuyifei1999_ got the explain manually [15:35:29] musikanimal: regarding T195836, I don't have a good solution right now [15:35:30] T195836: Cannot reliably get the EXPLAIN for a query on analytics wiki replica cluster - https://phabricator.wikimedia.org/T195836 [15:35:50] that's what the sql-optimizer is meant to be, a workaround [15:35:56] my suggestions is to setup a temporary workaround, where you get 2 connections on the same host (using @@GLOBAL.hostname) [15:36:18] until we can figure out a proper solution [15:36:24] I've managed to get a workaround, this isn't because of T195836 [15:36:38] yes, I am not saying it is related now [15:36:46] just an unrelated comment [15:36:50] oh okay :) [15:37:00] well, not totally unrelated [15:37:07] on the same subject :-) [15:37:56] if someone has an idea (that doesn't degrade the high availability), please shere it on the ticket [15:39:11] will do [15:48:07] zhuyifei1999_: I figured it out, indeed it's a timing issue. The query is auto-killed while EXPLAIN is still running [15:48:21] why does it get autokilled? [15:49:08] we just want to run EXPLAIN, not actually let the query finish, which could take forever if it's inefficient [15:49:19] I do this with `SET max_statement_time` [15:50:39] but I can try to go back to the old system of using `KILL QUERY`, that way we can run it after EXPLAIN is done [17:29:15] (03PS1) 10Jforrester: Add #wikimedia-ri-ci with relevant repos and Phab project [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/439984 [17:31:48] o/ cloud folks. Our team wiki http://commtech.wmflabs.org/ (mwvagrant) went down recently. This issue seems most related https://phabricator.wikimedia.org/T116500 but permissions seem okay at first glance. I would appreciate help. Thanks. :) [17:32:56] (03CR) 10BearND: [C: 04-1] "Minor question inline." (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/439984 (owner: 10Jforrester) [17:40:38] !log deployment-prep deploying ores 36037b6 [17:40:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [17:46:48] (03CR) 10Jforrester: Add #wikimedia-ri-ci with relevant repos and Phab project (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/439984 (owner: 10Jforrester) [18:02:48] (03CR) 10BearND: [C: 031] Add #wikimedia-ri-ci with relevant repos and Phab project (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/439984 (owner: 10Jforrester) [18:04:28] Niharika: Bryan is the resident mediawiki-vagrant export and he is away at the moment [18:04:39] Expert! I meant [18:04:45] :D [18:05:06] Yeah, I know. It's alright. I think it's something up with mediawiki and nothing to do with the infrastructure there. [18:05:12] Thanks for replying! [18:20:42] (03CR) 10Merlijn van Deen: [C: 032] "I think you will probably need to add the Kanban board as well if it removes the old tags, but it depends on how phabricator streams the e" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/439984 (owner: 10Jforrester) [18:21:08] (03Merged) 10jenkins-bot: Add #wikimedia-ri-ci with relevant repos and Phab project [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/439984 (owner: 10Jforrester) [18:21:20] (03CR) 10jenkins-bot: Add #wikimedia-ri-ci with relevant repos and Phab project [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/439984 (owner: 10Jforrester) [18:22:36] !log tools.wikibugs Updated channels.yaml to: 7b71ab65e663a9f2b186d695e6bf94902beb319e Add #wikimedia-ri-ci with relevant repos and Phab project [18:22:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [19:19:12] Does anyone know of a technique to follow page redirects -- purely in SQL? I know how I might do it combining the database replicas and getting wikitext trhough the API. I'm trying to join every user who received a "thanks" with their revision history. The issue is that the user_name is stored in the thanks logs and so won't join to rev-history if they changed their username. [19:34:12] notconfusing: iirc the page table has a redirect target, but let me check [19:34:50] hm, no, that's just whether it is a redirect or not [19:35:37] but you might be able to get it from pagelinks in this case [19:48:30] @valhallasw`cloud thanks for the double check. I'll look into pagelinks and otherwise just do it with mwclient [19:49:12] (03PS1) 10Ottomata: Add kafka_mirror_maker cert [labs/private] - 10https://gerrit.wikimedia.org/r/440008 [19:49:32] (03CR) 10Ottomata: [V: 032 C: 032] Add kafka_mirror_maker cert [labs/private] - 10https://gerrit.wikimedia.org/r/440008 (owner: 10Ottomata) [19:49:43] notconfusing: I would first check how many users this would really be about -- getting that should be relatively easy (just count NULLs, I guess), and will tell you if mwclient will scale reasonably [19:52:26] notconfusing: last option I can think of is join with logging on log_action='renameuser' and log_namespace=2 and log_title= [20:41:06] hey, project storage IO is slow on tools bastion (as in 4.7s for a simple `ls` slow) [20:43:34] MaxSem: it's pretty responsive atm, not sure if I missed someone hammering it or what [20:43:58] hmm, now it's fast for me too [20:44:18] a job has finished :P [20:44:37] valhallasw`cloud this is for aobut 8 users of a population of 10,000 users on plwiki. I think the mwclient method is good enough for me. [20:44:47] and seem to have it working [20:51:32] (03PS1) 10Ottomata: Regenerate all certificates that were signed by the now decommed puppetmaster02 [labs/private] - 10https://gerrit.wikimedia.org/r/440016 (https://phabricator.wikimedia.org/T195686) [20:51:59] (03CR) 10Ottomata: [V: 032 C: 032] Regenerate all certificates that were signed by the now decommed puppetmaster02 [labs/private] - 10https://gerrit.wikimedia.org/r/440016 (https://phabricator.wikimedia.org/T195686) (owner: 10Ottomata)