[00:01:42] ah [00:02:05] I went away to check through each of the projects halfak is in for 300G storage instances, and when I come back he already appeared :D [00:02:20] harej, https://quarry.wmflabs.org/query/20587 [00:02:30] delightful, thank you [00:02:35] n/p duder [00:02:37] * halfak runs away again [00:05:09] halfak: u2041__ores_p is in enwiki.labsdb server right? I was thinking of switching the database server to the new wikireplica-analytics and I guess that might break it :( [00:05:19] https://phabricator.wikimedia.org/T172704 [00:05:56] I'd be happy to copy it if you give me a task and couple days. [00:06:40] I don't think it allows creating tool-owned databases (see the note) [00:06:51] Oh... yeah. That'll be a problem then, I guess. [00:07:20] so idk if such usage can be supported in the future, or if we can make a workaround [00:08:02] A workaround for maintained "dataset" tables would be preferable. [00:08:09] I'm willing to put my maintenance where my mouth is. [00:08:12] halfak: another thing: what am i supposed to do with the figures so that they're comparable? the figure for all-of-wikipedia is extremely bigger than the one that is wikiproject specific [00:09:03] harej, read paper. But for a shortcut, check out http://paws-public.wmflabs.org/paws-public/6877667/WikiProject%20Women%20writers.ipynb [00:09:31] Which I used to make https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Women_writers#Quick_analysis_of_historical_content_coverage_about_women_writers [00:10:27] * halfak actually leaves. [00:10:31] o/ [00:30:46] 10cloud-services-team, 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3466006 (10Dzahn) The host is down again (since 1d 8h 34m 8s) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=1&host=labvirt1015 [00:31:56] 10Cloud-Services, 10VPS-project-libraryupgrader: "library-upgrader" labs project request - https://phabricator.wikimedia.org/T116697#3529668 (10Legoktm) [00:32:38] 10VPS-project-libraryupgrader: Puppetize setup on upgrader-02.wmflabs - https://phabricator.wikimedia.org/T173478#3529669 (10Legoktm) [00:32:52] PROBLEM - Puppet errors on tools-exec-1428 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:50:52] 10PAWS: Debugging notebook cell action/state - https://phabricator.wikimedia.org/T173416#3529751 (10Jprorama) The [[ https://tools.wmflabs.org/openstack-browser/project/paws | PAWS instances ]] do have [[ https://grafana-labs.wikimedia.org/dashboard/db/labs-project-board?var-project=paws&var-server=All&orgId=1 |... [01:07:51] RECOVERY - Puppet errors on tools-exec-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [02:14:35] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:49:35] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [03:49:42] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:09:40] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [04:24:44] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:38:30] (03CR) 10Dzahn: [C: 031] "cool! would be nice! has it been tested?" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) (owner: 10Paladox) [04:55:38] (03CR) 10Dzahn: [C: 031] Add phabricator-upstream to #wikimedia-dev and #wikimedia-devtools [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333553 (owner: 10Paladox) [04:56:16] (03CR) 10Dzahn: [C: 031] "@20after4 still -1?" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333553 (owner: 10Paladox) [04:57:23] (03CR) 1020after4: "Nah I won't obstruct this ;)" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333553 (owner: 10Paladox) [04:58:01] (03CR) 10Dzahn: [C: 04-2] "please update commit message. what is this for? still current?" [labs/private] - 10https://gerrit.wikimedia.org/r/363847 (owner: 10Paladox) [04:58:24] (03CR) 10Dzahn: [C: 04-1] ""-labs" should not be used anymore. renamed to "-cloud" or something" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307439 (owner: 10Paladox) [04:58:44] (03CR) 10Dzahn: [C: 04-1] ""-labs" has been renamed to "-cloud"" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307441 (owner: 10Paladox) [04:59:43] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [06:30:37] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:30:53] PROBLEM - Puppet errors on tools-exec-1403 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:05:37] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [07:10:52] RECOVERY - Puppet errors on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [10:41:06] 10Tools, 10Patch-For-Review: Tool "wikipedia-fetch-content" loads jquery and bootstrap from code.jquery.com and bootstrapcdn - https://phabricator.wikimedia.org/T173067#3530345 (10D3r1ck01) >>! In T173067#3529615, @zhuyifei1999 wrote: > Thanks, LGTM. Thanks @zhuyifei1999. And also, thanks for reporting this.... [13:38:12] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/372371 (owner: 10L10n-bot) [14:40:23] zhuyifei1999_, o/ Around? [14:40:33] I was wondering if you wanted to talk about userdbs and quarry some more [14:40:39] Sorry I had to take off yesterday. [14:41:53] 10wikitech.wikimedia.org, 10Deployment-Systems, 10User-MarcoAurelio: Create a Gadget to easily add/remove/modify patches for SWAT at wikitech:Deployments - https://phabricator.wikimedia.org/T171940#3530728 (10MarcoAurelio) >>! In T171940#3508604, @mmodell wrote: > This is something I've been wanting to do fo... [14:42:51] halfak: yeah [14:43:28] Was wondering if you could tell me when the new DB host is supposed to switch in. [14:44:22] it's in beta right now, afaik [14:45:50] I have no idea about their future plans. jynus and bd808 should know more [14:46:30] for quarry it's just a matter of switching a configuration variable and restarting the workers [14:47:37] halfak: we are going to announce a beta period sometime next week. The new replicas are live now. [14:48:24] bd808, do you think we could find an short-term solution for big tables like the ORES ones? [14:48:25] The long process of rebuilding all the shards finished about a week and a half ago now [14:48:58] I was hoping we could have something like a "datasets_p" database with some process around getting a table in there. [14:49:32] Sounds like something that needs a ticket if we don't have one yet [14:49:44] cc me :) [14:50:59] We do have some new labvirts that are slated to replace older db hosts. They won't be online for a while though. We are working on the new dumps hosting first [14:51:19] There's definitely an old ticket. [14:51:21] * halfak digs [14:52:26] damn phab search [14:52:50] I cheat and use Google quite often [14:53:07] Bonus of being open data :) [14:53:25] bd808: how I never thought about that o.O [14:55:22] hmm... I'm starting to believe that there was not a phab task for this. [14:55:30] The new wiki replica servers are probably not going to allow tool/project dbs and tables. We can't find a way to reliably mirror them across the multiple backend servers [14:56:32] This all needs some discussion though that I'll be kicking off soon [14:57:05] bd808, +1 for not mirroring and for letting it be the wild west. [14:58:06] with some major rewrite possibly dependent on T76466 I could possibly implement a 'server selector' [14:58:07] T76466: Add database selector - https://phabricator.wikimedia.org/T76466 [14:58:14] The new servers are in HA pairs so the server you connect to could change very very quickly [14:58:55] This makes the 'Wild West' scenario more like a hope and fail plan [14:59:01] to switch between tools-db, replicas, and quarry internal database [14:59:19] Sorry. I meant not wild west [14:59:44] So we'd probably want a process and utility to aid in loading the same table into two servers. [15:00:01] And potentially adding rows to said table after a period of time. [15:00:15] *nod* [15:01:33] Damn. I know I had a writeup about this somewhere but I think it might have falled down the etherpad black hole. [15:01:50] The DBAs need to tell us what they are comfortable with. I'm just a helper in documenting and announcing policies for the most part [15:02:27] And making 'hard' decisions about the overall product offering [15:03:08] bd808, from a product perspective, I think there's a lot of value in getting pre-processed data in a nice place in quarry. [15:03:30] There's a huge amount of excitement over the article quality dataset that I found hard to get DBA approval to put in a userdb. [15:03:55] c.f. https://phabricator.wikimedia.org/T146718 [15:05:31] *fought [15:05:35] *advocated :) [15:06:32] OK Time to bite the bullet and just re-articulate my thoughts. [15:07:12] * halfak curses the vampire of time and memory [15:07:22] ..and then queues up the song. [15:07:47] Hosting curated datasets is something I'm interested in supporting. We just need to work out the how and where. [15:08:37] Proposal incoming. [15:09:05] I'll read it when I'm not 'on vacation' ;) [15:11:31] :) Makes sense. [15:11:41] enjoy your day! [15:11:48] (s) [15:12:03] * halfak needs to schedule a vacation :S [15:18:46] 10cloud-services-team, 10Analytics, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3530850 (10Halfak) [15:19:47] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:22:35] 10cloud-services-team, 10Analytics, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3530873 (10Halfak) [15:22:38] 10cloud-services-team, 10Analytics, 10Project-Admins, 10Research: Create a phabricator project called "wikireplica-datasets" - https://phabricator.wikimedia.org/T173512#3530877 (10Halfak) [15:22:55] 10cloud-services-team, 10Analytics, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3530850 (10Halfak) [15:25:28] 10cloud-services-team, 10Analytics, 10Research: Create a database on the wikireplica servers called "datasets_p" - https://phabricator.wikimedia.org/T173513#3530910 (10Halfak) [15:26:00] 10cloud-services-team, 10Analytics, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3530850 (10Halfak) [15:27:00] 10cloud-services-team, 10Analytics, 10Project-Admins, 10Research: Create a phabricator project called "wikireplica-datasets" - https://phabricator.wikimedia.org/T173512#3530933 (10Aklapper) [15:30:20] 10cloud-services-team, 10Analytics, 10Research: Document the process for importing a new "datasets_p" table - https://phabricator.wikimedia.org/T173514#3530940 (10Halfak) [15:31:01] 10cloud-services-team, 10Analytics, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3530850 (10Halfak) [15:31:17] Thar! [15:31:25] zhuyifei1999_, if you have some thoughts, check out https://phabricator.wikimedia.org/T173511 [15:31:39] harej, you'll probably be interested in ^ too [15:32:12] k thanks [15:32:34] 10cloud-services-team, 10Analytics, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3530963 (10Halfak) [15:33:01] harej: did you ask me a question somewhere? my history is being weird [15:33:56] chasemp: probably something to do with getting a VPS with 300 GB of storage set up [15:34:52] harej: ah, is this an existing task or a new one? usually anything larger than the stock images requires more forethought on our part [15:35:38] It is a task I haven't created yet. [15:36:05] chasemp, related to https://phabricator.wikimedia.org/T161554 [15:36:09] But I would like for my proposal to be well articulated, hence this pre-question [15:36:12] Separate project, same idea [15:39:05] I'm not sure what we can do capacity wise atm, a single 300G instance is probably achievable but storage is our most contentious resource, etc etc [15:39:08] https://phabricator.wikimedia.org/T161554#3204737 [15:39:17] I thikn that task sat for a few months til we had the capacity [15:39:36] unf one of the new servers is also doa it seems [15:40:09] anyway, make the task and we'll quantify capacity, a lot of things are coming down teh line in the next 6m or so, it may be tight [15:40:10] Hi! I get a long time replication lag on dewiki [15:41:36] (03CR) 10Jean-Frédéric: [C: 032] Removing invalid namespace and non-existent parameter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/371745 (owner: 10Lokal Profil) [15:42:07] doctaxon: steps to reproduce? [15:42:22] 10cloud-services-team (Kanban), 10Analytics, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3531049 (10chasemp) [15:42:24] (03CR) 10Jean-Frédéric: [C: 032] Handle sparql http timeout issue [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/371602 (https://phabricator.wikimedia.org/T172690) (owner: 10Lokal Profil) [15:42:47] try to read page content on dewiki using API [15:43:05] see -ops it seems to be a known thing [15:43:19] and the replicas are not the only thing seein git [15:43:22] seeing it [15:43:56] (03Merged) 10jenkins-bot: Removing invalid namespace and non-existent parameter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/371745 (owner: 10Lokal Profil) [15:44:19] (03Merged) 10jenkins-bot: Handle sparql http timeout issue [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/371602 (https://phabricator.wikimedia.org/T172690) (owner: 10Lokal Profil) [15:44:38] 10cloud-services-team (Kanban), 10Analytics, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3531058 (10chasemp) p:05Triage>03Normal [15:45:51] halfak: https://phabricator.wikimedia.org/T173511 seems like the right thing, I'm not sure what teh process should be etc but thank you for making those tasks [15:46:06] do you want to schedule 30m to talk about it next week? [15:46:29] doctaxon: see -ops now for the culprit and I think they are mitigating [15:46:30] chasemp, no worried if we need to decline some or all of that. It is good to have a proposal top pick on :) [15:46:33] sure! [15:46:40] Oh... wait...next week is bad. [15:46:41] heh [15:46:45] I have a conference. Maybe early on Monday? [15:46:46] this week is bad for me [15:46:50] sure [15:47:12] everyone not me is on vacation from cloud services so I'm a bit frantic this week [15:47:23] Understood. :) [15:47:37] chasemp, I forget your timezone. :\ [15:47:46] central us :) [15:48:30] Cool. That'll work perfect [15:49:06] halfak: my intention is to share a bit of organization progress around managing the replicas and to sync up so we are pitching the same idea [15:49:15] then I'll talk it over in our weekly with the cloud team [15:49:21] and we can see what we can get done [15:49:33] for agenda [15:50:00] sounds great! [15:50:00] * halfak copy-pastes into calendar event [15:52:43] (03CR) 10jenkins-bot: Removing invalid namespace and non-existent parameter [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/371745 (owner: 10Lokal Profil) [15:53:22] (03CR) 10jenkins-bot: Handle sparql http timeout issue [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/371602 (https://phabricator.wikimedia.org/T172690) (owner: 10Lokal Profil) [15:55:39] 10cloud-services-team, 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3466006 (10chasemp) @Cmjohnson seems like a definite hardware failure to me man, we haven't even put this back in service. Next steps? [15:56:54] 10cloud-services-team (Kanban), 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3531087 (10chasemp) [15:57:21] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10Patch-For-Review: Add `wikitech-grep` to puppet - https://phabricator.wikimedia.org/T169820#3531089 (10chasemp) 05Open>03Resolved a:03chasemp [16:03:26] 10cloud-services-team (Kanban), 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3531116 (10Cmjohnson) @chasemp can you share some logs, I need to take this back to Dell [16:10:49] 10cloud-services-team (Kanban), 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3531150 (10chasemp) p:05Triage>03High [16:12:45] legoktm: I had made a note to reiew something I saw come across for you yesterday, can't find it. let me know what I"m missing :) [17:10:41] chasemp: probably https://gerrit.wikimedia.org/r/#/c/372213/ :) [17:11:12] I don't actually know puppet, I'm just copy-pasting something yuvi wrote for me 2 years ago [17:20:28] (03PS5) 10Legoktm: Add project phab-* [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333531 (owner: 10Paladox) [17:20:33] (03CR) 10Legoktm: [C: 032] Add project phab-* [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333531 (owner: 10Paladox) [17:21:06] (03Merged) 10jenkins-bot: Add project phab-* [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333531 (owner: 10Paladox) [17:21:15] (03CR) 10jenkins-bot: Add project phab-* [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333531 (owner: 10Paladox) [17:21:57] !log tools.wikibugs Updated channels.yaml to: 3993836f4c386168e5d3f1978049371179ed593c Add project phab-* [17:24:36] 10PAWS: Debugging notebook cell action/state - https://phabricator.wikimedia.org/T173416#3531316 (10Jprorama) The [[ https://tools.wmflabs.org/openstack-browser/project/tools | tools project ]] contains a list of all the instances. This includes the k8s nodes, grid workers, and PAWS fabric. This doesn't appear... [17:29:49] PROBLEM - Puppet errors on tools-static-11 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [17:53:27] 10Cloud-Services: Toolforge intermittent Puppet failures for puppet-enc - https://phabricator.wikimedia.org/T173526#3531473 (10chasemp) [17:53:35] 10Cloud-Services: Toolforge intermittent Puppet failures for puppet-enc - https://phabricator.wikimedia.org/T173526#3531485 (10chasemp) p:05Triage>03Normal [17:54:01] 10Cloud-Services: Toolforge intermittent Puppet failures for puppet-enc - https://phabricator.wikimedia.org/T173526#3531473 (10chasemp) @Andrew I'm wondering if could be related to new puppet masters in any way? Seems roughly I didn't notice this until all that shook out. [17:59:45] RECOVERY - Puppet errors on tools-static-11 is OK: OK: Less than 1.00% above the threshold [0.0] [18:17:44] (03CR) 10Paladox: "Thanks." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/333531 (owner: 10Paladox) [18:26:36] legoktm: left a few notes [18:27:34] thanks :D [18:27:34] 10Horizon, 10Marvin: Add @Niedzielski to the reading-web-staging group in horizon - https://phabricator.wikimedia.org/T172911#3531624 (10Niedzielski) Ok, I'm logged into Horizon now. My 2FA was enabled for Office but not Wikitech (which both @Krenair and @Jhernandez mentioned...). I was able to ssh into all th... [18:30:25] 10Horizon, 10Marvin: Add @Niedzielski to the reading-web-staging group in horizon - https://phabricator.wikimedia.org/T172911#3512921 (10chasemp) floating IP is an unrelated thing to instance creation. Creation: `Compute => Instances => Launch Instance` [18:42:37] 10Horizon, 10Marvin: Add @Niedzielski to the reading-web-staging group in horizon - https://phabricator.wikimedia.org/T172911#3531705 (10Niedzielski) @chasemp, thanks! Seems to be missing there too. {F9100184} [18:43:24] 10Horizon, 10Marvin: Add @Niedzielski to the reading-web-staging group in horizon - https://phabricator.wikimedia.org/T172911#3531715 (10chasemp) Are you a project member or a project admin? I think only admin users can create instances. [18:46:11] 10Horizon, 10Marvin: Add @Niedzielski to the reading-web-staging group in horizon - https://phabricator.wikimedia.org/T172911#3531723 (10Niedzielski) @chasemp, ah, just a standard user :] [20:46:15] 10Cloud-Services, 10cloud-services-team (Kanban), 10Privacy, 10Security: "last" command on WMF Labs/Tools allows users to view IPs of other toolforge users - https://phabricator.wikimedia.org/T172650#3532130 (10Bawolff) [20:46:41] 10Cloud-Services, 10cloud-services-team (Kanban), 10Privacy, 10Security: "last" command on WMF Labs/Tools allows users to view IPs of other toolforge users - https://phabricator.wikimedia.org/T172650#3504696 (10Bawolff) >>! In T172650#3531682, @chasemp wrote: > I think we should remove the restrictive secu... [20:52:50] 10Cloud-Services, 10cloud-services-team (Kanban), 10Privacy, 10Security: "last" command on WMF Labs/Tools allows users to view IPs of other toolforge users - https://phabricator.wikimedia.org/T172650#3504696 (10Krenair) `who` is another such tool. But I agree we should just make it clearer that such sharin... [21:05:21] 10Cloud-Services, 10cloud-services-team (Kanban), 10Privacy, 10Security: "last" command on WMF Labs/Tools allows users to view IPs of other toolforge users - https://phabricator.wikimedia.org/T172650#3504696 (10Luke081515) It's not restricted to toolforge users, since you can do that on bastions as well. [21:15:28] 10Cloud-Services, 10cloud-services-team (Kanban), 10Privacy, 10Security: "last" command on WMF Labs/Tools allows users to view IPs of other toolforge users - https://phabricator.wikimedia.org/T172650#3532217 (10Krenair) Maybe we could consider some extra hop box that only allows normal users to run SSH, an... [21:19:49] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [22:17:51] (03CR) 10Paladox: "> cool! would be nice! has it been tested?" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) (owner: 10Paladox) [22:20:46] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [23:01:24] 10Cloud-Services, 10cloud-services-team (Kanban), 10Privacy, 10Security: "last" command on WMF Labs/Tools allows users to view IPs of other toolforge users - https://phabricator.wikimedia.org/T172650#3532469 (10Luke081515) That would work on bastions I guess, but for toolforge bastions there are some comma... [23:07:32] !help Why can I not add my public SSH key to Toolforge Labs when I can add it to Wikitech? [23:07:33] PokestarFan: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [23:15:44] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [23:47:54] PokestarFan: how do you mean that? [23:48:21] Sagan: I tried adding my ssh key to toolforge but it failed [23:48:28] Sagan: But it worked on Wikitech. [23:48:32] Sagan: I don't get it [23:48:51] PokestarFan: were at toolforge? [23:48:54] *where [23:49:02] Sagan: the homepage [23:49:11] PokestarFan: can you give me the link? [23:49:18] Sagan: I meant https://toolsadmin.wikimedia.org/profile/settings/ssh-keys [23:49:50] * Sagan searches his 2 FA codes [23:50:13] Sagan: what is a a code? [23:50:20] ^fa [23:50:25] **fa [23:50:33] PokestarFan: on wikitech, I have multi factor authentification active [23:50:36] so I need the code [23:50:49] https://en.wikipedia.org/wiki/2FA [23:51:12] PokestarFan: is there an error when it fails, a error msg? [23:53:57] Sagan: Just noticed the key is already there [23:54:14] PokestarFan: yeah, that's why I'm asking. It's LDAP, so keys get synched [23:54:20] Sagan: so I am being given a generic error. It needs to say duplicate key [23:54:22] so if you get an error, then because they key is already there [23:54:34] PokestarFan: I think that's already a known issue, already reported [23:54:40] ok [23:54:48] Sagan: What is the ticket? [23:54:51] but the keys itself should work :) [23:55:04] Sagan: Can you grant toolforge access? [23:55:22] PokestarFan: nope. there is form for that at toolsadmin AFAIK [23:55:34] I know [23:55:38] I usbmitted it [23:55:41] *submitted [23:55:55] T167931 [23:55:55] T167931: Fatal error when adding a duplicate SSH key - https://phabricator.wikimedia.org/T167931