[00:42:41] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:43:46] !log suggestbot Converted all database tables to binary character set [00:43:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Suggestbot/SAL [01:06:04] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:12:41] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [01:36:03] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [01:51:48] bd808: I have some Outreachy microtasks which involve creating Toolforge tools doing something simple [01:51:54] so, lot's of one-off tools [01:52:07] just want to make sure that's not disruptive [03:01:17] That reminds me, I should probably create a bunch of tasks for Reports Bot [03:30:25] PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:25:22] RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [04:31:47] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3613653 (10bd808) I have a script that manages the domains that I'll upload as an operations/puppet.git patch soon. I used it to fully pop... [04:39:47] tgr: it should be ok. tool accounts are 'cheap'. The worst thing that happens is namespace consumption. If they are mostly intended to be short lived projects you might suggest that the have some common naming scheme like outreachy-15- or something. [04:40:13] but if they do something useful longer term that's cool too :) [04:42:49] I think a few exist already, but I'll ask for that going on [04:43:38] someday™ it will be easy to delete a tool when it is no longer useful. [04:44:43] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:02:05] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:14:43] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [05:37:07] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [05:58:04] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:04:33] bd808: I do look forward to that day. I alone have several legacy tool accounts and I worry they contribute to some kind of debt [06:05:12] Like, they are things that need to be kept track of even though they contribute nothing (the very definition of an underwater asset) [07:03:03] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [07:35:48] 10Cloud-Services, 10Toolforge: Update python-wikitools on Tool Labs - https://phabricator.wikimedia.org/T102862#1375558 (10smernst) I started a new task related to this topic: T176109. [07:36:43] 10Toolforge, 10Wiki-Loves-Monuments-Database: Toolforge tool.heritage webservice keeps crashing - https://phabricator.wikimedia.org/T176110#3613797 (10JeanFred) [07:40:48] 10Cloud-Services: salt keys being created and accepted with wrong hostname (no project name in hostname) - https://phabricator.wikimedia.org/T126324#3613815 (10MoritzMuehlenhoff) 05Open>03declined Salt is being removed. [08:15:38] 10Toolforge: Cleaning up and refactoring python-wikitools (for use with Python 3) - https://phabricator.wikimedia.org/T176109#3614028 (10Aklapper) This looks like (support) questions, not like a valid bug report or enhancement request? #1 to #3 should probably be asked on the mailing list at https://lists.wikime... [08:28:55] 10Toolforge: Cleaning up and refactoring python-wikitools (for use with Python 3) - https://phabricator.wikimedia.org/T176109#3614049 (10smernst) @Aklapper You're right. I am tracking down the stakeholders of the python-wikitools project and I though here is a good place to start ... it's one of the best (and th... [08:34:05] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:47:59] 10PAWS, 10cloud-services-team (Kanban), 10User-bd808: Not able to edit user-config.py file in PAWS - https://phabricator.wikimedia.org/T175167#3614098 (10Amishas157) Thanks a lot for taking care of this @bd808 @zhuyifei1999 🙇 [09:04:05] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [09:52:34] 10Cloud-Services, 10DBA: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3611967 (10jcrespo) Is this going to be a public or a private wiki? [09:54:00] 10Cloud-Services, 10DBA: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3614286 (10Ladsgroup) fishbowl wiki. Public but not editable (unless with account) like wmfwiki [09:55:55] 10Cloud-Services, 10DBA: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3614287 (10jcrespo) Ok, then this is not a blocker for the above ticket. We would thank a ping when the actual database is deployed. [12:12:17] !log wikilabels set eswiki labeling campaign as completed (T131963) [12:12:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [12:12:21] T131963: Complete eswiki edit quality campaign - https://phabricator.wikimedia.org/T131963 [12:32:09] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 12.50% of data above the critical threshold [0.0] [13:07:07] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [13:35:03] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:58:24] !log tools.heritage Killed `update_monuments` job as it had been stuck for 3 days on the `Update monuments_all table` step. [13:58:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [14:00:21] 10Toolforge, 10Wiki-Loves-Monuments-Database: Toolforge tool.heritage webservice keeps crashing - https://phabricator.wikimedia.org/T176110#3614758 (10chasemp) p:05Triage>03Normal [14:01:10] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3614764 (10chasemp) That's seems really great [14:08:28] 10cloud-services-team (Kanban), 10Patch-For-Review: Replace kernel and reboot labvirt1015, 1016, 1017, 1018 - https://phabricator.wikimedia.org/T176044#3611990 (10chasemp) ```Linux labvirt1001 4.4.0-81-generic #104~14.04.1-Ubuntu SMP Wed Jun 14 12:45:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Linux labvirt1002... [14:10:05] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [14:27:26] 10Toolforge: Namespaces not found (pykube, kubernetes) - https://phabricator.wikimedia.org/T176024#3614820 (10chasemp) p:05Triage>03Normal IIUC from reading on IRC this is due to k8s not understanding namespaces with an underscore. What ended up happening here? [14:31:04] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:42:17] 10cloud-services-team (Kanban), 10Patch-For-Review: Replace kernel and reboot labvirt1015, 1016, 1017, 1018 - https://phabricator.wikimedia.org/T176044#3614867 (10chasemp) a:03Andrew [15:06:04] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:02] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:00:06] 10Cloud-Services, 10monitoring, 10Labs-Sprint-109, 10Patch-For-Review, and 3 others: Monitor nova services - https://phabricator.wikimedia.org/T90784#1067402 (10Dzahn) Hi @Andrew is this a duplicate of T42022? It seems you already added some monitoring in the past, but this is still open. Are your comments... [16:09:29] (03PS2) 10Lokal Profil: [WIP] Ensure skipped image categorizations are mentioned in stats [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378641 (https://phabricator.wikimedia.org/T174871) [16:15:04] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:18:20] (03CR) 10Jean-Frédéric: "Wow, this looks good! Some much needed refactoring here :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378641 (https://phabricator.wikimedia.org/T174871) (owner: 10Lokal Profil) [16:45:08] 10cloud-services-team (Kanban), 10Patch-For-Review: Replace kernel and reboot labvirt1015, 1016, 1017, 1018 - https://phabricator.wikimedia.org/T176044#3611990 (10MoritzMuehlenhoff) Any errors shown for the non-working ones? [16:59:14] 10Tool-Article-request, 10Toolforge, 10User-Matthewrbowker: Uptimerobot monitoring for the Articlerequest tool flaps - https://phabricator.wikimedia.org/T175623#3615369 (10Matthewrbowker) @yuvipanda I have subscribed to Pingdom, and it reports the exact same problem much more frequently. The error text is "... [17:02:45] 10Tool-Article-request, 10Toolforge, 10User-Matthewrbowker: Uptimerobot monitoring for the Articlerequest tool flaps - https://phabricator.wikimedia.org/T175623#3615386 (10Matthewrbowker) Traceroute from pingdom: ``` traceroute to 208.80.155.131 (208.80.155.131), 30 hops max, 60 byte packets 1 184.75.214.65... [17:02:51] (03PS1) 10Jean-Frédéric: Use isort to sort imports [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378741 [17:04:03] (03CR) 10Jean-Frédéric: "I also tried adding a tox target to enforce this but it does not work :-(" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378741 (owner: 10Jean-Frédéric) [17:11:58] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [17:25:02] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:15] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Promote initial use of new Wiki Replica servers - https://phabricator.wikimedia.org/T172704#3615528 (10bd808) [17:55:04] 10VPS-Projects, 10Social-Tools: social-tools.wmflabs.org is running MediaWiki v1.28.0 - https://phabricator.wikimedia.org/T174958#3615639 (10SamanthaNguyen) a:03Legoktm [17:58:52] 10Toolforge, 10Kubernetes, 10Tracking: Packages to be installed in Toolforge Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110#3615649 (10bd808) [17:58:55] 10Toolforge, 10cloud-services-team (Kanban), 10Kubernetes, 10Patch-For-Review, 10User-bd808: Add composer to PHP base images - https://phabricator.wikimedia.org/T172358#3615647 (10bd808) 05Open>03Resolved ``` $ webservice --backend=kubernetes php5.6 shell If you don't see a command prompt, try pressi... [18:01:44] I just deployed a new PHP5.6 container image for Toolforge Kubernetes webservices. I don't expect any problems, but if anyone sees issues with their tools please file Phabricator tasks and poke me. [18:02:21] !log tools Updated PHP5.6 images for Kubernetes (T172358) [18:02:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:02:27] T172358: Add composer to PHP base images - https://phabricator.wikimedia.org/T172358 [18:16:01] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:19:14] (03PS1) 10Jean-Frédéric: Bump PHP dependencies in composer.json [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378758 [18:20:59] (03CR) 10Jean-Frédéric: "codesniffer could also use upgrading but I’ll that to a separate commit :)" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378758 (owner: 10Jean-Frédéric) [18:44:16] Question! I have a bot running on the wmf tools grid. It does stuff in multiple threads. How can I tell how many threads I should be using? [18:46:45] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that load resources from third party sites - https://phabricator.wikimedia.org/T172065#3615789 (10Epantaleo) [18:46:47] 10Tools: Tool "etytree" loads jquery, jquery-mobile, and jquery-ui from code.jquery.com, mixed http/https - https://phabricator.wikimedia.org/T172519#3615788 (10Epantaleo) 05Open>03Resolved [18:51:02] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [18:55:32] Kotz: is your question how many threads you are allowed to run or something else? [19:06:01] 10Tools: Tool "etytree" loads jquery, jquery-mobile, and jquery-ui from code.jquery.com, mixed http/https - https://phabricator.wikimedia.org/T172519#3615883 (10zhuyifei1999) 05Resolved>03Open ``` $ curl https://tools.wmflabs.org/etytree/ -s | grep jquery To play nice in Grid Engine it would honestly be better to figure out how to run several copies of the bot processing different sets of data than running one bot with many threads [19:18:57] If you are running under Kubernetes there is CPU accounting so adding more threads to the procs you are allotted will eventually hit a point where you are context switching more than you are processing [19:19:04] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that load resources from third party sites - https://phabricator.wikimedia.org/T172065#3615987 (10Epantaleo) [19:19:06] 10Tools: Tool "etytree" loads jquery, jquery-mobile, and jquery-ui from code.jquery.com, mixed http/https - https://phabricator.wikimedia.org/T172519#3615985 (10Epantaleo) 05Open>03Resolved Oh ok, missed that file! It should be ok now. [19:20:10] bd808 - is there a tool like "top" for the grid? [19:20:37] Hi! can a mentor or a volunteer help. To see how scripts/webservice works on executing the run command. I was trying out building a basic application on toolforge. But, while creating the tool it asks for default software license as well. [19:20:52] Kotz: http://tools.wmflabs.org/admin/oge/status [19:21:00] what should I mention in place of default software license? [19:21:07] Hi hi cloud peeps! :D [19:21:28] Is there an easy non authed way to get a list of instances in a project in a machine readable way? [19:22:24] Mridu: the default license applies to code that you put in the tool's home directory without an explict license. This is related to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Right_to_fork_policy [19:22:36] addshore: there is ... let me find the URL for you... [19:23:56] addshore: http://tools.wmflabs.org/openstack-browser/api/dsh/project/deployment-prep [19:24:03] thanks!!!!! [19:24:07] bd808: where is this default license question? striker? [19:24:07] change the name at the end to the project you need [19:24:23] zhuyifei1999_: yes. it is part of creating a tool now [19:24:28] bd808: trying to implement this https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool. [19:24:37] * zhuyifei1999_ is lagging behind [19:25:56] Mridu: *nod* we should update that page to make this answer clear. :) The examples on that page are licensed under the GPLv3+ license. So GPLv3 would be a reasonable choice for a tool that is derived from that. [19:27:19] But you could choose any other default license on the list and still be ok too. If you copy those examples directly including the license headers then the files would just be available under a different license than the default of the tool. [19:28:59] bd808: ok.. yes, can I update it? On how and which license to select. We can add the link as well. The one you shared with me. [19:29:15] Mridu: that would be great. :) [19:31:59] bd808: IMO. Point 1 of step 1 where it is mentioned to create a new tool. We can add a screenshot of create new tool page. [19:32:55] Make default licence choice as the one who have mentioned. And provide the link to how license needs to be chosen. [19:33:13] sorry you* not who. [19:34:31] Mridu: that sounds nice. Please do make the docs better. :) [19:36:06] Sure. Once I am done with running the command and making the tool. Will make the changes in docs as well :) [19:36:25] perfect [19:42:31] it asks for tags as well. Can you share an example of tag? [19:43:09] Mridu: in the screenshot, if you are using windows (idk about mac), try to crop out the window borders [19:44:18] and PNG is preferred over JPG for screenshots [19:44:21] Mridu: https://toolsadmin.wikimedia.org/tools/id/my-first-flask-oauth-tool [19:44:32] You should also be allowed to leave the tags blank [19:45:13] No, I am not using windows anymore. Have switched to ubuntu completely. Will be working on screenshot after I create the tool. This tags is a field asked on create new tool page. https://toolsadmin.wikimedia.org/tools/create/ [19:45:20] okay. [19:45:28] k [20:03:39] 10PAWS, 10Patch-For-Review: Enable downloading notebooks as PDF - https://phabricator.wikimedia.org/T159617#3616078 (10madhuvishy) @HaeB I've enabled pdf exports for SWAP now, but it may error out if it runs into unicode characters it can't parse, I'm not sure why, and it seems to be a longer task, so tabling... [20:05:30] After creating tool. Next step is to do a ssh to tools-login.wmflabs.org. Directly do I have to use this. Or have to replace tools login with my username on toolforge. [20:05:54] Mridu: use "tools-login.wmflabs.org" [20:06:20] that is a "bastion" server that all the Toolforge users have access to [20:06:45] from there you will be able to "become" your tool and then do the other tasks [20:07:07] It's coming permission denied.(publickey, hostbased) [20:08:34] Mridu: the server is logging that your ssh public key for the login attempt does not match the key that you setup in the LDAP directory [20:09:02] "Failed publickey for mridu from x.x.x.x port 55790 ssh2: RSA SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY" [20:09:38] You can check the fingerprints of the keys you have setup at https://toolsadmin.wikimedia.org/profile/settings/ssh-keys [20:09:49] ECDSA key fingerprint is SHA256:TybNtIoEmUacZxKi83BRYP3Q+TMeK5llxuMI6duBKEQ. [20:10:21] so you either need to use that key or add the public key that you are trying to use now [20:11:07] its is ok to have multiple keys. that's actually common if you are switching computers/operating systems as you work on various things [20:11:59] unable to understand why there is difference in key. I did all my installations and stuff on ubuntu only. [20:13:00] Mridu: try w/ 'ssh -i /path/to/key login.tools.wmflabs.org [20:13:03] ' [20:13:43] In LDAP has same key as wikitech and gerrit accounts. From where I have to ssh? I mean the folder. Is it from folder tools-webservice? Or my home directory [20:14:22] Mridu: it's from LDAP for Tools and I believe for gerrit you upload a key to the UI [20:14:29] wikitech also stores teh key used in LDAP for ssh to use [20:14:34] yes. did the same [20:14:58] Mridu: is your shell username mridu? [20:15:11] you are not a member of the Tools project if so [20:15:14] root@tools-bastion-03:~# id mridu [20:15:15] uid=17929(mridu) gid=500(wikidev) groups=500(wikidev) [20:15:39] Mridu: go to https://toolsadmin.wikimedia.org/auth/login/ and request to become a member of the project [20:15:48] Tools was not showing the key. So, I manually added the key. And the key I added is the same one which I have on gerrit UI. [20:16:05] gerrit UI is totally unrelated to this let's set that aside [20:16:15] no. shell username is mridub. [20:16:18] ah [20:16:18] ok [20:17:09] Mridu: this is the key in LDAP [20:17:09] ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCF8pwFLehzCXhbF1jfHWtd9d1LFq2NirplEBQYs7AOrGwQ/6ZZI0gvZFYiEiaw1o+F1CMfoHdny1VfWOJF3mJ1y9QMKAacc8/Z3tG39jBKRQCuxmYLO1SWymv7/Uvx9WQlkNRoTdTTa9OJFy6UqvLQEXKYaokfMIUHZ+oVFf1CgQ== [20:17:18] yes [20:17:25] are you using the matching private key to login? [20:17:47] can you try to login again? [20:17:53] to login.tools.wmflabs.org [20:18:00] yes will do [20:18:08] through ssh only? [20:18:13] yes [20:19:03] right, so it does say the incorrect key is being presented [20:19:04] error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup mridu failed, status 1 [20:19:04] Sep 18 20:18:37 tools-bastion-03 sshd[21180]: Failed publickey for mridu from 171.61.36.155 port 55810 ssh2: RSA SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY [20:19:09] permission denied [20:19:30] Mridu: either you are not presenting the private key you believe, or your private key is somehow corrupt [20:19:38] I would try to generate a new keypair, update wikitech, and try again [20:19:48] trying w/ [20:19:57] where can I check the private key? [20:20:02] ssh -i /path/to/key login.tools.wmflabs.org [20:20:07] is the best way to check for sure [20:20:14] but [20:20:16] don't know the path to key [20:20:31] Mridu: ls ~/.ssh [20:20:56] ~/.ssh/config should determine what is being used [20:21:18] id_rsa id_rsa.pub known_hosts output on doing ls ~/.ssh. [20:21:21] I would run ssh -vvv login.tools.wmflabs.org [20:21:40] or even better, explicitely set the username [20:21:44] I would run ssh -vvv mridub@login.tools.wmflabs.org [20:21:52] username is good [20:21:58] I agree [20:22:14] esp since /another/ user exists in our system w/ mridu [20:22:24] ha [20:22:25] yeah [20:22:33] error: AuthorizedKeysCommand /usr/sbin/ssh-key-ldap-lookup mridu failed, status 1 [20:22:38] permission denied [20:22:58] Mridu: the username is wrong [20:23:01] see that error [20:23:11] you said your shell name is mridub but you are sending mridu [20:23:17] Platonides nailed it in one [20:23:26] :) [20:23:59] it's quite common that people use different usernames on several machines [20:24:08] I tend to set User in .ssh/config [20:24:10] you are in fact failing to login with the correct key, but it's for someone elses account :) [20:24:18] I bet [20:24:21] oh..no. yes name on my machine is mridu :/ [20:24:52] ssh mridub@login.tools.wmflabs.org will do it, then [20:25:44] no it doesn't. [20:26:11] should I change my username on machine [20:26:15] Mridu: did you try again? [20:26:20] try one more time for me [20:26:25] w/ Platonides's command [20:26:34] Mridu: there's no need for that [20:26:37] yes I did ssh mridub@login.tools.wmflabs.org [20:26:50] that sets explicitely the username for ssh [20:26:57] Sep 18 20:25:23 tools-bastion-03 sshd[25859]: Failed publickey for mridub from 171.61.36.155 port 55814 ssh2: RSA SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY [20:27:01] so it doesn't fall back to your local username [20:27:11] so, wrong username and wrong key? :) [20:28:12] so I see two failed attempts w/ assume correct username [20:28:14] Sep 18 20:22:26 tools-bastion-03 sshd[23751]: Failed publickey for mridub from 171.61.36.155 port 55812 ssh2: RSA SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY [20:28:14] Sep 18 20:25:23 tools-bastion-03 sshd[25859]: Failed publickey for mridub from 171.61.36.155 port 55814 ssh2: RSA SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY [20:28:32] ssh-key-ldap-lookup mridub [20:28:32] ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCF8pwFLehzCXhbF1jfHWtd9d1LFq2NirplEBQYs7AOrGwQ/6ZZI0gvZFYiEiaw1o+F1CMfoHdny1VfWOJF3mJ1y9QMKAacc8/Z3tG39jBKRQCuxmYLO1SWymv7/Uvx9WQlkNRoTdTTa9OJFy6UqvLQEXKYaokfMIUHZ+oVFf1CgQ== [20:28:39] that's SHA256:j7HQoQ6fIuEgDHjONjI2CZ+2Iwxqgo2Ur5LbPqBgxOU [20:28:52] I added the gerrit key in toolforge yesterday. As by default I was not seeing anykey in toolforge. [20:30:08] Mridu: please run ssh-keygen -l -f ~/.ssh/id_rsa.pub [20:30:13] and also ssh-keygen -l -f ~/.ssh/id_rsa [20:31:39] output on running ssh-keygen -l -f ~/.ssh/id_rsa.pub: 2048 SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY [20:32:45] output on running ssh-keygen -l -f ~/.ssh/id_rsa: 2048 SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY [20:32:56] they match then (as they should) [20:33:02] You should add the contents of ~/.ssh/id_rsa.pub on wikitech [20:33:06] AFAICT you are sending the wrong key without a doubt try going to https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack and replacing whatever is there w/ the contents of your id_rsa.pub [20:33:27] then: ssh mridub@login.tools.wmflabs.org [20:35:50] I added 2048 SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY as public key and removed the earlier one. It shows The provided SSH key was wrong or in unknown format. [20:36:06] I believe that [20:38:01] *And I am confused* [20:39:22] Mridu: no worries, what's the current status? [20:40:22] public key is not getting added on to wikitech. Through toolforge UI I have created new tool. Doing ssh was the next step. [20:40:36] Mridu: you need to add the contents of ~/.ssh/id_rsa.pub [20:40:48] "2048 SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY" is just a summary of that file [20:41:17] open it with whatever tool suits you, eg. cat ~/.ssh/id_rsa.pub [20:41:20] oh I misunderstood the statement actually, I thought that was the old contents [20:41:21] yes [20:41:24] it will probably begin with ssh-rsa … [20:41:33] literal contents of ~/.ssh/id_rsa.pub [20:41:55] mridu@mridu-HP-15-Notebook-PC:~$ ssh-keygen -l -f ~/.ssh/id_rsa.pub 2048 SHA256:PuxQqFHOY5CW6q6L7Z03X7fCEKuJ7Tt3uhlIcOCOVYY mridu@mridu-HP-15-Notebook-PC (RSA) [20:42:12] Mridu: show us the contents of 'cat ~/.ssh/id_rsa.pub' [20:42:17] this is the output [20:42:29] nope [20:42:44] ssh-keygen generated the *fingerprint* (a summary) [20:42:51] cat ~/.ssh/id_rsa.pub [20:43:00] ssh-rsa [20:43:03] ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyVqptm/6QhY6bvcwn0kUOtYgurZDTFVxJVhAPuCoXnsfzwlZGRiwp0s3xK0x7QBTSh0U0dDApIM/J1e6Flq0DPqxcdbzRtsMyd5AX+vv0uf6EywOPFFi92DpU3wfo1HIQwDrfAQpDBMdqyTMhagMCBcD+tTS7z80GrBoOnsqVRTDqMpBCYdXeZfUSFwHGXeSgDCTI7aDrL4iirgbqvrtZnHExQsDInCTz17B2zLMljJCYQYRZnCouVYESvYe2KX+/aME8seoVKmWzSCw+vVysvMYk4uzVoEJaAjYIqOjcZvviOYrR8idHTzIJ7V9bYiO76yQXqymvhiFrTRAzFlUZ [20:43:10] that seems valid [20:43:23] this was what we had deleted [20:43:26] Mridu: add /that/ to the wikitech field at https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack [20:43:33] yep [20:43:35] copy that [20:43:41] Mridu: no [20:43:49] you had: [20:43:50] ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCF8pwFLehzCXhbF1jfHWtd9d1LFq2NirplEBQYs7AOrGwQ/6ZZI0gvZFYiEiaw1o+F1CMfoHdny1VfWOJF3mJ1y9QMKAacc8/Z3tG39jBKRQCuxmYLO1SWymv7/Uvx9WQlkNRoTdTTa9OJFy6UqvLQEXKYaokfMIUHZ+oVFf1CgQ== [20:43:56] this is [20:43:57] ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyVqptm/6QhY6bvcwn0kUOtYgurZDTFVxJVhAPuCoXnsfzwlZGRiwp0s3xK0x7QBTSh0U0dDApIM/J1e6Flq0DPqxcdbzRtsMyd5AX+vv0uf6EywOPFFi92DpU3wfo1HIQwDrfAQpDBMdqyTMhagMCBcD+tTS7z80GrBoOnsqVRTDqMpBCYdXeZfUSFwHGXeSgDCTI7aDrL4iirgbqvrtZnHExQsDInCTz17B2zLMljJCYQYRZnCouVYESvYe2KX+/aME8seoVKmWzSCw+vVysvMYk4uzVoEJaAjYIqOjcZvviOYrR8idHTzIJ7V9bYiO76yQXqymvhiFrTRAzFlUZ [20:44:07] this was the one present earlier. And not working. I will recheck [20:44:15] most rsa keys begin the same way [20:44:20] but are actually different [20:44:21] yes different [20:44:28] adding [20:45:11] done. [20:45:42] now again the ssh command right? [20:45:55] for login [20:45:57] yes [20:45:57] please [20:46:33] I wonder why ssh-key-ldap-lookup mridub | ssh-keygen -l -f /dev/stdin fails, though :/ [20:46:39] it says /dev/stdin is not a public key file. [20:47:06] ssh tools-login.wmflabs.org. permission denied [20:47:49] Mridu: ssh mridub@login.tools.wmflabs.org [20:49:02] works it seems. [20:49:32] it's showing toolforge [20:50:54] yep [20:50:58] you are logged in :) [20:51:00] finally [20:51:02] Oh yes! it showing the tool I created under the Your tools field [20:51:34] yes. :) Thanks. What are these public keys for? [20:52:04] 10Data-Services, 10cloud-services-team (FY2017-18), 10DBA, 10Goal: Decommission labsdb1001 and labsdb1003 - https://phabricator.wikimedia.org/T142807#3616218 (10bd808) [20:52:07] 10Data-Services, 10cloud-services-team (Kanban), 10Analytics, 10DBA, 10Research: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#3616219 (10bd808) [20:52:15] you authenticate with your private key [20:52:22] the contents of ~/.ssh/id_rsa [20:52:34] the public key (~/.ssh/id_rsa.pub) is the matching one [20:52:49] that allows verifying that you indeed know the private one [20:53:03] this way, you can authenticate yourself [20:53:19] without having to conceal the contents of id_rsa.pub [20:53:29] you could use the same public key on dozens of servers [20:53:34] and unlike a password [20:53:49] that wouldn't allow a compromise on one of them [20:53:56] to access your account in the other ones [20:54:29] (assuming the key is big enough, properly generated, etc.) [20:55:16] you can read more about this on https://en.wikipedia.org/wiki/Public-key_cryptography [20:56:07] yes will go through it. :) [20:56:44] feel free to ask any doubt [21:00:46] sure. [21:04:05] Run become to change to the tool user is the next step. The command works. But, what is the need behind changing the tool user? [21:05:05] Mridu: tool user allows shared maintenance [21:05:31] so both you can someone else (or even more people) can maintain the tool [21:05:51] and in case of you giving up the tool comeone can take over [21:05:54] *someone [21:06:07] also it allows you to maintain multiple tools [21:06:07] oh.. okay. [21:06:36] so if you don't have a tool account you have to do it under your own personal account [21:06:50] and you essentially can only have one tool [21:06:59] i.e. /~username/ [21:07:08] but now we can do /toolname/ [21:11:34] (03PS1) 10Lokal Profil: Add mechanism for storing wikipage locally instead of writing to wiki [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378800 (https://phabricator.wikimedia.org/T174614) [21:13:26] In the docs its mentioned ssh tools-login.wmflabs.org. And I used ssh mridub@login.tools.wmflabs.org. What's the difference? Does the one present in docs work for every user? [21:17:50] And for creating flask webservice do I need to exit from toolforge first. [21:19:20] Mridu: different external url to the same place [21:19:24] tools-login.wmflabs.org is old format [21:19:29] login.tools.wmflabs.org is new format [21:21:40] in place of login should there be a shell username? [21:22:18] Mridu: I don't understand the question [21:22:28] login.tools.wmflabs.org is an FQDN for a host [21:22:36] the user is prepended to specify [21:22:39] @login.tools.wmflabs.org [21:23:42] I meant ssh login.tools.wmflabs.org is the right way. Or ssh mridub@login.tools.wmflabs.org is the right way. [21:24:05] neither is wrong, the second works for you since your local user and remote user are different [21:24:21] oh okay. got it. [21:29:00] 10PAWS, 10Patch-For-Review: Enable downloading notebooks as PDF - https://phabricator.wikimedia.org/T159617#3616335 (10Tbayer) Thanks! I was able to download one SWAP notebook successfully already. [21:37:12] webservice --backend=kubernetes python shell even after pressing enter unable to see command prompt. Following this tutorial step by step. [21:37:27] https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool [21:49:32] Works. tools.thankyou@interactive has come. [22:10:45] webservice on tools-login claims the service is not running when it is [22:10:49] https://tools.wmflabs.org/list/?list=example [22:10:57] [22:09 UTC] tools.list at tools-bastion-03.tools.eqiad.wmflabs in ~ [22:10:58] $ webservice stop [22:10:58] Your webservice is not running [22:12:19] Krinkle: there is a bug for that somewhere I believe, can you kubectl stop to do it manually? [22:12:52] 1 $ kubectl get pods [22:12:52] NAME READY STATUS RESTARTS AGE [22:12:52] list-3619978338-2t1kp 1/1 Running 0 40d [22:13:04] kubectl stop ods [22:13:06] kubectl stop pods [22:13:33] kubectl stop pods --all [22:13:34] probably [22:13:37] Got it [22:13:45] I was looking in help but didn't see stop [22:13:49] Looks like it's an alias to delete [22:13:54] * chasemp nods [22:14:47] chasemp: Hm.. okay, but it still says it's not runnig and not letting me shell into the pod [22:14:55] It apparently restarted itself [22:15:13] ah teh deployment is still active [22:15:13] $ kubectl delete pods --all [22:15:13] pod "list-3619978338-c3zf7" deleted [22:15:13] [22:15 UTC] tools.list at tools-bastion-03.tools.eqiad.wmflabs in ~ [22:15:13] $ kubectl delete pods --all [22:15:13] pod "list-3619978338-su49q" deleted [22:15:14] [22:15 UTC] tools.list at tools-bastion-03.tools.eqiad.wmflabs in ~ [22:15:14] $ kubectl delete pods --all [22:15:15] pod "list-3619978338-su49q" deleted [22:15:15] pod "list-3619978338-t0nf2" deleted [22:15:18] no commands in between [22:15:46] I guess I need it to be gone and then started by webservice for status/shell/start/stop to work again? [22:15:49] Krenair: I did it [22:15:50] $ kubectl delete deployments list [22:16:03] the short version is webservice creates a deployment which has a replicaset of 1 [22:16:09] which makes sure a prod is running [22:16:21] so nuking the pod is a "normal" event and the scheduler compenstated [22:16:31] nuking the deployment pulls out the supervisor logic [22:16:36] Done with making the app. And ran webservice --backend=kubernetes python start. Webservice is running. But on browser it's showing 502 bad gateway. [22:17:51] Mridu: if you don't figure it out create a task, I have to go afk for a bit [22:18:02] Krinkle: let me if that gets you sorted [22:18:16] Trying to start it now via webservices --backend k8s [22:18:34] Yay [22:18:36] Thanks chasemp [22:19:11] chasemp: yes would do. Thanks [22:20:48] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that load resources from third party sites - https://phabricator.wikimedia.org/T172065#3616449 (10Krinkle) [22:20:52] 10Tools: Tool "list" redirects to GitHub without consent - https://phabricator.wikimedia.org/T172658#3616447 (10Krinkle) 05Open>03Resolved The tool home page now shows a short page with link to Documentation instead o... [22:26:33] (03PS3) 10Lokal Profil: Ensure skipped image categorizations are mentioned in stats [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378641 (https://phabricator.wikimedia.org/T174871) [22:43:38] (03CR) 10Lokal Profil: "this seems to work and allowed me to test the new categorize_images output as well as the statistics and unknown fields output." [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378800 (https://phabricator.wikimedia.org/T174614) (owner: 10Lokal Profil) [22:46:20] (03CR) 10Lokal Profil: "the test output was produced by adding the list of sites in the commit message to a TEST_SITES variable and then, in the top of the countr" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378641 (https://phabricator.wikimedia.org/T174871) (owner: 10Lokal Profil) [22:48:23] (03CR) 10Lokal Profil: [C: 032] Bump PHP dependencies in composer.json [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378758 (owner: 10Jean-Frédéric) [22:49:22] (03Merged) 10jenkins-bot: Bump PHP dependencies in composer.json [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378758 (owner: 10Jean-Frédéric) [22:51:57] (03CR) 10jenkins-bot: Bump PHP dependencies in composer.json [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378758 (owner: 10Jean-Frédéric) [22:54:25] 10cloud-services-team (Kanban), 10Patch-For-Review: Replace kernel and reboot labvirt1015, 1016, 1017, 1018 - https://phabricator.wikimedia.org/T176044#3611990 (10faidon) @chasemp mentioned this odd issue at the meeting today. If there are no (useful?) logs, are there perhaps any hosts that exhibit the non-wor... [22:57:54] (03CR) 10Lokal Profil: "Should isort then be added to the dev-requirements?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378741 (owner: 10Jean-Frédéric) [23:04:51] (03CR) 10Lokal Profil: "is there a particular reason for using bashate instead of shellcheck?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378006 (https://phabricator.wikimedia.org/T175906) (owner: 10Jean-Frédéric) [23:11:47] (03CR) 10Lokal Profil: "> is there a particular reason for using bashate instead of shellcheck?" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378006 (https://phabricator.wikimedia.org/T175906) (owner: 10Jean-Frédéric) [23:23:05] (03CR) 10Lokal Profil: Add Bash linting using bashate (031 comment) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/378006 (https://phabricator.wikimedia.org/T175906) (owner: 10Jean-Frédéric)