[05:30:32] !log tools.merlbot-web Shutdown webservice running on Trusty grid. Maintainer non-responsive to email notifications, no source code, no license. [05:30:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.merlbot-web/SAL [05:34:34] !log tools.random Shutdown webservice running on Trusty grid. Maintainer non-responsive to email notifications, no source code, no license. [05:34:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.random/SAL [07:07:58] bd808: is ldap3 the recommended python3 library for doing LDAP stuff? (my tool has been using python-ldap>=3.0 but I don't remember what the level of upstream support was) [10:22:44] legoktm, looks like ldap3 is preferred: https://phabricator.wikimedia.org/R1962:f63db10f38f75dfe794deb5379cfbf967e9a90d1 [10:22:47] but: https://phabricator.wikimedia.org/T214541 [13:17:42] Looks like AnomieBOT jobs 351098 and 839511 are stuck now. 839511 looks like blocking on NFS again AFAICT. I can't even ssh to tools-sgeexec-0938 to look at the other. [13:23:15] !log wikidata-dev wikidata-shex configured wgWBSchemaShExSimpleUrl (T217331) [13:23:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-dev/SAL [13:23:18] T217331: [Story] Add link to Schema page to check against list of entities - https://phabricator.wikimedia.org/T217331 [14:02:34] !log ores staging ores-wmflabs-deploy:9a0e589 [14:02:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [14:05:15] !log ores deploying ores-wmflabs-deploy:9a0e589 [14:05:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL [14:05:57] OK maintenance notice is posted! [14:21:46] I just posted our proposed goals for Q4. See https://www.mediawiki.org/wiki/Wikimedia_Technology/Annual_Plans/FY2019/TEC5:_Scoring_Platform/Goals#Q4_Goals [14:21:54] Woops. Wrong channel :) [14:54:12] !log git updated apt repo for packages.sury.org to update apt.gpg key for apt repo. [14:54:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [15:19:37] anomie: do you need those jobs force deleted? [15:20:18] bd808: I can try deleting them myself first if you want. [15:52:40] bd808: I qdeled them. We'll see if they notice. [16:08:04] !log tools.anomiebot Force deleted jobs 351098 839511 that were stuck [16:08:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.anomiebot/SAL [16:08:29] Thanks [17:53:34] !fingerprint [17:54:43] Betacommand: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [17:55:24] bd808: yep, was just seeing if we had a bot for that here or not [17:55:47] bd808: I was fielding the question in a different channel [17:58:31] !fingerprint is https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints [17:58:31] Key was added [17:59:24] +1 [17:59:31] thanks for that [17:59:32] bd808: thanks [18:25:17] !log tools removing postStart hook for PWB update and restarting hub while gerrit.wikimedia.com is down [18:25:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:25:36] ha, missed PAWS enterely on that message [18:26:08] .com? :P [18:26:40] * chicocvenancio is not in the best "log message writing" mojo today [18:26:49] but PAWS works again [18:40:44] !log rebooting tools-static-13 in hopes of fixing some nfs mounts [18:40:45] andrewbogott: Unknown project "rebooting" [18:40:56] !log tools rebooting tools-static-13 in hopes of fixing some nfs mounts [18:40:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:41:11] !log tools PAWS: deleting pods stuck in Unknown state with ` --grace-period=0 --force` [18:41:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:42:08] !log tools PAWS: 3 nodes still in not ready state, `worker-10(01|07|10)` all else working [18:42:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:43:18] !log tools Rebooting tools-static-12 [18:43:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:54:45] !log liangent-php deleting liangent-php/mw-log because it is causing ext4 index errors and NFS errors [18:54:45] bstorm_: Unknown project "liangent-php" [18:54:45] bstorm_: Did you mean to say "tools.liangent-php" instead? [18:55:04] Why yes I did, stashbot, thanks :-p [18:55:20] !log tools.liangent-php deleting liangent-php/mw-log because it is causing ext4 index errors and NFS errors [18:55:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.liangent-php/SAL [18:55:33] such a helpful bot ;) [19:36:55] !log tools.liangent-php commenting out all crons to stop log file creation [19:36:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.liangent-php/SAL [19:38:27] !log tools.liangent-php stopped continuous job [19:38:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.liangent-php/SAL [20:59:30] !log tools.stashbot Restarted irc bot as part of investigation of wikitech auth issues [21:09:52] Krenair: oh wonderful, thansk [21:25:37] !log wikibugs Restarting wikibugs. Deleting jobs first, then relaunching. [21:25:38] hauskatze: Unknown project "wikibugs" [21:25:38] hauskatze: Did you mean to say "tools.wikibugs" instead? [21:25:46] !log tool.wikibugs Restarting wikibugs. Deleting jobs first, then relaunching. [21:25:46] hauskatze: Unknown project "tool.wikibugs" [21:25:50] fxs [21:26:02] !log tools.wikibugs Restarting wikibugs. Deleting jobs first, then relaunching. [21:26:02] hauskatze: Failed to log message to wiki. Somebody should check the error logs. [21:26:04] tools.wikibugs :) [21:26:32] well, now I spelled it right [21:27:55] valhallasw`cloud: so python3 manage.py start_jobs does not exist anymore [21:28:06] is the crontab command valid to restart the bot? [21:32:36] !log tools.wikibugs Restarting wikibugs. Deleting jobs first, then relaunching. [21:32:37] hauskatze: Failed to log message to wiki. Somebody should check the error logs. [21:32:47] stashbot is in troll-mode today [21:32:47] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [21:33:06] wee [21:33:08] :) [21:34:17] valhallasw`cloud: nvm, restarted it already :) [21:37:37] hauskatze: yeah, stashbot is sick. It can't auth to wikitech at the moment. working on it :/ [21:37:55] blocked? [21:38:00] or IP blocked? [21:38:02] bug [21:38:51] mw sucks [21:42:25] fwiw wikibugs refuses to enter the pywikibot channel apparently [21:42:35] although changes have been made to tasks [21:50:26] hauskatze: thanks for restarting it. Maybe wb2-phab is having issues - not sure if there were any auth changes wirh the changes today. Can't look into it right now, but will look at it tomorrow evening CET [21:50:53] I can take a look at the logs meanwhile, but thanks if you can take a look tomorrow [21:54:03] yeah, it's wb2-phab [21:54:05] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) [21:54:18] when tail wb2-phab -n 50 [22:30:43] wb2-phab seems to be working again [22:35:44] Hi, so my inbox has been inundated with this "Tools you maintain are running on Trusty job grid" warning over the past weeks, and I've successfully procrastinated about it until now. Unfortunately, I'm too incompetent to understand the email and the associated help page, so was hoping for some help here ... First of all, may I just ask if the "tool" (in my case: a lighttpd instance - https://tools.wmflabs.org/otrsreports/index.html -) will stop [22:35:44] working if I do nothing about this? [22:36:56] pajz: yes [22:36:59] next week [22:37:31] its probably not hard to migrate though [22:40:55] chicocvenancio, I see, thanks! Is this - https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP#How_to_set_up_PuTTY_for_direct_access_to_your_Toolforge_account - still how you set up PuTTY? Back then, someone helped me set it up but I've since migrated to a different computer and the configuration seems to have gone lost alas. [22:42:34] pajz: yes [22:42:42] I can help you find any error [22:44:23] That would be great :). [22:44:27] pajz: I think those instructions should work. The hostnames in the screen shots are not the most current. I would recommend using "login.tools.wmflabs.org" anywhere it says "tools-login.wmflabs.org" and "stretch-dev.tools.wmflabs.org" instead of "tools-dev.wmflabs.org" [22:47:00] Ok, thanks. Hope I'll find that private key somewhere ... [22:49:03] if you can log into wikitech you should just be able to create a new key and upload the new public part to wikitech prefs [22:49:52] * chicocvenancio was about to write that, but is not sure if wikitech is logable atm [22:50:08] Should be fine for normal users [22:50:14] if you can't log into wikitech well you should fix that : [22:50:14] :) [22:50:38] Reedy: I think scap-update-interwiki-cache is also affected by the https commit push ban? [22:50:43] yes, wikitech is working for me. just horizon and tools admin are not [22:50:55] use the ldap pass hauskatze [22:50:55] hauskatze: Yup, along with branching and stuff [22:51:07] paladox: Doesn't help [22:51:19] I don't really want my ldap password on disk in a file [22:51:21] Reedy: well maybe we can use the mwscript dumpInterwiki.php script Reedy [22:51:29] oh, does scap-update-interwiki-cache use phabricator? [22:51:29] so, pajz, if you can't find the private key, but can find the 2fa. you can log in to wikitech and change the private key [22:51:57] paladox: no, it fetches m:Interwiki_map and asks for your gerrit HTTPS password to update a patch on your behalf [22:52:11] bd808: Looks like jobs 387053 and 351101 are probably stuck now, they don't seem to have gone away with qdel. 351103 was seemingly stuck but did go away with qdel. [22:52:48] oh, Well it would be really insecure if you stored your ldap pass on disk. [22:52:51] but use ssh :) [22:52:53] chicocvenancio, we're talking about wikitech.wikimedia.org, yes? [22:53:05] pajz: yes [22:53:16] paladox: That doesn't help [22:53:25] We can't use SSH as we won't use agent forwarding in prod [22:53:30] !log tools.anomiebot Force deleted stuck jobs 387053 & 351101 [22:53:30] Because, that's also insecure [22:53:31] bd808: Failed to log message to wiki. Somebody should check the error logs. [22:53:32] oh, ok. [22:53:48] And by uploading the public part, you mean through Special:Preferences / OpenStack? [22:53:50] coulden't https (without auth) work? [22:54:34] pajz: yes. that's where you add ssh public keys to your account [22:55:06] paladox: How are you going to push? [22:56:24] i thought it was pulling not pushing :) then i re read hauskatze comment and now realise it pushes, but nvm. Each way is insecure then. [23:08:20] Thanks [23:19:46] bd808, chicocvenancio, ok, created a key pair using PuTTY's key generator, changed the public key on wikitech, and tried to connect to login.tools.wmflabs.org, but am getting a fatal error: "No supported authentification methods available (server sent: publickey,hostbased)." Any ideas what I might be doing wrong? [23:20:28] seems you did not set the key in putty [23:20:48] * chicocvenancio gets putty opened to see the same screens [23:21:49] pajz: how is your Connection > SSH > Auth tab in putty? [23:23:06] the `Private key file for authentication:` should have your ppk file therex' [23:23:53] The same as in https://upload.wikimedia.org/wikipedia/labs/7/7f/20130526_2133_Putty_Login_Connection_SSH_Auth.png, except there is an additional option "Display pre-authentification banner" (enabled). And the "private key file for authentification" field has the path of my private key in it. [23:24:54] huh [23:25:05] pajz: and you have your username set? [23:25:21] the username is my wikitech user name, right? [23:25:32] it may be different [23:25:52] what is your wikitech username? [23:26:07] Pajz [23:26:12] pajz: it is your "Instance shell account name:" from https://wikitech.wikimedia.org/wiki/Special:Preferences [23:26:48] in that case, pajz [23:27:10] so, you have that set in Connection > data or on the hostname? [23:27:17] is that case-sensitive? [23:27:41] yes [23:27:55] oh, then that might be the issue [23:29:22] perfect, looks like that's done the trick :) [23:29:54] pajz: I imagine you're in `tools-sgebastion-07` [23:30:02] yeah [23:31:22] you have a lighttpd gridengine tool [23:32:03] you can easily migrate that to a kubernetes php tool. I think they share directory assumptions [23:32:37] bd808: is this true ^? [23:34:40] chicocvenancio: mostly, yes. The lighttpd grid type is mostly the same as the php5.6 and php7.2 kuberntes types. On the grid you can mix php, python, and perl fcgi handlers; on kubernetes you have to pick one of the set. [23:36:55] pajz: it all seems to be just static files right? [23:37:12] yes [23:37:24] then I'd just go `webserive stop` [23:37:32] `webservice stop` [23:38:08] huh, not having done that, I think that will error out [23:38:35] yeah, he needs to be on the trusty bastion to stop the current webservice [23:38:36] Do I need to follow the commands suggested in ? [23:39:07] yes [23:39:26] namely open a new putty connection, can be cloned from the current one and change the host to login-trusty.tools.wmflabs.org [23:39:41] * bd808 apologizes for gendering pajz without asking [23:40:31] so easy to slip in to gendered language for no good reason [23:40:47] bd808, all fine, also, it's accurate :) [23:43:25] chicocvenancio, if I've done that, I don't need to use "ssh @login-trusty.tools.wmflabs.org", or do I? [23:43:36] no [23:43:56] that is a shorthand for non-putty users [23:45:14] webservice stopped [23:45:48] do you still have the tools-sgebastion-07 connection going, pajz ? [23:45:56] yep [23:46:09] in that connection you can start it again [23:46:21] I recommend you also change the type and backend [23:46:24] so you'd do [23:47:13] `webservice --backend=kubernetes php7.2 start [23:47:17] `webservice --backend=kubernetes php7.2 start` [23:47:29] as your tool [23:49:01] done that [23:50:12] that's it [23:50:35] https://tools.wmflabs.org/otrsreports/index.html is working from kubernetes now [23:51:33] you should stop getting emails [23:51:40] nice. this was one-time thing, right? [23:51:44] yes [23:51:59] until some 5 years in the future or something [23:52:27] but since you moved to kubernetes as well, it is likely things may be updateable without you intervention [23:52:40] maybe [23:52:59] What host do I have to connect to in the future if I want to, say, change some files? [23:53:22] the standard one, login.tools.wmflabs.org [23:53:36] the trusty one will die sometime next week [23:55:25] I see. Alright - thanks a lot to you, chicocvenancio and bd808 :)