[00:08:05] !log tools shut off tools-flannel-etcd-01/02/03 T246689 [00:08:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:08:08] T246689: Toolforge: cleanup legacy kubernetes cluster - https://phabricator.wikimedia.org/T246689 [01:31:23] you would THINK people would at least check to make sure it's running properly ... [01:31:46] or - at least running [02:49:17] !log tools.bash Update to d7de56f (Fix Elastica call to add a quip) [02:49:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bash/SAL [03:32:02] !log tools.bash Updated to 18f7e14 (Bump Elastica to 7.0.0-beta.3) [03:32:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bash/SAL [04:10:08] reviewing recent changes spelling fixes only to find more to do :P [10:31:49] Around 8:10-8:45 UTC today Scholia had a 4xx problem https://grafana-labs.wikimedia.org/d/toolforge-k8s-namespace-resources/kubernetes-namespace-resources?orgId=1&var-namespace=tool-scholia&from=1584417853301&to=1584438303281&fullscreen&panelId=27 There was a lack of response from the Python web application, though I could see some normal activity in uwsgi.log file and no extraordinary issue. Grafana or the log file does not g [10:31:49] ive me any hint of what happened. Is there someone how can help me understand the issue? [10:33:21] hi there fnielsen ! [10:34:07] Hi arturo! [10:34:18] I'm looking at the graph [10:36:10] fnielsen: 4xx may mean people requesting pages that don't exist [10:42:58] Would these page requests be recorded in the uwsgi.log? [10:44:07] I don't think so! [10:49:07] !log tools.wdmm deployed 333328a0d1 (kowiktionary override) [10:49:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmm/SAL [10:49:35] fnielsen: here is a random sample: [10:49:37] https://www.irccloud.com/pastebin/KZcEKM7u/ [10:51:25] "GET /scholia/Q17354701 403 should not be forbidden. I will now redirect to https://tools.wmflabs.org/scholia/topic/Q17354701 [10:52:04] I cannot see why it should generate 403 [10:53:19] the redirect is being generated in your webservice or a custom ingress or something? [10:53:47] That is in the webservice [10:54:20] did you try debug mode or the like? [10:54:52] How do I do that on Toolforge? [10:55:27] your webservice is using which runtime environment? [10:55:29] I suppose I could put in some extra log messages here and there. [10:55:32] python, php, etc? [10:56:03] I am doing this: webservice --backend=kubernetes -m 1G python3.7 restart [10:57:58] I can run it locally on my own computer with "python runserver.py" [10:58:54] Scholia has recently been upgrade to verion 3.7 of Python. [10:59:13] yes, extra log messages (printf based debug) is a valid approach fnielsen [13:29:03] !log tools set `profile::toolforge::bastion::nproc: 200` for tools-sgebastion-08 (T219070) [13:29:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:29:07] T219070: Failure of "kubectl get pods" for the editgroups project - https://phabricator.wikimedia.org/T219070 [14:11:36] question: https://tools.wmflabs.org/cdnjs/ isn't responding for me. is there an alternative way to see the list of javascript libraries hosted on toolforge's CDN? [14:17:30] A couple things here - 1: the obvious spam/advertisement - 2: This doesn't look like a valid username - https://wikitech.wikimedia.org/wiki/Special:Contributions/%22newsblog%22 [15:09:03] !log tools.cdnjs rebuilding the venv on a bastion T246096 [15:09:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cdnjs/SAL [15:09:06] T246096: Tools cdnjs returns blank page - https://phabricator.wikimedia.org/T246096 [15:11:10] !log tools.cdnjs cleared the logs T246096 [15:11:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cdnjs/SAL [15:12:10] !log tools.cdnjs running jsub -l h_vmem=3145728k /data/project/cdnjs/update-index.sh T246096 [15:12:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cdnjs/SAL [15:49:35] thanks bstorm_ i'll watch that task! [17:09:31] Hi any admin to help with simple Toolforge task? [17:13:07] osm4wiki tool claims in its HTML/Perl files utf-8 encoding, but files are apparently saved in Windows-1250 encoding. Tool maintainers are not responding for almost a year. Could someone re-save osm4wiki files in utf-8 encoding as they claim and fix a bug task in Phabricator? [17:13:56] (and thus make the tool work correctly again) [17:41:25] Dvorapa: most of us are busy right now, but that seems like a task that we could help with "soon". It would be helpful if there was a Phabricator task to track the request and what we do. [18:14:07] Is there a way to check which databases a tool has write or modify access on, without knowing the database name? [18:16:38] - that's user-facing via an ssh console session [18:24:30] bd808: This is mainly tracked here: https://phabricator.wikimedia.org/T226481 [18:24:45] DSquirrelGM: iirc the db the tool can r/w is anything within its namespace [18:25:25] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases [18:26:57] the question is more for the purposes of identifying the databases to clean up, etc. like if you need to remove test databases but don't remember the name used [18:27:34] looking for that sort of use case [18:27:38] that idk [18:33:01] DSquirrelGM: On ToolsDB default write permission is based on database name. Specifically the +write permission is granted for databases named "_*" [18:33:41] there can be special grants added to allow other write access, but it is relatively uncommon [18:34:23] DSquirrelGM: https://tools.wmflabs.org/tool-db-usage/ is the public info about which tools have created databases on ToolsDB [18:35:53] * Zppix wonders what other tools.wmflabs.org/* things exist that give info [18:36:26] Zppix: https://tools.wmflabs.org/admin/tools -- there's the big ugly list [18:36:58] help! hi team, can you please provide access to Rummana Yasmeen to be able to ssh to "deployment-eventlog05.eqiad.wmflabs" [18:37:48] mayakpwiki: That requires joining the deployment-prep project. The folks in #wikimedia-releng might be good to ask about getting that access [18:37:58] she's a QA on the technology team and needs to be able to help product-analytics team with QAing the data in all-event.log file in beta cluster [18:38:32] I got my access 3 weeks by requesting here.. [18:38:41] this is the channel for cloud services - that's outside of our department, so to speak [18:38:43] I got my access 3 weeks ago by requesting here.. [18:39:15] mayakpwiki: you did, because Krenair was paying attention and took care of it. He is one of the active admins for the deployment-prep project [18:39:48] I wonder how many "Assamese Calendar" tools we need (I count 3-4) [18:40:04] DSquirrelGM: not necessarily, I was just pointing to #wikimedia-releng with a larger concentration of deployment-prep admins [18:41:43] Zppix: we need however many folks maintain. :) It would often be nice to see more collaboration and less duplication, but that's more about getting better support for good tools than about reducing the total number of tools [18:42:07] bd808: the thing is they all have the same maintainer :P [18:43:09] Zppix: ah. that looks like they are actually all the same tool account but that there multiple toolinfo.json records [18:43:18] oh [18:43:25] and that's kind of a bug in toolsadmin/Striker I bit [18:43:29] *I bet [18:44:28] yea, there seems to be no method to delete duplicate / unused records [18:44:52] at least not on the management interface [18:45:09] patches welcome :) [18:45:40] no idea where to even look for where they're actually stored [18:46:06] DSquirrelGM: https://wikitech.wikimedia.org/wiki/Toolsadmin.wikimedia.org is a good place to start learning [21:47:49] !log tools.zppixbot-test chmod -R g+w [21:47:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [21:52:41] Can anyone tell why I can see sftp://login.tools.wmflabs.org/mnt/nfs/labstore-secondary-tools-project/zppixbot/.sopel but not sftp://login.tools.wmflabs.org/mnt/nfs/labstore-secondary-tools-project/zppixbot-test/.sopel in my client. It should be there as I can ssh into the tool and cd there but ftp won't show it [21:55:03] My access should and seems to be the same [21:57:55] I don't see anything off hand that would cause that, the .sopel directories look the same [21:58:08] typo in the project name? zppixbot-test/.sopel [21:58:17] nope, sorry [21:58:46] `ls -lad /mnt/nfs/labstore-secondary-tools-project/zppixbot-test/.sopel` looks OK from our end [21:58:55] I've edited it before when I logged in today, It won't show [21:59:05] can you see the zppixbot-test directory? [21:59:27] I can see sftp://login.tools.wmflabs.org/mnt/nfs/labstore-secondary-tools-project/zppixbot-test/ [21:59:42] but .sopel is missing from it when I open it [21:59:45] do you see any other hidden directories, like .toolkube ? [22:00:38] https://usercontent.irccloud-cdn.com/file/fDDY1tje/Screenshot%202020-03-17%20at%2022.00.29.png [22:00:44] hmm, good point [22:01:01] yeah, you've got hidden directories hiddne [22:01:03] n [22:01:23] there should be a right-click menu option for it, might be Cmd-H as well [22:01:46] that would be it [22:04:04] also looks like you could use a cleaning on some of the logs here, like a 4.6MB error log [22:04:12] !log tools.zppixbot-test rhinosf1@tools.zppixbot-test syncing files [22:04:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [22:04:34] !log tools.zppixbot-test rhinosf1@tools.zppixbot-test syncing files & dbs from productions [22:04:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [22:07:21] DSquirrelGM: which log? [22:08:24] cron-tools-*-1.err [22:09:22] . not -, oops but that file [22:09:50] DSquirrelGM: still can't find it, what's full path to it? [22:11:00] !log tools.zppixbot accidentally moved (rather than copied) know users database, recovered db [22:11:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [22:11:32] !log tools.zppixbot-test rebooting to sync database and config [22:11:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL [22:11:41] the err file in /mnt/nfs/labstore-secondary-tools-project [22:11:46] oops nm [22:11:56] forgot the subdirectory [22:12:17] zppixbot [22:13:04] oh that, I'll look [22:13:36] the access.log is almost as large [22:14:45] cron is full of: [22:14:50] https://www.irccloud.com/pastebin/H5rc5YCt/ [22:15:44] access.log hasn't been cleared since 2016? [22:17:25] anyway was originally just checking to see if any permissions may have been missing for the files not to show up in the directory listing [22:19:27] I have no clue what's triggering them alerts, I've asked other maintainers about scrapping other logs