[12:54:07] Hey there! :) Can anyone help me with a bot operating in Toolforge? My bot has been operating fine for some days and then suddenly the last week (or a bit prior to that) it stopped, without me doing any change on its code. I get an error which I don't really understand. Strangely enough, I can't even replicate the first example detailed here now: [12:54:08] https://www.mediawiki.org/wiki/Manual:Pywikibot/replace.py Anyone with some free time willing to help me? [12:56:14] !help [12:56:14] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [12:58:39] Bigem: I might be able to help, but I'm about to go grab some food [12:58:49] Bigem I'd recommend opening the task and putting the error there, I can take a quick look, though I have no experience with it I know some python :) [12:59:09] * arturo waves to dcaro [12:59:21] XD Okay, so: [12:59:33] arturo: heyo \o [13:03:00] https://paste.toolforge.org/view/b9a39e96 [13:03:04] This is what I get. [13:04:28] Bigem: does the script /mnt/nfs/labstore-secondary-tools-project/smallem/Smallem.sh exist? [13:04:59] I'm not sure about "/mnt/nfs/labstore-secondary-tools-project/smallem/" but "Smallem.sh" exists. [13:05:05] That's what I'm trying to run. [13:05:14] As the job. [13:05:21] That's the source code of the bot. [13:07:21] okok, hmm, from the trace it seems that the code is failing to get the password prompt when trying to get a password to authenticate to wikipedia:sq [13:07:47] is it running on the background as a daemon? (it might require some auth configuration) [13:08:46] (btw. I'm kinda configuring my irc client, so I might disconnect/connect from time to time) [13:10:33] It used to run like that. It was set to run forever (jstart) and I used a crontab to "start" it every 2 weeks (in case for whatever reason it might have stopped). [13:11:15] It fixes reference errors in SqWiki. But then some days ago, I found it it wasn't fixing any errors anymore. [13:11:57] And I opened it to see what was happening and I saw that I had been getting that error for many days straight (in Smallem.err) and it was stuck on a loop. [13:12:29] how do you authenticate the script to do the modifications? (careful don't pass any user/pass/apikey/...) [13:12:30] I stopped it and tried restarting it, it didn't work. I tried using "jsub", to run it only once, it still didn't work. [13:12:51] And ever since, I haven't been able to make it work. [13:13:04] Hmm, it has an account on SqWiki. [13:13:33] so how do you pass the script the user/pass/apikey of that account? [13:13:35] And I've logged there many weeks ago one time with user-generated files, if I'm not wrong. [13:14:21] To be honest, I'm not sure because it literally has been months since I had to do the first authentication (and the last). [13:14:50] I think it was related to autogenerating the user-config.py files. [13:15:22] Ever since those files were generated, it got authenticated and it would start and do edits. [13:16:35] it sounds as if the auth might have expired, but I don't know enough of pywikibot yet to help much there right away [13:17:16] you can try regenerating the user-config.py (I'd make a backup first just in case), see if that helps [13:18:03] (maybe python pwb.py login [13:18:06] is enough) [13:18:21] Let me try it... [13:19:31] python: can't open file 'pwb.py': [Errno 2] No such file or directory [13:19:38] This is really strange. [13:19:51] I had once in the past a situation like this. [13:20:22] It turned out the code behind pywikibot was damaged by a certain user trying to update it and he had to do the fix, not me. [13:20:51] I couldn't even autogenerate the user-config.py files back then because it stopped doing that (for everyone, not just me). [13:21:02] That's why I'm afraid to delete them. :P [13:22:59] I'm trying to regenerate them without deleting. Maybe it helps... [13:24:39] okok, keep us posted :) [13:25:12] btw. are you under ~/pywikibot? [13:27:47] Wait, it turns out I'm an idiot. :P How do you give the command to autogenerate the files without deleting them? :P [13:27:58] Is there a way? :P [13:29:05] python3 /data/project/shared/pywikibot/stable/generate_user_files.py [13:29:06] Found it :P [13:29:23] neat [13:31:00] https://paste.toolforge.org/view/24678745 And I can't do even that. I get this error. [13:31:53] I really thing there is an error with the stable core branch of Pywikibot and it's not related to me. [13:32:36] https://phabricator.wikimedia.org/T261771 This was the problem back then and we had to report it to Phabricator for it to get fixed. [13:33:20] that error looks different though, "OSError: [Errno 27] File too large", I wonder what file is it [13:35:17] I don't know what to say. :/ But I guess everyone should be able to autogenerate its user files no matter what, no? "5. Configure Pywikibot. [13:35:34] tools.tool@tools-login$ python3 /data/project/shared/pywikibot/stable/generate_user_files.py [13:35:59] In this page: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot it says it should just work. No matter what. [13:40:35] well, unexpected situations happen xd, trying to give a quick look at the code at that point, see what the "too large" file is [13:41:21] it seems to be a logfile, commands.log [13:42:01] that should be somewhere where your datafilepath config points to, under logs/ directory [13:42:22] does that ring any bells? (maybe with the cron running it never got rotated, and just grew too much) [13:43:48] Well, the script file IS too large. Smallem.sh 49000 lines of code. That's intended. :P [13:44:15] But it used to work fine. It worked fine for some weeks before having this problem. [13:44:36] no, the script looks ok, it's the logfile that might be too big [13:44:44] the commands.log logfile [13:44:57] Maybe I should try delete it? [13:45:24] (that fits well with working for a while and then start crashing, as it just appends data to the logfile) [13:45:54] try first just moving it to something like command.log.bkp_20201020, just in case [13:46:12] if that works, then probably can be removed (ask arturo first though, not sure what's in there) [13:47:19] wait, let me find the said file first. [13:47:32] yep +1 [13:53:58] Aaaand, no. I can't find what you meant. :P [13:56:21] hmm, do you have a PYWIKIBOT_DIR environment var defined? (try running 'echo $PYWIKIBOT_DIR') [13:57:13] It shows nothing. [13:57:14] Bigem: you might want to check also under ~/.pywikibot/logs/ [13:58:00] Illegal file name - that's what it reads [13:58:11] I mean, I tried: vi ~/.pywikibot/logs/ [13:58:27] And it showed me an empty page and in the bottom, it read Illegal file name. [13:58:55] can you try to 'ls -la' instead? [13:59:14] (you might not have the plugin to read directories in vi) [13:59:18] Just a second. [13:59:27] no rush [14:00:16] Yes, I got something. [14:00:21] Should I paste it here? [14:00:53] I'm not sure what it means though. First time accessing this and I'm not that literate on Linux. :P [14:02:55] probably pastebin, but there should be a 'commands.log' file there, with a very big size [14:07:05] https://paste.toolforge.org/view/423b4165 I get this [14:08:54] ok, there's a 'logs' directory there, can you do 'ls -la' of that one too? [14:09:16] (I guess something like 'ls -la logs') [14:09:51] total 77368864 [14:09:56] I get this. ^ :P [14:10:13] The number does seem big. [14:11:10] nice, is there a 'commands.log' file listed there? [14:11:33] What do you mean? [14:11:45] You didn't saw what I sent above? [14:12:35] in that listing there was also a 'logs' directory, not a 'commands.log' file [14:12:48] (unless I missed something xd) [14:13:10] Wait, let me try again with pastebin. [14:13:20] okok [14:13:27] https://paste.toolforge.org/view/a49abdea [14:15:10] yep, that's around 80GB xd, try renaming: 'mv commands.log commands.log.bkp.20201020' (from inside that directory) [14:15:51] then try running the script again, if it fails, it should be for another reason al least xd [14:16:47] got to go though, I'll be around later I guess, if that does not work, try creating the ticket so someone can help you further/later [14:16:53] (fingerscrossed) [14:17:59] Well, 2 questions: How do I rename it? And why not just delete it altogether? :P [14:21:42] bd808, hello there! Sorry to disturb! You've helped me the last time with my bot. Maybe you can help again? :P (If you have some free time.) [14:48:44] !log tools.smallem Deleted 74G ~/logs/commands.log log file [14:48:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.smallem/SAL [14:49:06] !log admin [codfw1dev] reimaging labtestvirt2003 (cloudgw) to test puppet code (T261724) [14:49:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:49:08] T261724: cloudgw: evaluate / validate setup in codfw1dev - https://phabricator.wikimedia.org/T261724 [15:47:05] !log admin changing DNS recursor ACLs (https://gerrit.wikimedia.org/r/c/operations/puppet/+/635314) this can be reverted any time if it causes problems (T261724) [15:47:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:47:09] T261724: cloudgw: evaluate / validate setup in codfw1dev - https://phabricator.wikimedia.org/T261724 [15:56:10] !log quarry restarting nginx on quarry-web-01 [15:56:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [16:00:30] !log quarry rebooting quarry-web-01; lots of cruft in /tmp [16:00:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [16:11:31] !log quarry restarted mariadb on quarry-db-01 so it pointed to the right data directory [16:11:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [17:05:57] !log clouddb-services shut down replication on clouddb1002 [17:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:07:27] !log clouddb-services shut down replication on clouddb1002 (now with task) T263677 [17:07:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:07:29] T263677: Upgrade cloudvirt1019 and cloudvirt1020 to Buster - https://phabricator.wikimedia.org/T263677 [17:08:13] !log clouddb-services stopping mariadb on clouddb1002 T263677 [17:08:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:08:46] !log clouddb-services poweroff clouddb1002 T263677 [17:08:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:13:42] !log clouddb-services stopping postgresql on clouddb1003 T263677 [17:13:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [17:13:45] T263677: Upgrade cloudvirt1019 and cloudvirt1020 to Buster - https://phabricator.wikimedia.org/T263677 [17:14:15] !log clouddb-services shutting down clouddb1003 T263677 [17:14:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [18:36:34] !log clouddb-services brought up mariadb and replication on clouddb1002 T263677 [18:36:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [18:36:38] T263677: Upgrade cloudvirt1019 and cloudvirt1020 to Buster - https://phabricator.wikimedia.org/T263677