[11:35:09] (03PS1) 10Paladox: Remove ORES-worker06 and ORES-worker05 [labs/icinga2] - 10https://gerrit.wikimedia.org/r/455142 [11:35:46] (03PS2) 10Paladox: Remove ORES-worker06 and ORES-worker05 [labs/icinga2] - 10https://gerrit.wikimedia.org/r/455142 [11:35:59] (03CR) 10Paladox: [V: 032 C: 032] Remove ORES-worker06 and ORES-worker05 [labs/icinga2] - 10https://gerrit.wikimedia.org/r/455142 (owner: 10Paladox) [11:44:50] (03PS1) 10African Hope: Added some unit tests for UD methods [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) [11:53:28] (03CR) 10D3r1ck01: [C: 04-1] "Thanks for the patch, some fixes :)" (035 comments) [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [11:57:47] (03CR) 10D3r1ck01: [C: 04-1] "recheck" [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [11:58:16] (03CR) 10jerkins-bot: [V: 04-1] Added some unit tests for UD methods [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [12:07:20] o/ how do I find out which lab virt integration-slave-docker-1025 is on? [12:07:26] arturo: ^^ :D [12:08:00] addshore: you can't :-) I can do it for you [12:08:05] thanks! :D [12:10:13] arturo: hi! Maybe the labvirt host could be exposed in Horizon? It might just be a permission settng of some sort :] [12:10:23] or in openstack browser :D [12:12:17] (03PS2) 10African Hope: Added some unit tests for UD methods [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) [12:13:24] addshore: labvirt1008.eqiad.wmnet [12:13:28] https://www.irccloud.com/pastebin/r1QYViwQ/ [12:13:57] hashar: disk util doesn't look too bad on that one [12:14:34] maybe run 5 quibble jobs in parallel is too much [12:14:36] :\ [12:14:45] perhaps [12:14:52] hashar: sounds a fair request, but right now we are dealing with other stuff and is unlikely we work on that in the short term. Feel free to open a phab task requesting the feature though :-) [12:15:17] arturo: its a shame, im pretty sure it used to be exposed on wikitech in the Nova pages [12:16:10] :-/ [12:16:40] arturo: how about integration-slave-docker-1014 ? [12:17:06] addshore: labvirt1007 [12:17:12] and 1008? [12:17:15] integration-slave-docker-1008 [12:17:31] labvrit1005 [12:17:51] okay, so no connection between those long jenkins jobs and the underlying machines :p [12:18:05] how about integration-slave-docker-1024 lastly arturo ? [12:18:31] labvirt1004.eqiad.wmnet [12:19:01] thanks! [12:22:56] (03CR) 10D3r1ck01: "recheck" [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [12:23:21] (03CR) 10jerkins-bot: [V: 04-1] Added some unit tests for UD methods [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [12:23:22] addshore: hashar: store this query for future usage, just ask one of us to run it for you :-P [12:23:26] https://www.irccloud.com/pastebin/oRkv4LKV/ [12:27:51] ah found it [12:27:51] $ nova hypervisor-list [12:27:52] ERROR (Forbidden): Policy doesn't allow os_compute_api:os-hypervisors to be performed. (HTTP 403) [12:27:52] :D [12:27:55] * hashar fills a task [12:29:04] hashar: but that won't give you a lot of info [12:29:08] https://www.irccloud.com/pastebin/zMhKCQ4l/ [12:29:27] ahhhhh [12:35:08] done as T202724 :] [12:35:09] T202724: Expose hypervisor of instances in Horizon interface - https://phabricator.wikimedia.org/T202724 [12:37:56] thanks hashar !! [12:50:44] zhuyifei1999_: my disk is full again. [12:51:00] But all of my cron entries now use &> /dev/null [12:51:14] I'm on an aircraft and can't ssh right now [12:51:27] zhuyifei1999_: happy flying [12:51:36] can you run that lsof command? [12:52:27] and did you make sure no crons started before you edited your crontab? [12:58:37] Yes, everything is running with the new cron entries [12:59:36] zhuyifei1999_: /tmp/tmpfj8EDS3 [13:17:10] zhuyifei1999_: it came from one single process. [13:17:17] So I killed it. [13:17:19] arturo, fair enough, I figured it was better to ask first before submitting more than a few jobs [13:37:11] (03PS3) 10African Hope: Added some unit tests for UD methods [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) [13:41:35] !log wikilabels b3f1b67 is going to staging [13:41:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [13:43:21] (03CR) 10African Hope: "Patch Set 3:" [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [13:44:05] !log wikilabels cf15bcc is going to staging [13:44:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [13:46:54] !log wikilabels cf15bcc is going to prod [13:46:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [13:52:43] (03CR) 10African Hope: "Patch Set 3:" [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [13:53:09] (03CR) 10African Hope: "Patch Set 3:" [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [13:54:16] (03CR) 10African Hope: "recheck" [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [13:58:06] (03CR) 10D3r1ck01: "recheck" [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [13:59:24] Cyberpower678: ok [14:00:09] zhuyifei1999_: but I'm rapidly losing disk again [14:00:12] what's the fd number (the one with a u suffix) and process name though? [14:00:50] I'm landing in 3 hours [14:01:12] zhuyifei1999_: I've got a fuck ton of deleted files now [14:01:22] All either 1u or 2u [14:01:29] (03CR) 10D3r1ck01: "Few more minor fixes" (032 comments) [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [14:01:45] just check the big ones [14:01:58] I'm now losing 1% every 10 seconds [14:02:09] I used `awk '$7 > 65536'` to filter [14:02:12] oh my [14:02:52] so that's 1000 seconds, ~ 16 minutes [14:03:12] could you paste a sample crontab entry? [14:03:45] zhuyifei1999_: didn't you see it last night? [14:03:51] I'm nearly inaccessable to anything besides irc (and no I can't even refresh this page) [14:04:11] zhuyifei1999_: * * * * * flock -n /home/IABot/flock/lvwikimaster.lockfile php /home/IABot/IABot/deadlink.php lvwiki master &> /dev/null [14:04:13] yes I saw, but I cannot ssh in at the moment [14:04:45] I killed the errant process. Disk usage down to 24% again, [14:05:18] ok so what processes and corresponding fd numbers are that big file? [14:05:30] The wierd thing is, it remains stable for a while, then it just goes beserk and chews up disk [14:05:44] php 9683 cyberpower678 1u REG 254,3 9319296882 3073 /tmp/tmpfBDJoet (deleted) [14:05:44] php 9683 cyberpower678 2u REG 254,3 9319296882 3073 /tmp/tmpfBDJoet (deleted) [14:06:12] any other process names? [14:06:30] fd 1 and 2 are stdout and stderr [14:06:44] Plenty, but the largest now is 6 MB and holding [14:07:12] is there any named 'bash' 'flock' or 'cron'? [14:07:26] (03PS4) 10African Hope: Added some unit tests for UD methods [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) [14:07:52] I'm certain IABot doesn't create 9GB of stdout in a matter of minutes. I can see stderr though if something critical failed. [14:08:02] zhuyifei1999_: you can name? [14:08:17] Oh wait. [14:08:25] you can see the stdout/err via /proc [14:08:44] It's only PHP and flock [14:09:03] disk usage is stable for now [14:10:02] But it's only a matter of time until it goes nuts again, for some reason. [14:10:50] I just checked flock.c of util-linux 2.32.1 (my local install) and don't to it creating temp files [14:11:06] *to => see [14:11:38] The only temp file it creates is a locked file that acts as a switch that the PHP program is still running. [14:11:42] I mean. it's not supposed to change stdout or err [14:11:53] yes ik [14:12:18] Doesn't the PHP program pass the stdout and stderr to flock which in turn outputs it? [14:12:38] you said 'I can see stderr though if something critical failed.', does that mean you can see the stderr of that running process? [14:13:14] What I meant is that if something broke in IABot, that I wasn't aware of, I can see it spewing error messages. [14:13:36] I wouldn't have figured it would create 9GB of messages in minutes though. [14:13:39] (03CR) 10D3r1ck01: "Sorry, thought you had fixed this, just 1 more little issue :)" (031 comment) [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [14:13:49] But I haven't actually looked into the file. [14:13:53] I'm pretty sure php inherits the std streams from flock, which in turn inherits from bash [14:14:00] (03CR) 10D3r1ck01: "recheck" [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [14:14:33] &> /dev/null is a bash-ism to redirect both out and err to /dev/null [14:15:13] ok can you check the parent processes of flock? [14:15:46] like whether bash execve'ed into flock or it opens flock as a child process [14:16:12] umm... [14:17:41] and yeah, I'm pretty sure you should really read that stderr stream from /proc. gigabytes of warnings is too crazy [14:17:42] (03PS5) 10African Hope: Add some unit tests for UD methods [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) [14:18:42] zhuyifei1999_: any chance of making it possible to figure which specific process that stream belongs to? [14:18:43] (03CR) 10D3r1ck01: "recheck" [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [14:18:53] I have 23 instances of PHP doing different things. [14:19:05] It would be nice if I could name these processes. [14:19:20] say the lsof output is [14:19:21] php 9683 cyberpower678 1u REG 254,3 9319296882 3073 /tmp/tmpfBDJoet (deleted) [14:19:34] then the PID is 9683 [14:20:01] I get that much, but how do I see what arguments was passed to the call that spawned 9683 [14:20:29] (03CR) 10D3r1ck01: [C: 032] Add some unit tests for UD methods [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [14:20:57] Because it's only one process going wild while everything else is running as it should, and they use identical code. [14:20:58] (03Merged) 10jenkins-bot: Add some unit tests for UD methods [labs/tools/awmd-stats] - 10https://gerrit.wikimedia.org/r/455146 (https://phabricator.wikimedia.org/T202231) (owner: 10African Hope) [14:21:33] the /proc way would be `cat /proc/9683/cmdline | tr '\0' ' '; echo` [14:21:49] or you can do it with ps with `ps ufww 96831 [14:21:59] * `ps ufww 9683` [14:23:14] * zhuyifei1999_ uses /proc when things are really broken [14:24:56] I'll use it when the disk usage spins out of control once again. [14:25:13] Still stable at 25% [14:25:35] This is why IABot 2 is still not ready to go out of beta. :p [14:26:38] ok [14:27:33] * zhuyifei1999_ thinks most PHP warnings should be turned into exceptions to to force people to write good code [14:27:54] and keep logs clean [14:38:30] zhuyifei1999_: I don't like them either. [14:38:40] IABot runs cleanly most of the time. [14:40:34] :) [14:41:33] zhuyifei1999_: the moment I see a PHP Notice, is the moment I insert a FIXME note and a breakpoint whenever the notice gets triggered. [14:41:49] ok [14:42:35] well, I don't know how PHP debugging works [14:43:13] the most 'complex' debugging I've done on PHP so far it generating a backtrace via gdb [14:43:37] (I mean a PHP stack, not the C stack) [14:49:50] lol [14:50:11] zhuyifei1999_: IABot used to multithread [14:50:25] now? [14:50:52] I mean how is it working now? [14:50:53] No. It used to. I pulled the code from it when the pthreads implementation was vastly different across PHP versions [14:51:13] Now it runs on a single process [14:51:32] pthreads can be different?! I thought it's just libpthread [14:51:42] On PHP it can be. [14:51:44] Apparently [14:52:12] When I tried to run it, it didn't even like the code when I tried to start the bot on PHP 7 [14:52:48] though I wonder, how does PHP protect mutual exclusions in critical regions from race conditioning across multiple pthreads [14:52:51] it only worked on PHP 5 and since I want the bot to run on all PHP versions of 5.4 and up, I ripped the code out and cleaned it up. [14:53:34] python uses the global interpreter lock, which makes python stuck in a big mutex, which sucks [14:53:35] No idea. I do not you need a TS build of PHP or it's going to get ugly. [14:53:39] *know [14:53:55] ok [14:54:39] I may re-implement it at one point and load different code libraries depending on the version of PHP it's running, but that's a low priority right now. [14:54:40] (javascript is also essentially single-threaded afaik) [14:54:47] ok [14:54:55] But it was amazing. I could get it to run on 200 articles/second [14:55:15] ruby also has a GIL, bug in a different name [14:55:17] *but [14:55:50] Debugging a multithreaded application, that's fun. [14:55:56] :p [14:55:59] and PHP is like the only scripting language that I know of can multithread [14:56:57] Isn't Python a scripting language too? [14:57:32] I know betacommand used to write scripts that caused server meltdowns. [14:59:06] Python has GIL [14:59:20] so only one python thread can run at once [15:00:22] (unless the GIL is released by some C code for the duration of running C-only code. any python code will try to grab the GIL) [15:01:39] if you search up in phab phame abut ORES melting down due to 'fake threads', this is what it's talking about [15:02:50] ORES's timing out function simply didn't work when some C code (regex) just refuse to release the GIL [15:03:58] so thinking of time.sleep(some time)? forget it. it's never gonna return. after the sleep it'll just wait for the mutex forever until it melts [15:07:17] (ORES has since moved to a different implementation of a timer that doesn't rely on threads iirc) [15:10:12] ok I gotta take a nap. jetlag is gonna be terrible [15:22:40] The above is why I once re-implemented an entire codebase into async stuff (which can occasionally put off the problem). Python3 has a much better GIL than python 2, though....still has it. [15:23:11] javascript is single threaded, but with node as the backend, it is not. [15:24:11] with ruby you can dodge GIL problems and threading issues by using jruby...because introducing Java never caused any problems. Concurrency is best achieved via async code in those languages. [15:24:21] aka fake concurrency [15:26:25] More correctly node is not, but the JS you run is. :-p You are still stuck with async and promises [15:37:17] bstorm_: 'much better GIL' in Python 3 is arguable :P. yes better in some ways, but worse on some other ways [15:38:49] More correctly node is not, but the JS you run is. <= we could just use cython :) [15:39:12] ;-) [15:39:36] it's just the interpreter being hungry for GIL [15:41:30] by 'async stiff' do you mean the greenlets (i.e. user-land threads)? [15:45:32] * zhuyifei1999_ honestly thinks the kernel just does a better job at context switching than user-land. yes kernel has overhead but a lot of that is memory management. and memory management is much less of a problem in threads where memory is shared anyways [15:46:12] the real terrible overhead is those mutexes :/ [15:49:17] zhuyifei1999_: you're command is beautiful. It's the Latvian Wikipedia worker that's going kookoo [15:49:30] * Cyberpower678 shuts down the worker. [15:50:08] k [15:50:32] you might want to figure out why it's going kookoo :P [15:50:51] Indeed. It could be an issue with the configuration data. [15:51:09] Which would probably suggest the process that's defining it might need work. [15:51:28] k [15:52:33] Wow there 1665 configuration variables. [15:52:51] sounds like mediawiki [15:53:04] (config bloat lol) [15:53:55] Well it's managed in a DB table. It's primarily generated by the UI and used by the bot to determine it's behavior on any given wiki. [15:54:36] lvwiki ones are broken for some reason. [16:02:21] by async, I mean async primitives, event loops and so forth--keep it single-threaded and fake it. I avoid threading except where really important [16:03:24] yeah that's those greenlets [16:10:53] uh, green threads (https://en.wikipedia.org/wiki/Green_threads) is probably a better term [16:47:23] bstorm_: I mentioned it here last week, but filed a task as well – https://phabricator.wikimedia.org/T202757 [16:47:35] I suspect the refactoring of tools-* workboards into sub projects broke my tools' workboard :/ [16:53:29] zhuyifei1999_: yeah, but the greenlet libraries (gevent and friends) suck...the coroutine libs generally work better for interacting with green threads. Same thing on the backend, but.... [16:53:42] IMHO [16:54:46] Krinkle: Ouch... er. [16:55:10] bstorm_: Andre (in PM) says he's looking into it. [16:55:11] fyi :) [16:55:30] Oh good. I was about to look at it blankly and wonder why it was happening :) [17:00:17] Krinkle, this sounds like a problem we've seen twice before [17:00:32] Yep Andre just fixed it, linking to your VE bug. [17:00:46] the trick last time was to go to a search URL under any other workboard, and change the project ID to your one, and save it as the default filter [17:17:37] (03CR) 10Jean-Frédéric: [C: 032] Update IRC channel [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/455092 (owner: 10Lokal Profil) [17:18:29] (03CR) 10Jean-Frédéric: [C: 032] Handle missing commonsTrackerCategory gracefully [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/455023 (https://phabricator.wikimedia.org/T147750) (owner: 10Lokal Profil) [17:18:59] (03Merged) 10jenkins-bot: Update IRC channel [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/455092 (owner: 10Lokal Profil) [17:19:52] (03Merged) 10jenkins-bot: Handle missing commonsTrackerCategory gracefully [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/455023 (https://phabricator.wikimedia.org/T147750) (owner: 10Lokal Profil) [17:20:08] (03CR) 10jenkins-bot: Update IRC channel [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/455092 (owner: 10Lokal Profil) [17:21:47] (03CR) 10jenkins-bot: Handle missing commonsTrackerCategory gracefully [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/455023 (https://phabricator.wikimedia.org/T147750) (owner: 10Lokal Profil) [18:15:50] When I use Quarry, I seem to get a SQL error if I use the `with` common table expression. Is this an undocumented behaviour? [18:16:02] Or am I doing something wrong? [18:58:26] notconfusing: link to the query? [19:58:53] !log shinken Restarting ircecho on shinken-01, messages have been going into /var/log/ircecho/irc-releng.log but not arriving in IRC [19:58:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [20:32:05] !log shinken more details about IRC bot problems in #wikimedia-releng [20:32:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [20:48:54] (03PS1) 10MarcoAurelio: build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 [20:49:24] (03CR) 10jerkins-bot: [V: 04-1] build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [20:52:54] dear CI... [20:56:31] (03CR) 10MarcoAurelio: "This is weird. "composer test" returned no errors locally. I'm not sure why it's failing here." [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [20:58:24] (03PS2) 10MarcoAurelio: build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 [20:58:52] (03CR) 10jerkins-bot: [V: 04-1] build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:01:06] (03PS3) 10MarcoAurelio: build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 [21:01:39] (03CR) 10jerkins-bot: [V: 04-1] build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:03:11] (03CR) 10MarcoAurelio: "recheck" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:03:28] (03CR) 10Framawiki: [C: 032] Implement user prefs and browser notifications [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/427952 (https://phabricator.wikimedia.org/T124625) (owner: 10Framawiki) [21:03:41] (03CR) 10jerkins-bot: [V: 04-1] build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:03:52] (03Merged) 10jenkins-bot: Implement user prefs and browser notifications [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/427952 (https://phabricator.wikimedia.org/T124625) (owner: 10Framawiki) [21:04:28] aarrgghh [21:11:22] (03PS1) 10Framawiki: Use json.jsonify instead of json.dumps [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/455237 [21:11:38] (03CR) 10jerkins-bot: [V: 04-1] Use json.jsonify instead of json.dumps [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/455237 (owner: 10Framawiki) [21:11:52] (03PS2) 10Framawiki: Use flask.jsonify instead of json.dumps [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/455237 [21:12:07] (03CR) 10jerkins-bot: [V: 04-1] Use flask.jsonify instead of json.dumps [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/455237 (owner: 10Framawiki) [21:13:29] (03CR) 10MarcoAurelio: "> minus-x check ." [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:14:46] (03PS3) 10Framawiki: Use flask.jsonify instead of json.dumps [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/455237 [21:15:02] (03CR) 10jerkins-bot: [V: 04-1] Use flask.jsonify instead of json.dumps [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/455237 (owner: 10Framawiki) [21:15:28] (03PS4) 10MarcoAurelio: build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 [21:15:45] lol [21:15:53] (03CR) 10jerkins-bot: [V: 04-1] build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:15:55] Hauskatze: heu [21:15:57] hey [21:16:21] see jenkins message! " 21:03:33 Error: /src/SULWatcher/SULWatcher.sql should not be executable " [21:16:43] (03PS4) 10Framawiki: Use flask.jsonify instead of json.dumps [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/455237 [21:16:45] rxy: yeah, but running composer test locally doesn't return any error [21:16:53] hmm [21:16:54] and composer fix doesn't fix anything either [21:17:04] is this Windows messing with permissions? [21:17:20] windows !? [21:17:30] I'm a Windows user [21:17:38] windows does not support Linux permission [21:17:59] I've been successfully submitting patches to repos with minus-x for several time now [21:18:06] this is all weird [21:18:46] I'll look it [21:21:37] !log quarry deployed 4814d58 (T124625) to quarry-main-01 [21:21:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [21:21:40] T124625: Show desktop notification when a query is done - https://phabricator.wikimedia.org/T124625 [21:22:01] (03CR) 10MarcoAurelio: "I really don't get this. Running composer locally doesn't throw any errors." [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:24:22] -rwxr-xr-x 1 rxy rxy 1308 Aug 25 06:24 SULWatcher.sql [21:26:02] -rw-r--r-- 1 [...] 197121 1344 ago. 24 22:38 SULWatcher.sql [21:26:43] rxy: maybe you could git review -d that patch, run composer and reupload it, if you're on an UNIX system? [21:27:19] I'm on debian-stretch (on Hyper-V) [21:30:44] (03PS5) 10Rxy: build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:31:03] https://gerrit.wikimedia.org/r/#/c/labs/tools/stewardbots/+/455233/5/SULWatcher/SULWatcher.sql "old mode 100755" [21:31:11] "new mode 100644" [21:31:39] :) [21:31:52] https://gerrit.wikimedia.org/r/#/c/labs/tools/stewardbots/+/455233/5/SULWatcher/SULWatcher.sql [21:31:53] (03CR) 10Jforrester: "That'd do it. :-)" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:31:54] yeah [21:32:03] so it was Windows [21:32:20] thanks a bunch rxy [21:32:23] * Hauskatze merges [21:32:53] (03CR) 10MarcoAurelio: [C: 032] "Thanks Rxy and James. Windows messing with file permissions :-)" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:33:30] (03Merged) 10jenkins-bot: build: Add Minus-X [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [21:34:49] zhuyifei1999_: soooo.... [21:35:28] I decided to run the lvwiki worker on my terminal and have stdout and stderr print to my screen. I walk away for 30 minutes and come back to all of my RAM being gone. [21:56:27] Cyberpower678: all your ram, as in locally or on the cloud instance? [21:56:53] locally [21:58:04] does your terminal store all the output in ram? [21:58:25] instead of keeping like last 10k lines? [22:11:04] zhuyifei1999_: yes, until it's closed [22:12:55] well, I guess you gotta fix your terminal then (or use screen/tmux) [22:31:58] zhuyifei1999_: well I found where it failed, but I don't know why it failed. [22:32:12] It's a numerical base converter [22:32:27] what’s that? [22:35:36] It converts numbers from base 10 to a user specified [22:35:37] base [22:35:51] In my case, I'm converting decimals to base 62 numbers [22:36:24] zhuyifei1999_: ^ [22:37:01] and what’s the error message? [22:37:12] PHP Notice: Uninitialized string offset: -7 in /home/IABot/IABot/APII.php on line 1963 [22:37:12] PHP Notice: Uninitialized string offset: -39 in /home/IABot/IABot/APII.php on line 1968 [22:37:13] PHP Notice: Uninitialized string offset: -40 in /home/IABot/IABot/APII.php on line 1968 [22:37:14] PHP Notice: Uninitialized string offset: -50 in /home/IABot/IABot/APII.php on line 1968 [22:37:16] PHP Notice: Uninitialized string offset: -60 in /home/IABot/IABot/APII.php on line 1968 [22:37:18] PHP Notice: Uninitialized string offset: -16 in /home/IABot/IABot/APII.php on line 1968 [22:37:20] PHP Notice: Uninitialized string offset: -1 in /home/IABot/IABot/APII.php on line 1968 [22:37:23] With the last one repeating infinitely [22:37:33] Never, ever has that happened before. [22:38:10] sounds like bad algorithm / invalid data / some PHP weirdness [22:39:05] No. It can only be bad data, as that algorithm has been in use for the 3 years IABot has been running. [22:39:41] That bug has never once happened [22:39:55] hmm [22:40:10] maybe print the hex values of the string? [22:40:49] just to isolate the cause [22:40:55] No. It only happens on lvwiki, so I'm going to insert a FIXME: there and have it break whenever it triggers. [22:42:18] But first I'm going to backtrack. I know which article triggered the problem, so I'm going to attempt to backtrack the bug. [22:43:00] ok [22:44:10] Nacionālo devīžu uzskaitījums [22:44:25] The converter usually kicks in when it deals with WebCite URLs [22:45:07] the 9 character string on those URLs is actually a microsecond epoch [22:46:42] Though it appears the algo received a negative number. [22:47:03] Numbers going in there are supposed to be unix epochs [22:52:08] zhuyifei1999_: yep. It's as I thought, it's getting a negative number [22:53:43] Yep, there it is. https://web.archive.org/web/15060714215545/http://www.coprince-fr.ad/frances/consti.htm [22:54:28] The internal parser is trying to handle that timestamp which reads as July 14, 1506 21:55:45 [22:58:18] (03CR) 10Legoktm: "Because of how Windows handles file permissions, MinusX basically doesn't work on Windows. (there's no exectuable permission ("x") on Wind" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/455233 (owner: 10MarcoAurelio) [22:58:46] Cyberpower678: ok [22:59:36] is that some sort of since-unix-epoch thing? [23:00:38] Well when it converts to a Unix epoch, which is seconds since January 1, 1970 00:00:00, I got a massive negative number. -14625626655 more specifically [23:01:10] ok [23:01:57] The converter, which I didn't design to handle numbers because I never thought I would encounter such outrageous timestamps got spun into a infinite loop trying to convert the number and it kept calling a bad string index. [23:02:00] -1 [23:03:15] The simple solution, convert the number to a positive and then back. The ideal solution, detect these ridiculous archive URLs and replace them with something actually meaningful