[00:00:00] how to sort shinken by duration [00:00:07] there are more fails but some days old [00:00:39] ah, found the tab for just tools [00:01:55] !log tools fixing puppet runs on tools-webgrid-* via salt [00:01:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [00:01:59] what the heck salt [00:02:08] different results on different runs again [00:04:28] shinken-wm: can haz recoveries [00:14:21] RECOVERY - Puppet failure on tools-webgrid-generic-1404 is OK Less than 1.00% above the threshold [0.0] [00:14:25] why do i see them all running puppet fine but shinken still... [00:14:26] heh [00:14:33] more of that ..come on [00:15:59] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1406 is OK Less than 1.00% above the threshold [0.0] [00:19:22] RECOVERY - Puppet failure on tools-webgrid-generic-1403 is OK Less than 1.00% above the threshold [0.0] [00:20:56] RECOVERY - Puppet failure on tools-exec-1410 is OK Less than 1.00% above the threshold [0.0] [00:21:39] RECOVERY - Puppet failure on tools-exec-1406 is OK Less than 1.00% above the threshold [0.0] [00:23:55] 6Labs, 10Tool-Labs: Cron job updating file running mysql fails with no output - https://phabricator.wikimedia.org/T105565#1446885 (10Sitic) The problem is that piping won't work combined with jsub, try: ``` jsub -N autopatrolled_candidates -once -quiet -mem 1g -i ~/autopatrolled_candidates.sql -o ~/public_html... [00:28:57] ^ there, i fixed the puppet issues [00:30:27] Hello [00:30:58] There seems to be a problem with XTools [00:40:08] Cyberpower678? [00:40:33] mk03, who do I have the pleasure of addressing? [00:40:47] There seems to be a problem with XTools [00:40:55] Specifically, the Article History tool isn't working [00:41:11] It still has the "estimated uptime in 1 week" message from a few weeks ago [00:41:36] It couldn't have been a few weeks already. [00:41:50] Anyways, we're still working to debug it. [00:42:16] I truely am sorry. :/ [00:42:17] I see [00:42:24] Estimated uptime? [00:42:41] mk03, I'm making progress, so hopefully by the end of tomorrow. [00:42:49] I see. Thank you. [00:42:52] Hopefully. [00:42:58] I can't make any promises. [00:43:01] By the way, can I make a suggestion to the "topedits" tool? [00:43:07] Sure. [00:44:21] I know that some tools (such as XTools' own article created list) have the option to list either only redirects, all, or exclude redirects [00:44:31] Could such an option be added to the edit counter and top edits? [00:45:53] (03PS1) 10Sitic: Add checkboxes to multiple select lists [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224207 (https://phabricator.wikimedia.org/T100157) [00:47:24] I'm sorry, I'm not familiar with tech [00:47:25] What was that? [00:48:44] (03CR) 10Sitic: [C: 032 V: 032] Add checkboxes to multiple select lists [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224207 (https://phabricator.wikimedia.org/T100157) (owner: 10Sitic) [00:48:51] (03PS1) 10John F. Lewis: add dummy lists.wm.o key [labs/private] - 10https://gerrit.wikimedia.org/r/224208 [00:48:57] I see [00:49:09] So the suggestion has been added to the list of suggestions, right? [00:49:11] (03PS1) 10Sitic: Stop click event propagation for [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224209 [00:49:30] (03CR) 10Sitic: [C: 032 V: 032] Stop click event propagation for [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224209 (owner: 10Sitic) [00:51:13] RECOVERY - Puppet failure on tools-bastion-01 is OK Less than 1.00% above the threshold [0.0] [00:53:24] (03CR) 10John F. Lewis: "needed for https://gerrit.wikimedia.org/r/#/c/224210/ (to have a successful puppet run)" [labs/private] - 10https://gerrit.wikimedia.org/r/224208 (owner: 10John F. Lewis) [01:01:40] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1407 is OK Less than 1.00% above the threshold [0.0] [01:03:03] (03PS1) 10BBlack: copy files/ to modules/secret/secrets, like prod [labs/private] - 10https://gerrit.wikimedia.org/r/224211 [01:03:41] (03CR) 10BBlack: [C: 032 V: 032] copy files/ to modules/secret/secrets, like prod [labs/private] - 10https://gerrit.wikimedia.org/r/224211 (owner: 10BBlack) [01:06:04] (03PS2) 10John F. Lewis: add dummy lists.wm.o key [labs/private] - 10https://gerrit.wikimedia.org/r/224208 [01:12:55] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1408 is OK Less than 1.00% above the threshold [0.0] [01:13:29] RECOVERY - Puppet failure on tools-exec-1409 is OK Less than 1.00% above the threshold [0.0] [01:14:47] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK Less than 1.00% above the threshold [0.0] [01:20:00] RECOVERY - Puppet failure on tools-exec-1402 is OK Less than 1.00% above the threshold [0.0] [01:20:10] RECOVERY - Puppet failure on tools-exec-1407 is OK Less than 1.00% above the threshold [0.0] [01:20:38] RECOVERY - Puppet failure on tools-exec-1405 is OK Less than 1.00% above the threshold [0.0] [01:21:22] andrewbogott_afk: ^ that's still me fixing them [01:21:25] RECOVERY - Puppet failure on tools-exec-catscan is OK Less than 1.00% above the threshold [0.0] [01:21:34] salt doesnt work :p [01:22:31] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1410 is OK Less than 1.00% above the threshold [0.0] [01:22:55] RECOVERY - Puppet failure on tools-exec-1403 is OK Less than 1.00% above the threshold [0.0] [01:23:13] RECOVERY - Puppet failure on tools-exec-1404 is OK Less than 1.00% above the threshold [0.0] [01:24:49] RECOVERY - Puppet failure on tools-webgrid-generic-1401 is OK Less than 1.00% above the threshold [0.0] [01:27:28] RECOVERY - Puppet failure on tools-webgrid-generic-1402 is OK Less than 1.00% above the threshold [0.0] [01:28:22] RECOVERY - Puppet failure on tools-exec-1408 is OK Less than 1.00% above the threshold [0.0] [01:29:58] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1405 is OK Less than 1.00% above the threshold [0.0] [01:31:58] RECOVERY - Puppet failure on tools-bastion-02 is OK Less than 1.00% above the threshold [0.0] [01:34:43] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1401 is OK Less than 1.00% above the threshold [0.0] [01:38:38] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1402 is OK Less than 1.00% above the threshold [0.0] [01:39:18] RECOVERY - Puppet failure on tools-exec-1401 is OK Less than 1.00% above the threshold [0.0] [01:47:00] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1403 is OK Less than 1.00% above the threshold [0.0] [01:48:21] Hello all, where do I need to look to learn about scheduling jobs on Labs? [01:52:01] I just want to get a particular script to run once a week on a certain day, is all. [08:28:02] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1447097 (10Ricordisamoa) [08:43:53] 10Quarry: Add a list/table of popular queries - https://phabricator.wikimedia.org/T71266#1447112 (10Edgars2007) @He7d3r can think of some reasons, why also this won't be good (recent ones probably have less page views; if the link to query is on WP:VPT or similar help page just for asking help etc.). But this is... [09:30:59] 10Quarry: Add a list/table of popular queries - https://phabricator.wikimedia.org/T71266#1447161 (10yuvipanda) We could have a 'featured query' section that is manually curated [09:36:26] 10Quarry: Add a list/table of popular queries - https://phabricator.wikimedia.org/T71266#1447176 (10Edgars2007) Yeah, that would be nice [09:41:19] 6Labs, 10Tool-Labs: Cron job updating file running mysql fails with no output - https://phabricator.wikimedia.org/T105565#1447179 (10Rillke) 5Open>3Invalid a:3Rillke >>! In T105565#1446885, @Sitic wrote: > The problem is that piping won't work combined with jsub, try: > ``` > jsub -N autopatrolled_candid... [09:45:17] 10Quarry: Add a list/table of popular queries - https://phabricator.wikimedia.org/T71266#1447192 (10eranroz) Other possible metrics for featured queries: * links to queries from externallinks table * most stared queries (manually - similar in some sense to featured query yuvi suggested above) [09:57:48] 10Quarry: Add a list/table of popular queries - https://phabricator.wikimedia.org/T71266#1447202 (10Edgars2007) >>! In T71266#1447192, @eranroz wrote: > * links to queries from externallinks table If I put in some Wikipedia link to query like [[quarry:query/896]], this won't be recorded in externallinks table.... [10:07:18] 10Quarry: Add a list/table of popular queries - https://phabricator.wikimedia.org/T71266#1447207 (10eranroz) ``` select iwl_title, count(*) from iwlinks where iwl_prefix ='quarry' group by iwl_title order by count(*) desc limit 5; ``` but unfortunately even in enwiki the most "popular" query have count of 3 :) [10:28:00] Coren - via NFS there is very long replag time of 25 minutes operating the tasks [10:28:08] right now [10:28:39] this was even shortly before the last big server crash [10:33:09] Coren YuviPanda - is everythings okay? [10:40:31] 6Labs, 10Tool-Labs: Lost connection to MySQL server during query when executing large query's - https://phabricator.wikimedia.org/T105468#1447241 (10Steinsplitter) The new query does not work for me :-(. I simply don't have time to change my script every 5 minutes because every 5 minutes something changed on... [10:41:02] 6Labs, 10Tool-Labs: Lost connection to MySQL server during query when executing large query's - https://phabricator.wikimedia.org/T105468#1447242 (10Steinsplitter) 5Open>3Invalid a:3Steinsplitter [10:41:18] 6Labs, 10Tool-Labs: Lost connection to MySQL server during query when executing large query's - https://phabricator.wikimedia.org/T105468#1444393 (10Steinsplitter) a:5Steinsplitter>3None [10:42:18] 6Labs, 10Tool-Labs: Cron job updating file running mysql fails with no output - https://phabricator.wikimedia.org/T105565#1447247 (10Rillke) mangled characters in output (?????) instead of user names --> I ended up writing a wrapper shell script due to T60784 ``` #! /bin/bash export LANG=en_US.UTF-8 export P... [10:59:44] steinsplitter - i cannot execute any tasks due to database replag for 45 minutes now. do you know anything about [11:00:08] doctaxon: no [11:00:12] sorry :( [11:00:20] coren and YuviPanda do not reply here [11:01:06] 6Labs, 7Tracking: Instance for running OpenOCR (OCR as a service) in a Docker container - https://phabricator.wikimedia.org/T105584#1447253 (10Ijon) 3NEW [11:05:28] Steinsplitter but there was the same replag shortly before the big nfs server crash last time [11:33:13] 6Labs, 10Tool-Labs: Replications lag on multiple databases - https://phabricator.wikimedia.org/T105585#1447265 (10Steinsplitter) 3NEW [11:34:41] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447273 (10Steinsplitter) [11:43:41] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447296 (10jcrespo) labsdb1002 crashed yesterday at 2:38 UTC due to excessive memory usage. I've restarted replication, there is not much to do now but wait. [11:52:05] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447299 (10doctaxon) ah, replag has been corrected now. It's over. Thanks! [11:55:22] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447303 (10doctaxon) but yesterday 2:38 UTC, it was running still till 10:15 UTC today [12:04:36] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447306 (10jcrespo) 5Open>3Resolved a:3jcrespo db was running, replication wasn't. Replication being stopped for 8 hours is consistent with the effects seen. Does that answer your question?... [12:06:06] 6Labs, 10Tool-Labs: Lost connection to MySQL server during query when executing large query's - https://phabricator.wikimedia.org/T105468#1447312 (10Rillke) >>! In T105468#1447241, @Steinsplitter wrote: > I simply don't have time to change my script every 5 minutes because every 5 minutes something changed on... [12:13:31] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447314 (10doctaxon) 5Resolved>3Open Jcrespo: Why could my tasks work till 10:15 AM UTC today, if you say, it crashed 2:38 UTC yesterday [12:17:41] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447319 (10jcrespo) 5Open>3Resolved When mysql crashes, mysqld_safe, the watchdog process restarts mysql automatically. To avoid replication errors, replication is configured to not restart au... [14:12:27] 6Labs, 10Tool-Labs, 10Wikimania-Hackathon-2015, 15User-Bd808-Test: Create template PHP application for use on Tool Labs based on Slim, Twig and Wikimedia libraries - https://phabricator.wikimedia.org/T90092#1447356 (10Ricordisamoa) [14:16:52] 6Labs, 10Tool-Labs, 7I18n: Internationalize Tool Labs' homepage - https://phabricator.wikimedia.org/T105590#1447365 (10Ricordisamoa) 3NEW [15:03:09] (03PS1) 10Sitic: Fix bug for logevents with spaces in title [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224233 [15:03:42] (03CR) 10Sitic: [C: 032 V: 032] Fix bug for logevents with spaces in title [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/224233 (owner: 10Sitic) [15:07:31] doctaxon: We do, when we're around. [15:07:55] hihi, yes it's okay [15:08:52] Coren, how could this problem not to be noticed for more than 24 hours? [15:10:17] doctaxon: I'm sorry - what problem? [15:10:53] Coren: T105585 [15:11:05] * Coren goes look. [15:15:07] I'm not sure I understand what you mean by this? From what I can tell in the ticket, the issue was fixed in some 8 hours by Jaime and - judging from the time - early on a Saturday morning. That doesn't seem so bad to me? [17:38:59] Coren: "the issue was fixed in some 8 hours | by Jaime [17:39:38] but 2 UTC yesterday to 10 UTC this day are 32 hours [17:43:50] oh Coren, I just heard, the problem is only solved for dewiki_p but not commonswiki_p [17:44:01] there is still replag [17:49:56] 6Labs, 10Tool-Labs: Replication lag on multiple databases on tool-labs - https://phabricator.wikimedia.org/T105585#1447514 (10doctaxon) 5Resolved>3Open this task is resolved only for dewiki_p but not for commonswiki_p , there is still replication lag [18:18:09] hi guys [18:18:22] anyone can check why I can't insert data? [18:18:27] into the databases. [18:18:55] YuviPanda: there seems to be an error of permissions with my user or sth wrong with the tables. [18:20:23] marmick: which db? maybe itis the "_" issue [18:20:36] what "_"? [18:20:50] Steinsplitter: I tried many tables in my db. [18:21:00] u3532__. which is created in every server [18:21:16] "INSERT INTO u3532__.'+lang+'_page_titlesdict (page_title,page_id) VALUES (%s, %s)" [18:21:21] __ schould work _ no longer (for me at least) [18:21:30] substitute lang for itwiki, iswiki, cawiki, etc. [18:22:20] I just checked again: [18:22:35] I can with the terminal, but I cannot with the python connector. [18:22:44] connecting to cawiki.labs. [18:22:56] cawiki.labsdb. [18:24:52] I opened a ticket, with no answer.... https://phabricator.wikimedia.org/T105503 [19:21:29] 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1447561 (10Huji) Andrew, can you please change the quota such that we have 1 (one) public IP available to us? We need a public wiki so we can test Gadgets on it, implement various configurations, etc. [19:23:27] 6Labs: Create a project for Persian Wikipedia technical users - https://phabricator.wikimedia.org/T105356#1447562 (10Huji) Actually, never mind, it seems I can get what I want just using a proxy. [20:10:07] 6Labs, 10Labs-Infrastructure: wikidatawiki_p (and maybe others) not available on Labs - https://phabricator.wikimedia.org/T105607#1447611 (10Magnus) 3NEW [20:17:10] 6Labs, 10Labs-Infrastructure: wikidatawiki_p (and maybe others) not available on Labs - https://phabricator.wikimedia.org/T105607#1447631 (10jcrespo) There has been corruption on labdb1002, probably caused by a crash after an OOM due to excessive memory usage: https://wikitech.wikimedia.org/wiki/Server_Admin_L... [20:22:34] Change on 12www.mediawiki.org a page Wikimedia Labs was modified, changed by BDavis (WMF) link https://www.mediawiki.org/w/index.php?diff=1746759 edit summary: [+83] Links and fixups for team infobox [20:24:38] 6Labs, 10Labs-Infrastructure: wikidatawiki_p (and maybe others) not available on Labs - https://phabricator.wikimedia.org/T105607#1447635 (10jcrespo) labsdb1002 should be up again, but I cannot assure the stability of the db, first because there could be more corruption on user tables, which for the most part... [21:46:01] so what's the best new tool on labs in the past year or so? [23:52:45] 6Labs, 7Database: Tables corrupted or impossible to work with them - https://phabricator.wikimedia.org/T105503#1447765 (10marcmiquel) A while ago I couldn't connect... Traceback (most recent call last): File "cira_filter_db.py", line 894, in mysql_con = mdb.connect(lang + '.labsdb', 'u3532', 'h...