[09:46:48] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL 30.00% of data above the critical threshold [0.0] [10:33:32] the tool "checkwiki" is broken: can someone restart it? it's currently running both "delay" and "munch" tasks in parallel, which doesn't make sense, wasting resurces for hundreds of hours now [10:35:12] https://tools.wmflabs.org/?status [10:37:24] harry3: I have no clue what those jobs are supposed to do, so I don't feel comfortable killing them [10:38:00] Please ask Magioladitis, Tim Landscheidt or Bgwhite [10:38:10] who are the actual developers of the tool [10:38:11] valhallasw`cloud: any log that can show where they got stuck? [10:38:22] again, check with the developers [10:38:32] ok [10:56:47] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK Less than 1.00% above the threshold [0.0] [10:58:19] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mahairod was created, changed by Mahairod link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Mahairod edit summary: Created page with "{{Tools Access Request |Justification=I plan to deploy webservice which will be a helpful tool in making wiki-links automatically. This service converts text fragment from wik..." [11:03:25] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mahairod was modified, changed by Mahairod link https://wikitech.wikimedia.org/w/index.php?diff=172821 edit summary: [13:05:01] 6Labs, 10Tool-Labs, 5Patch-For-Review: Return human-readable 404 for non-existing projects - https://phabricator.wikimedia.org/T85738#1501469 (10valhallasw) 5Open>3Resolved a:3valhallasw [13:06:14] 10Tool-Labs-tools-Other: Citations button causes Bad Request error - https://phabricator.wikimedia.org/T107649#1501471 (10valhallasw) [13:09:04] 6Labs, 10Tool-Labs: 'webservice not available' message no longer shown - https://phabricator.wikimedia.org/T104870#1501472 (10valhallasw) [13:09:06] 6Labs, 10Tool-Labs, 5Patch-For-Review: Return human-readable 404 for non-existing projects - https://phabricator.wikimedia.org/T85738#1501473 (10valhallasw) [13:09:50] 6Labs, 10Tool-Labs: Extend 'tool not found' 404 page - https://phabricator.wikimedia.org/T107697#1501478 (10valhallasw) 3NEW [13:10:28] 6Labs, 10Tool-Labs: Install composer on tools-bation - https://phabricator.wikimedia.org/T104789#1501491 (10valhallasw) [13:10:30] 6Labs, 10Tool-Labs, 7Tracking: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#1501490 (10valhallasw) [13:10:50] 6Labs, 10Tool-Labs: Install composer on tools-bation - https://phabricator.wikimedia.org/T104789#1501494 (10valhallasw) p:5Triage>3Normal [13:11:18] 6Labs, 10Tool-Labs: Something went wrong with redis 2015-06-04 to 2015-06-05 - https://phabricator.wikimedia.org/T101514#1501495 (10valhallasw) p:5Triage>3Low [13:11:29] 6Labs, 10Tool-Labs: 'new webproxy' checklist - https://phabricator.wikimedia.org/T104768#1501497 (10valhallasw) p:5Triage>3Normal [13:11:34] 6Labs, 10Tool-Labs: can't login to tools-shadow-01 - https://phabricator.wikimedia.org/T104781#1501498 (10valhallasw) p:5Triage>3High [13:12:13] 6Labs, 10Tool-Labs: Set up A-based SPF for tools.wmflabs.org - https://phabricator.wikimedia.org/T104733#1501506 (10valhallasw) p:5Triage>3Normal [13:12:22] 6Labs, 10Tool-Labs: shinken is too "volatile" and imprecise to be of use - https://phabricator.wikimedia.org/T107297#1501507 (10valhallasw) p:5Triage>3High [13:14:39] 6Labs, 10Tool-Labs: /srv on tools-static-01 running out of inodes - https://phabricator.wikimedia.org/T107657#1501513 (10valhallasw) p:5Triage>3High If I understand correctly, this is currently breaking cdnjs updates (or even just git cloneing it? not sure)? If not, please reduce the priority to 'normal'. [13:15:06] 6Labs, 10Tool-Labs, 7Tracking: Missing Toolserver features in Tools (tracking) - https://phabricator.wikimedia.org/T60791#1501515 (10valhallasw) p:5Triage>3Normal [13:18:14] 6Labs, 10Tool-Labs: Permission issues and/or failure to load Ruby environment on trusty - https://phabricator.wikimedia.org/T106170#1501522 (10valhallasw) p:5Triage>3Normal [13:18:50] 6Labs, 10Tool-Labs, 5Patch-For-Review: new labs host sends out "mpt raid status change" emails - https://phabricator.wikimedia.org/T104779#1501524 (10valhallasw) p:5Triage>3Normal [13:19:16] 6Labs, 10Tool-Labs, 5Patch-For-Review: new labs host sends out "mpt raid status change" emails - https://phabricator.wikimedia.org/T104779#1501525 (10valhallasw) 5Open>3stalled Waiting for: >>! In T104779#1475271, @scfc wrote: > (For this to be fixed, after merging the patch new images need to be create... [13:20:29] 6Labs, 10Tool-Labs: Update python-wikitools on Tool Labs - https://phabricator.wikimedia.org/T102862#1501527 (10valhallasw) >>! In T102862#1375938, @scfc wrote: > As I'm [[https://bugs.debian.org/594130|packaging]] this for Debian, assigning to myself. Any updates on this? [13:24:47] 10Tool-Labs-tools-Other, 7Epic: Convert all Labs tools to use cdnjs for static libraries - https://phabricator.wikimedia.org/T103934#1501529 (10Ricordisamoa) [13:35:23] 6Labs, 10Tool-Labs: Tool Labs: jsub starts multiple instances of tasks declared as "once" - https://phabricator.wikimedia.org/T62862#1501530 (10valhallasw) p:5High>3Normal [13:35:35] 6Labs, 10Tool-Labs, 3Labs-Sprint-100: Clean up huge logs on toollabs - https://phabricator.wikimedia.org/T98652#1501532 (10valhallasw) p:5High>3Low [14:14:06] duh, phabricator mails are so hard to detect with a mobile mail client :\ [14:33:59] 10Tool-Labs-tools-Database-Queries, 6Phabricator: Archive Tool-Labs-tools-Database-Queries project - https://phabricator.wikimedia.org/T107699#1501553 (10Bugreporter) 3NEW [15:09:31] (03PS1) 10Sitic: Log more ORES error information [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228663 [15:09:33] (03PS1) 10Sitic: Fix link to user page and contribs [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228664 (https://phabricator.wikimedia.org/T107681) [15:10:07] (03CR) 10Sitic: [C: 032 V: 032] Log more ORES error information [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228663 (owner: 10Sitic) [15:10:18] (03CR) 10Sitic: [C: 032 V: 032] Fix link to user page and contribs [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228664 (https://phabricator.wikimedia.org/T107681) (owner: 10Sitic) [15:13:01] (03PS1) 10Sitic: Fix ORES error handling [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228665 [15:13:27] (03CR) 10Sitic: [C: 032 V: 032] Fix ORES error handling [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/228665 (owner: 10Sitic) [15:41:50] 10Tool-Labs-tools-Database-Queries, 6Phabricator: Archive Tool-Labs-tools-Database-Queries project - https://phabricator.wikimedia.org/T107699#1501600 (10scfc) https://quarry.wmflabs.org/ has a different scope than #Tool-Labs-tools-Database-Queries. The former requires that the user knows the SQL query string... [15:45:02] 6Labs, 10Tool-Labs: Tool Labs: Enable php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1501601 (10valhallasw) I'm puzzled by this. Almost all packages have their respective config files in conf.d -- mcrypt.ini is the only exception. In the packages, I can't figure out why any others would be in... [16:11:48] 6Labs, 10Tool-Labs, 5Patch-For-Review: Add python3 equivalent packages for existing python packages - https://phabricator.wikimedia.org/T101646#1501622 (10scfc) I don't like the goal of this task nor the patch :-). I prefer: "If there is demand, then do something, otherwise not." If noone requests the Pyth... [16:13:12] 6Labs, 10Tool-Labs, 5Patch-For-Review: Add python3 equivalent packages for existing python packages - https://phabricator.wikimedia.org/T101646#1501623 (10scfc) (Which still leaves the genpp.py script useful for knowing what packages to add for a specific Python module; but I don't want to build an semi-auto... [16:21:06] 6Labs, 10Tool-Labs, 5Patch-For-Review: Add python3 equivalent packages for existing python packages - https://phabricator.wikimedia.org/T101646#1501627 (10Matanya) I was the original requestor on IRC. there are packages i need and tracking them down one by one is somewhat annoying. [16:55:32] 6Labs, 10Tool-Labs, 7Database: Lost connection to MySQL server during query - https://phabricator.wikimedia.org/T107029#1501631 (10Giftpflanze) 5Open>3Resolved a:3Giftpflanze error: mysqlreceive/db server: Lost connection to MySQL server during query while executing "mysqlreceive [set db [get_... [17:08:52] 6Labs, 10Tool-Labs, 5Patch-For-Review: Add python3 equivalent packages for existing python packages - https://phabricator.wikimedia.org/T101646#1501634 (10valhallasw) >>! In T101646#1501622, @scfc wrote: > Regarding the patch: > - The patch assumes that there is a 1:1 relation between Python 2 and 3 packages... [18:48:52] is this a known problem? when ssh-ing to tool labs, the connection hangs (but does not break, as it usually does) [18:50:20] um, i mean, after some time, it hangs [19:05:29] gifti: not that I know. Please paste the output of ssh -v -v tools.wmflabs.org? [19:06:58] ok, will try [19:25:14] 6Labs, 10Tool-Labs: Tool Labs: Enable php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1501724 (10Danmichaelo) The best would of course be to fix this upstream, but not sure how hard that is.. There is a bug here (also listing some other modules like imap): https://bugs.launchpad.net/ubuntu/+s... [19:32:06] valhallasw`cloud: https://phabricator.wikimedia.org/P1811 (but i guess that's not very useful to the problem?) [19:38:25] gifti: oh, wait, it doesn't hang during connections, but later [19:38:36] could be a connection loss [19:38:36] yeah [19:38:48] when I switch locations, my ssh connections also just hang [19:39:39] hm, could be, i have only dorm wifi [19:44:49] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL 30.00% of data above the critical threshold [0.0] [19:46:31] 6Labs, 10Tool-Labs: Tool Labs: Enable php5-mcrypt on Trusty - https://phabricator.wikimedia.org/T97857#1501734 (10valhallasw) The first is the change that caused this (requiring php5enmod). The second indeed mentions the following: https://bugs.launchpad.net/ubuntu/+source/php-mcrypt/+bug/1243568/comments/22... [19:49:05] 6Labs, 10Tool-Labs: lighttpd does not correctly close connections (CLOSE_WAIT) - https://phabricator.wikimedia.org/T104799#1501736 (10Danmichaelo) Note that I enabled https://tools.wmflabs.org/croptool/server-statistics . When the same problem occured six days ago, the status page was still accessible and show... [20:08:54] so I'm trying to debug https://phabricator.wikimedia.org/T107145 [20:08:59] > Warning: Unable to read "dist/themes/apex/images/icons/quotesAdd-ltr.svg" file (Error code: EMFILE). Use --force to continue. [20:09:10] EMFILE means you have too many files open apparently [20:09:15] ulimit -n -> 1024 [20:09:31] does that mean the account can only have 1024 files open across all processes? or just that one process? [20:10:47] valhallasw`cloud, YuviPanda: ^? [20:11:49] all processes, I think? not sure... [20:21:31] legoktm I think maybe you can raise it yourself? [20:22:00] oh, how? [20:23:27] legoktm I have terrible internet, can't reach manpage nor google. google? [20:23:29] stupid lag [20:23:33] :( [20:23:35] ok [20:23:35] iowa sucks etc [20:23:41] lol [20:23:43] cornfields? [20:23:54] yes [20:24:02] no [20:24:07] ^ not me. :P [20:24:10] yes, cornfields [20:24:25] evasive cornfields. Took us 5 tries to get a picture of one [20:24:47] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1409 is OK Less than 1.00% above the threshold [0.0] [20:32:02] YuviPanda: apparently it's as simple as "ulimit -n 2048" [20:32:37] yeah that's how I remember it to be [20:37:31] 6Labs, 10Tool-Labs: Encoding issue with jsub - https://phabricator.wikimedia.org/T107713#1501786 (10Huji) 3NEW [20:39:17] 6Labs, 10Tool-Labs: Encoding issue with jsub - https://phabricator.wikimedia.org/T107713#1501794 (10Huji) [20:40:16] 6Labs, 10Tool-Labs: Encoding issue with jsub - https://phabricator.wikimedia.org/T107713#1501797 (10valhallasw) [20:41:44] valhallasw`cloud: thanks for the dup. I was sure this can't be a new bug, but I couldn't find the original eitehr [20:55:35] would there be an issue if I use crontab for a program that is NOT being called through jsub? [21:40:39] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mahairod was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=172840 edit summary: [21:43:56] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Saqib was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=172845 edit summary: [22:42:47] 6Labs, 10Analytics, 10Labs-Infrastructure, 5Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1501986 (10yuvipanda) Pinging @ArielGlenn to ask about rsync - where are the rsyncs for the dumps? [22:45:47] PROBLEM - Puppet failure on tools-webgrid-lighttpd-1409 is CRITICAL 22.22% of data above the critical threshold [0.0] [22:46:13] 6Labs, 10Analytics, 10Labs-Infrastructure, 5Patch-For-Review: Set up cron job on labstore to rsync data from stat* boxes into labs. - https://phabricator.wikimedia.org/T107576#1501988 (10yuvipanda) @Ottomata looks like in https://gerrit.wikimedia.org/r/#/c/215585/11/modules/dataset/files/labs/labs-rsync-cr... [23:01:04] legoktm valhallasw`cloud i'm trying to refactor quarry to get rid of the global connection object.... [23:01:07] not sure if I can do that at all [23:01:13] so now I depend on g.conn everywhere [23:01:16] which sounds terrible [23:23:28] Guys what is the easiest way to deal with the timestamps from Labs databases? They are in UNIX format (20150802...) and I want to run time differences on them. [23:24:24] They are *not* in UNIX timestamp format actually [23:30:01] yes they're in MW timeformat [23:30:12] I think it's just YYYYMMDDHHmmSS? [23:30:14] or something ike tht [23:30:56] that is correct. I was wondering if there is an easy way to do date compares in that format YuviPanda (right now I am writing a complex query to do that) [23:31:39] not sure huji. I think people sometimes use substrings? look at the queries in quarry.wmflabs.org - I think some people use date comparisons in some of their queires [23:32:38] Yeah, substring is what I am doing right now [23:33:53] huji: I thinkt he MySQL 'date' functions work on it too [23:35:29] Thanks YuviPanda