[00:00:04] YuviPanda: you rule, thanks [00:00:23] abartov: yw! let me know if the URL thing has issues [00:00:36] since it could possibly be on the proxy's end and might need me to poke at stuff [00:00:47] and you should be good to go for now :) [00:23:44] 6Labs, 6operations: Untangle labs/production roles from labs/instance roles - https://phabricator.wikimedia.org/T119401#1830233 (10Andrew) Sounds good to me. [00:40:56] 6Labs, 10Tool-Labs, 7Database: Client loses connection to database replica and cannot connect to it any further - https://phabricator.wikimedia.org/T119577#1830273 (10Giftpflanze) mysqlsel/db server: Lost connection to MySQL server during query while executing "mysqlsel $dewiki_p "select el_to from e... [00:41:25] YuviPanda: garggh: [00:41:30] https://www.irccloud.com/pastebin/2XrAnYrI/ [00:41:32] YuviPanda: does that qualify as a small program? [00:42:19] abartov: add -mem 4G to your jstart commandline? [00:42:52] abartov: pthread_cancel always means you ran out of memory and the default memory allocation is crap [00:42:56] gifti: looking moment [00:43:11] gifti: hmm, I've no idea how to read tcl but I'll take a look in 10-15mins [00:43:25] abartov: ah! yes, of course. And that error message should have totally clued me in that this was about memory. [00:43:26] * abartov sighs [00:43:46] YuviPanda: just pretend it's Lua :-p [00:43:51] hehe [00:44:23] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Isart was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=208876 edit summary: [00:44:40] abartov: if it works would you mind changing the doc to add the -mem line? [00:44:45] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Antigng was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=208877 edit summary: [00:44:47] * YuviPanda has to go to office for the open data thing [00:44:54] YuviPanda: sure! [00:45:06] abartov: and if it doesn't work let me know :) [00:45:53] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Mardam was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=208880 edit summary: [00:48:21] * YuviPanda goes to the office now [00:51:59] * mutante leaves the office now [10:06:14] 6Labs, 10Tool-Labs, 7Database: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#1830725 (10jcrespo) 3NEW a:3jcrespo [10:08:00] 6Labs, 10Tool-Labs, 7Database: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#1830749 (10jcrespo) [10:08:01] 6Labs, 10Tool-Labs, 7Database: s51053 (tools.jackbot) is abusing resources on labsdbs, throttle his grants - https://phabricator.wikimedia.org/T114559#1830748 (10jcrespo) [10:09:07] 6Labs, 10Tool-Labs, 7Database: Faebot is crashing labsdb1002 - https://phabricator.wikimedia.org/T119604#1830759 (10jcrespo) 3NEW a:3jcrespo [10:09:54] 6Labs, 10Tool-Labs, 7Database: Faebot is crashing labsdb1002 - https://phabricator.wikimedia.org/T119604#1830759 (10jcrespo) 5Open>3Resolved Throttling connections of s51457 to 1 per user. [10:09:55] 6Labs, 10Tool-Labs, 7Database: Certain tools users create multiple long running queries that take all memory from labsdb hosts, slowing it down and potentially crashing (tracking) - https://phabricator.wikimedia.org/T119601#1830725 (10jcrespo) [10:11:02] 10Tool-Labs-tools-Global-user-contributions, 6Stewards-and-global-tools: Global user contributions doesn't work - https://phabricator.wikimedia.org/T119414#1830777 (10jcrespo) This new issue was caused by T119604. It should be fixed now. [10:47:49] admin? [10:48:10] Coren:Grid problems [12:25:54] Coren [12:27:52] Hello, can someone help me? I want to protect a file against overwriting by puppet. Is there a way to do this? [12:34:31] Luke081515: by fixing the puppet manifest, mostly [12:35:35] Luke081515: I don't think there really is another way, other than not running puppet at all... [12:41:35] admin? [12:48:56] UA31_: just ask your question [12:52:44] I have problems with the grid [12:53:00] I can't kill some jobs [12:53:49] I have to go, unfortunately, but please mention the tool, job numbers and the errors you get -- others might be able to help, even if they are not admins [12:54:43] 5202 and 9630 [12:55:58] I can't kill the jobs with qdel [12:56:23] The status is dr in deletion, but the system can't delete them [12:58:30] UA31_: Did you tried to delete them with a) qdel AND b) qdel ? [12:59:25] yes [13:02:01] hm [13:02:57] is the job "perl3"? [13:03:04] The state is Continuous / Deleting [13:03:58] so he trys to delete this job already, but the grid don't finished this yet [13:04:25] same situation at perl41 [13:04:29] exactly [13:26:04] If it's in state dr, that means the grid is workibg on killing it, but failing? Not sure. You could ssh into the host where the job ran/runs and kill it there [13:32:19] UA31_: valhallasw`cloud has the right of it. If your job is really stuck it might need a kill -9 [13:50:00] Thanks, the jobs have been killed [14:33:02] valhallasw`cloud: any idea if I can point a custom domain at a tool? [14:33:17] and if so how? :D I imagine it might need some magic in places? [14:33:41] addshore: There is no support for this, nor is there plans to add it. [14:33:46] okay!1 [14:34:18] To labs with this tool then! ;) [14:45:56] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:38] * Coren looks at that ^ [14:48:00] Hm. Not filesystem. [14:55:48] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 936037 bytes in 2.732 second response time [15:39:54] 6Labs, 10Tool-Labs: Migrate some tools nodes away from labvirt1002, it's getting full - https://phabricator.wikimedia.org/T119399#1831508 (10coren) Only candidate that takes significant space is tools-web-static-02 that I can see. Migrating it. [15:42:09] !log tools migrating tools-web-static-02 to labvirt1010 to free space on labvirt1002 [15:42:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [16:40:11] * qchris foo [17:24:46] 6Labs, 10Tool-Labs: Migrate some tools nodes away from labvirt1002, it's getting full - https://phabricator.wikimedia.org/T119399#1831819 (10coren) a:5coren>3Andrew tools-web-static-02 has been live-migrated away to 1010 for great justice! Most of the big space users left on 1002 are in the deployment-pre... [17:26:24] qchris: bar [17:26:55] Coren: Sorry. A command went nuts :-(( [17:46:57] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:50] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 935458 bytes in 3.729 second response time [19:11:34] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/PhiLiP was created, changed by PhiLiP link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/PhiLiP edit summary: Created page with "{{Tools Access Request |Justification=Build useful helper tools for Wikipedia. |Completed=false |User Name=PhiLiP }}" [19:27:24] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/PhiLiP was modified, changed by Tim Landscheidt link https://wikitech.wikimedia.org/w/index.php?diff=209339 edit summary: [19:28:00] 6Labs, 10wikitech.wikimedia.org: "Edit with form" missing on a Tools access request page - https://phabricator.wikimedia.org/T118136#1832216 (10scfc) 5Resolved>3Open Happened again with https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/PhiLiP. [20:04:13] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/PetrohsW was created, changed by PetrohsW link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/PetrohsW edit summary: Created page with "{{Tools Access Request |Justification=Generar herramientas para emplear durante los editatones. Generate tools for use during the edit-a-ton |Completed=false |User Name=Pet..." [20:21:56] 6Labs, 10wikitech.wikimedia.org: "Edit with form" missing on a Tools access request page - https://phabricator.wikimedia.org/T118136#1832350 (10Luke081515) At the first time, I clicked at that link, I can't see the "Edit form" button. Now I can see him, but I don't know why... [20:49:05] A labs admin on? [20:50:23] White_Master: What's up? [20:50:42] Coren, wikitech.wikimedia.org is down [20:50:48] 6Labs, 10Labs-Infrastructure, 6operations, 5Patch-For-Review: Set up LVS for labs dns recursors - https://phabricator.wikimedia.org/T119660#1832410 (10Andrew) 3NEW a:3Andrew [20:51:02] White_Master: It works for me; what symptom are you experiencing? [20:51:12] 6Labs, 10Labs-Infrastructure, 6operations, 5Patch-For-Review: Set up LVS for labs dns recursors - https://phabricator.wikimedia.org/T119660#1832417 (10chasemp) p:5Triage>3Normal [20:53:22] Coren, When I log in wikitech presents error. But I went to log in and already solved. [20:54:05] I'm glad I could help. :-) [20:54:45] Coren, thanks anymore :D [20:55:01] 6Labs, 7Tracking: Create a labs project for Wikimedia Venezuela' Engineering Technology Committee - https://phabricator.wikimedia.org/T119661#1832420 (10White-Master) 3NEW [20:55:10] Huh, fast [21:04:41] 6Labs, 6operations, 5Patch-For-Review, 7Puppet: Self hosted puppetmaster is broken - https://phabricator.wikimedia.org/T119541#1832441 (10MaxSem) Disregard the above patch :P [22:17:08] 6Labs, 6operations, 5Patch-For-Review, 7Puppet: Self hosted puppetmaster is broken - https://phabricator.wikimedia.org/T119541#1832634 (10MaxSem) [22:18:23] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Edoderoo was created, changed by Edoderoo link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Edoderoo edit summary: Created page with "{{Tools Access Request |Justification=Firstly, to run the script that updates regularly (monthly) the statistics of the Dutch sysops activity, to trace inactive sysops. See h..." [22:34:00] 6Labs, 10Tool-Labs: Migrate some tools nodes away from labvirt1002, it's getting full - https://phabricator.wikimedia.org/T119399#1832689 (10Andrew) 5Open>3Resolved ok, moved -- labvirt1002 now has 10% free disk space, I'm happy and icinga is happy. [22:50:39] 6Labs, 10Tool-Labs: Silence sudo warnings about unresolvable host - https://phabricator.wikimedia.org/T119672#1832715 (10scfc) 3NEW [23:12:47] 6Labs, 6operations, 5Patch-For-Review: Investigate whether to use Debian's jessie-backports - https://phabricator.wikimedia.org/T107507#1832767 (10BBlack) Some systems already had the jessie backports repo in sources.list, I think it varies depending on the install date of the system. Will probably have to... [23:22:38] 6Labs, 6operations, 5Patch-For-Review: Investigate whether to use Debian's jessie-backports - https://phabricator.wikimedia.org/T107507#1832776 (10hashar) I very welcome the decision to add jessie-backports. That has hit me when setting up Nodepool, the POC had jessie-backports but when we rebuild it we had... [23:25:17] 6Labs, 10wikitech.wikimedia.org: "Edit with form" missing on a Tools access request page - https://phabricator.wikimedia.org/T118136#1832781 (10Krenair) Shall we leave this open and wait for someone to come up with reliable steps to reproduce the issue? [23:48:19] 6Labs, 10Tool-Labs: Rounding and missing units in VMEM values on http://tools.wmflabs.org/?status create misleading values - https://phabricator.wikimedia.org/T119680#1832833 (10liangent) 3NEW