[00:18:34] hmm i have created a jessie node but it doesn't have the same (facter) facts as trusty nodes. specifically it's missing ec2_*. known issue? [00:59:36] (i changed from ec2_instance_id to ec2id to avoid the problem) [01:06:24] Are there any automated edit counters that work for en.wikipedia.beta.wmflabs.org content? [01:06:42] jgage, maybe try replying to andrew on labs-l about that? [01:06:53] quiddity, automated edit counters? [01:07:05] like, count all the edits x user has made to y wiki? [01:08:32] Krenair, yup. I'm trying to determine if https://phabricator.wikimedia.org/T63887 is fixed (it's in "product review"). Normally I would just compare the Navpopups edit-count to the page-by-page usercontributions, but Navpopups isn't giving me that information at beta. and I'm not sure where else to find it. [01:08:42] thanks krenair, i will. [01:09:27] (I tried supercount, which didn't work. It just gave me enwiki results.) [01:12:37] quiddity, http://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&list=users&ususers=Quiddity&usprop=editcount ? [01:12:50] RECOVERY - Host tools-webproxy-test is UP: PING OK - Packet loss = 0%, RTA = 0.82 ms [01:12:58] I think that's probably what you want. [01:12:59] Krenair, aha! perfect, thanks [01:13:37] Be aware that independent (*.wmflabs.org/*) tools may decide to count differently. [01:13:37] So many glorious tools. (I was just reminded about http://contropedia.net/ today) [01:14:20] yup. I'm familiar with the eccentricities of the various counters. Making people with editcountitis go crazier. :) [01:14:33] :) [02:28:47] PROBLEM - Host tools-webproxy-test is DOWN: CRITICAL - Host Unreachable (10.68.16.113) [06:25:18] Hmpf, https://tools.wmflabs.org/erwin85/ doesn't load [06:27:23] "Restarting webservice......... restarted. [06:27:44] Is too optimistic. You still get a 404 for a minute or so after the supposed completion :) [06:54:47] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [07:19:46] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [07:24:14] Coren: are you still around ? [07:25:00] 3Labs: Delete 'commonsarchivebot' from toollabs - https://phabricator.wikimedia.org/T89807#1045869 (10Fastily) 3NEW [07:26:16] 3Labs, Tool-Labs: Delete 'commonsarchivebot' from toollabs - https://phabricator.wikimedia.org/T89807#1045876 (10Fastily) [07:46:20] !log restarted webserver for stewardbots [07:46:20] restarted is not a valid project. [07:46:39] !log stewardbots webserver for restarted [07:46:39] stewardbots is not a valid project. [07:46:43] what ever [07:49:29] 3Wikimedia-Labs-wikitech-interface, operations: wikitech instances list is blank - https://phabricator.wikimedia.org/T89808#1045882 (10mmodell) 3NEW [08:02:41] 3Tool-Labs, Wikimedia-General-or-Unknown: Missing information template links in templatelinks database - https://phabricator.wikimedia.org/T89441#1045904 (10Springle) Percona Toolkit pt-table-sync is running for s3 (T89689) and s4 here, logging discrepancies. Step one it to fix the data, and step two to figure o... [11:37:00] PROBLEM - Puppet staleness on tools-exec-15 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [12:42:51] 3Tool-Labs, Wikimedia-General-or-Unknown: Missing information template links in templatelinks database - https://phabricator.wikimedia.org/T89441#1046184 (10Aschroet) @Springle, if there is anything that a normal labs user as me could do to solve or even further analyze this issue let me know. Otherwise i can ju... [13:04:30] <_joe_> !log deployment-prep installed new version of the hhvm extensions packages [13:04:33] Logged the message, Master [15:38:24] 3Wikimedia-Labs-wikitech-interface, operations: wikitech instances list is blank - https://phabricator.wikimedia.org/T89808#1046454 (10scfc) The first case (no instances showing up on https://wikitech.wikimedia.org/wiki/Special:NovaInstance) happens usually when the OpenStack auth expires, but not the MediaWiki... [15:53:35] andrewbogott_afk: Coren instances when deleted will disappear from shinken next puppetrun [15:53:44] * Guest30012 disappears into vacation again [15:53:54] Random act of knowledge! [16:59:06] Coren: can you please reinstate me (gifti) as maintainer of the tool catscan3? [16:59:43] Why aren't you there anymore? [17:00:39] i rage-quit some time ago >.> [17:00:58] 3Tool-Labs: Migrate tools to trusty - https://phabricator.wikimedia.org/T88228#1046899 (10scfc) With production moving to Jessie, does actively moving Tools to Trusty make sense any more (for the remaining hosts)? [17:02:03] 3Tool-Labs: Migrate tools to trusty - https://phabricator.wikimedia.org/T88228#1046901 (10coren) Probably not at this point - s/Trusty/Jessie/ [17:09:11] Coren: so? [17:10:04] 3Tool-Labs: Define rules for using Ganglia for individual tool statistics - https://phabricator.wikimedia.org/T50737#1046940 (10scfc) 5Open>3Invalid With Ganglia gone, this needs to be rethought anyhow. [17:20:11] 3Labs, Tool-Labs: Migrate tools-redis to a bigger instance - https://phabricator.wikimedia.org/T87107#1046977 (10scfc) `tools-redis` now is an instance with 16 GByte memory and 160 GByte disk space. IIRC the problem was not with memory; of the 160 GByte disk space, only the default 18 GByte are mounted at the m... [17:21:37] duh [17:21:56] i think it is time for a ticket again [17:23:00] Coren, let [17:23:10] corn, ping [17:23:13] Coren, ping [17:27:38] 3Tool-Labs: Reinstate maintainer of catscan3 - https://phabricator.wikimedia.org/T89851#1047021 (10Giftpflanze) 3NEW [17:29:26] Hi, I have created an instance a few days ago, but it's not listed anymore on the wikitech wiki instances list! [17:29:34] (03PS1) 10Merlijn van Deen: Log ALL the things! [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/191361 [17:29:56] but the instance seems to be still accessible, it's name is "mwoffliner2" [17:29:57] legoktm: ^ no time to deploy though [17:30:00] but worked locally [17:30:10] any explanation why? [17:30:33] (03CR) 10Legoktm: [C: 032] Log ALL the things! [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/191361 (owner: 10Merlijn van Deen) [17:30:47] (03Merged) 10jenkins-bot: Log ALL the things! [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/191361 (owner: 10Merlijn van Deen) [17:30:54] Kelson: have you tried logging out and logging in again? wikitech is weird like that... [17:31:33] !log tools.wikibugs legoktm: Deployed 8ba77ed2d2c039a231f3265da01215e721480ce0 Merge "Log ALL the things!" wb2-phab [17:31:36] Logged the message, Master [17:31:43] legoktm: same, still not in https://wikitech.wikimedia.org/w/index.php?title=Special:Ask&offset=0&limit=500&q=%5B%5BResource+Type%3A%3Ainstance%5D%5D&p=searchlabel%3Dinstances%2Fformat%3Dbroadtable&po=%3FInstance+Name%0A%3FInstance+Type%0A%3FProject%0A%3FImage+Id%0A%3FFQDN%0A%3FLaunch+Time%0A%3FPuppet+Class%0A%3FModification+date%0A%3FInstance+Host%0A%3FNumber+of+CPUs%0A%3FRAM+Size%0A%3FAmount+of+Storage%0A [17:32:05] legoktm: I can only see my first instance "mwoffliner1" [17:32:21] I'm not sure then... andrewbogott ^ ? [17:33:08] andrewbogott: hi, an idea about the problem described below? [17:33:16] Kelson: is it visible in the ‘Manage Instances’ page? [17:34:06] andrewbogott: yes, it's here [17:34:22] ok — best to not worry about it then… SMW is erratic about when/if it updates. [17:35:04] andrewbogott: ok, I have then a request to mount mount/create "/srv" on this instance, I think this is not something I can do myself. [17:35:38] it is — ‘configure instance’ and then select the labs::lvm::srv class. [17:38:56] Coren, do I have access to wikiviewstats? [17:38:57] andrewbogott: not sure I do it right. I have checked the checkbox and saved [17:39:05] andrewbogott: and then rebooted the VM [17:39:17] No need to reboot, but that should do it. [17:39:25] Otherwise just running puppet (sudo puppet agent -tv) would do it [17:39:48] 3Tool-Labs: webservice2 not starting - https://phabricator.wikimedia.org/T87641#1047087 (10scfc) 5Open>3Resolved a:3yuvipanda [17:39:53] andrewbogott: ok, it's there :) Only a small latency. Thank you for your help [17:40:02] np [17:40:31] 3Tool-Labs: webservice2 not starting - https://phabricator.wikimedia.org/T87641#995883 (10scfc) This should have been fixed by rOPUP8642f89f67c0fccbd172c2c198dbc240c220fb2f. [17:41:01] gifti: Can do, give me a minute. [17:42:33] 3Tool-Labs: Reinstate maintainer of catscan3 - https://phabricator.wikimedia.org/T89851#1047102 (10coren) 5Open>3Resolved a:3coren Done. [17:43:49] andrewbogott: Chime in on https://phabricator.wikimedia.org/T88802 ? [17:44:43] Coren: I’m sorry that this outage is coming on the tail end of another outage, but… yeah, seems OK. [17:46:54] 3Tool-Labs: Install byobu terminal multiplexer package on toollabs - https://phabricator.wikimedia.org/T88989#1047133 (10scfc) p:5Volunteer?>3Normal a:3scfc [17:50:22] andrewbogott: labs loves outages :p [17:51:50] legoktm: note that restarting redis2irc might cause it to revert back to the new color scheme =p [17:52:11] Coren, do I have access to wikiviewstats? [17:52:50] 3Tool-Labs: Migrate tools to trusty - https://phabricator.wikimedia.org/T88228#1047145 (10scfc) 5Open>3declined a:3scfc Okay, then I'll create another task for moving to Jessie. [17:52:51] valhallasw`cloud: uhhhh, the new-bad color scheme or the new-good one? [17:53:04] legoktm: new-good hasn't been merged yet =p [17:54:13] Cyberpower678: Doesn't look like it does. Hedonil is the one. [17:54:26] s/it does/you do/ [17:54:37] Can you add me? [17:54:54] Cyberpower678: Please to ask Hedonil. [17:55:01] /facepalm [17:55:32] Coren, hedonil is inactive, I've been pestering you about taking over that tool... :p [17:55:48] Oh, duh, sorry - misfiled in my brain. [17:55:56] :p [17:56:48] 3Tool-Labs: Migrate tools to trusty - https://phabricator.wikimedia.org/T88228#1047152 (10scfc) 5declined>3Open *argl* I fell through the trap door of tools vs. Tools. This task is about moving individual tools from Precise to Trusty to even out the grid load. This needs to be assessed on the basis of wha... [17:58:09] Cyberpower678: {{done}} You'll have to get a new OAUTH key of cousre. [17:58:25] I'm merging it with xTool's new OAuth key. [17:58:44] So that won't be a problem/ [17:59:45] T13|mobile, we have access to Wikiviewstats? :DDDDDDDD [18:01:22] Coren, can you add the clone project as well? Wikiviewstats2? [18:07:18] Yeah, done. [18:07:52] :D [18:11:12] T13|mobile, now we can fix the tool and restore functionality to the gadget. [18:27:01] 3Labs: Wikitech 'manage instances' displays "PHP Fatal error: Call to a member function getImageName() on a non-object" - https://phabricator.wikimedia.org/T89856#1047212 (10Andrew) 3NEW a:3Andrew [18:30:04] T13|mobile, I added you to both. [18:32:19] Kk [18:46:40] T13|mobile, now go fix it. :p [18:47:55] I'll look in a bit. [18:58:44] 3Tool-Labs: Open Grid Engine Job dumps core (node) - https://phabricator.wikimedia.org/T86905#1047324 (10scfc) 5Open>3Resolved a:3scfc [19:07:55] 3Tool-Labs: LOAD INTO FILE permission on MySQL tools-db server for user created databases - https://phabricator.wikimedia.org/T72956#1047359 (10scfc) 5Open>3declined `tools-db` is a separate server with a different file system. You should be able to use `LOAD DATA LOCAL INFILE` (NB: `LOCAL`) which allows to... [19:11:41] 3Tool-Labs: Random "can't get password entry for user "tools.liangent-py". Either the user does not exist or NIS error!" error - https://phabricator.wikimedia.org/T71529#1047383 (10scfc) Where did the error occur, and is it still happening? [19:13:01] 3Tool-Labs: Request for the Flickr::API perl module to be installed on toollabs - https://phabricator.wikimedia.org/T74800#1047387 (10scfc) 5Open>3Invalid Please reopen if you still need to have this module installed. [19:19:24] hmm, 502 for https://tools.wmflabs.org/catscan2/catscan2.php [19:28:56] 3Tool-Labs: Missing or wrong information in meta_p.wiki table - https://phabricator.wikimedia.org/T56962#1047474 (10scfc) At the moment, there are two wikis apart from `centralauth` that have `name` set to `NULL`: ``` MariaDB [enwiki_p]> SELECT * FROM meta_p.wiki WHERE name IS NULL; +-------------+------+------... [19:29:02] hmm, we should have a common 5xx page that gives the users some useful info.... [19:35:17] 3Tool-Labs: Common http error response pages - https://phabricator.wikimedia.org/T89864#1047487 (10TheDJ) 3NEW [19:44:02] Coren: ? [19:45:21] matanya: Myes? [19:45:53] fyi - https://github.com/puppetlabs/puppet/pull/3619/files [19:46:23] Coren: the file mode issue ticket worked to some extent. https://github.com/puppetlabs/puppet-docs/pull/453 for some ref [19:49:47] matanya: That works (that is, "at least four" is reasonable). I'm annoyed at 'The docs will periodically use "standard" to mean "customary" or "normal."' as a matter of principle but not to the point of raising a fuss over it. :-) [19:50:22] I mean, really, if they mean "customary" or "normal" why not use one of those words? :-) [19:50:31] so Coren , would you now accept 2755 for the grid engine? :) [19:51:44] Is there a pr for the style guide? (Which, IIRC, is what drives the lint decisions) [19:52:50] how could i find out in which version of MW was action=compare introduced? [19:53:04] But yeah, I'd be okay with '2755' to shut lint up -- though honestly I'd accompany it with a comment reminder that this is interpreted as octal despite the lack of leading 0. :-) [19:54:19] Coren: this one: https://github.com/rodjek/puppet-lint/issues/394 [19:55:05] Yeah, that one says "we follow the style guide" which is okay from a process pov - my point was that the style guide is dubious in that case. :-) [19:55:44] yes, that is a classic catch 22 [20:02:15] Can anybody help get my Tomcat up again on Tool Labs? http://tools.wmflabs.org/languagetool/ shows just a white page and when I restart Tomcat it gets stuck in state 'pending'. [20:10:00] 3Tool-Labs: DB replication results are slow. - https://phabricator.wikimedia.org/T75420#1047755 (10scfc) 5Open>3Resolved At the moment, the query returns `14231 rows in set (2.81 sec)`. That looks alright to me. [20:11:55] danielnaber: Lemme take a look. [20:15:05] danielnaber: Aha - no fault of yours: the tomcat queue is in error state. [20:17:25] danielnaber: Almost certainly caused by the hardware failing yesterday. I've cleared the queue - how is your webservice now? [20:17:49] RECOVERY - Host tools-webproxy-test is UP: PING OK - Packet loss = 0%, RTA = 0.90 ms [20:22:03] Coren: thank, it's working again. [20:27:13] Hello! My tool is down since 28 Jan. [20:27:20] How do I troubleshoot? [20:27:31] Last error: [20:27:43] 2015-01-27 22:26:51: (log.c.166) server started 2015-01-28 01:51:05: (network.c.358) can't bind to port: 4000 Address already in use 2015-01-28 01:51:20: (server.c.1558) server stopped by UID = 0 PID = 12044 [20:27:54] Hmm, let me try that on separate lines... [20:28:01] 2015-01-27 22:26:51: (log.c.166) server started [20:28:07] 2015-01-28 01:51:05: (network.c.358) can't bind to port: 4000 Address already in use [20:28:13] 2015-01-28 01:51:20: (server.c.1558) server stopped by UID = 0 PID = 12044 [20:28:28] from ~/error.log [20:29:43] try using a different port [20:29:49] 4000 seems already in use [20:31:19] slashme: That can be caused by a tool having accidentally taken the wrong port, and when yours tries to get its assigned one in fails. Give me a minute to find out which. [20:31:33] Thanks! [20:32:02] slashme: What queue are you trying to use it on? (I.e.: is it lighttpd and are you using trusty?) [20:32:49] Hmm, I set up the tool during the last Wikimania, as my first ever Labs project, and I just took default settings. [20:33:04] I can quickly check my environment... [20:33:29] sounds like it's not within toollabs but a separate project then? [20:33:30] No, that's okay - if you're using the defaults then I know which queue it's using. :-) [20:33:42] OK, cool :-] [20:33:55] slashme: What's your tool name? [20:34:07] .parliamentdiagram [20:34:13] without the . [20:34:21] bad cut/paste from terminal [20:36:00] http://tools.wmflabs.org/parliamentdiagram/ seems to work once started; but it gives me a 404 - dunno if that is expected? [20:36:40] That's certainly not expected. And I haven't touched it in months. [20:36:45] Let me see what's going on there. [20:37:06] Ah, now it's working! [20:37:20] http://tools.wmflabs.org/parliamentdiagram/parliamentinputform.html [20:37:35] Ah, you have nothing in the root of your tool. [20:37:38] I guess I can make http://tools.wmflabs.org/parliamentdiagram/ redirect. [20:38:16] But a moment ago the tool was giving me an error page saying that the address wasn't serviced. [20:38:34] slashme: I did a webservice start on it [20:38:37] Did you restart the tool, and if so how, and is it something I should be doing if users tell me it's down? [20:38:40] Ah, OK. [20:38:55] And is that something I can do when logged in as the tool? [20:39:17] * Coren nods. [20:39:31] * slashme thanks Coren for excellent help and support! [20:39:32] In fact, you must be logged in as the tool to do it. [20:39:38] Right, gotcha. [20:41:37] * slashme leaves by a slashme-shaped exit in the brick wall [22:02:29] 3hardware-requests, Labs, ops-eqiad, operations: virt1000 memory upgrade - https://phabricator.wikimedia.org/T89266#1048219 (10Cmjohnson) I can do this on Tuesday at 1500-1700UTC same time frame as Labstore1001. Please confirm if this will work for everyone. [22:05:59] hi, is the webservice for xtools crashed, or is it purposely not running? [22:06:24] akoopal: I don't know that it is; I heard nothing about it. [22:07:16] ahh, working again :-) [22:07:25] https://tools.wmflabs.org/xtools/blame/ [22:07:39] got the standard webservice not running [22:30:08] 3Wikimedia-Labs-Infrastructure, Beta-Cluster: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1048301 (10Krenair) [22:31:43] 3Labs, Wikimedia-Labs-Infrastructure: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1048313 (10hashar) [22:32:14] 3Labs, Wikimedia-Labs-Infrastructure: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1041223 (10hashar) I have removed the beta cluster, that is impacting all labs projects. [22:34:14] 3Labs, Wikimedia-Labs-Infrastructure, operations: Make labs/private really private - https://phabricator.wikimedia.org/T89642#1048321 (10Krenair) [23:12:15] which ldap implementation are we using? grepping through operations/puppet i think we are using openldap? [23:13:32] <^d> opendj, I think? [23:13:39] <^d> (same thing? me has no clue) [23:14:51] :) [23:59:15] 3Wikimedia-Labs-wikitech-interface, operations: wikitech instances list is blank - https://phabricator.wikimedia.org/T89808#1048561 (10mmodell) I haven't logged out and the problem seems to have resolved it's self. I'm not sure what the issue was, I had assumed it was related to the outage on labs yesterday. [23:59:26] 3Wikimedia-Labs-wikitech-interface, operations: wikitech instances list is blank - https://phabricator.wikimedia.org/T89808#1048562 (10mmodell) 5Open>3Invalid a:3mmodell