[00:02:19] Welcome to another thrilling round of Backport That Patch [00:03:15] <^d> No whammys, no whammys! [00:06:00] ori-l: yep, thx doing so. i suppose the ldap_ parameters don't change since it uses labs [00:07:21] I'm back [00:07:26] anything anyone wants/needs? [00:07:59] a macchiato [00:08:40] !log mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/resources/ext.multimediaViewer/ext.multimediaViewer.js 'Make VE and MMV work together again' [00:08:55] Logged the message, Master [00:09:07] marktraceur: Done? [00:09:14] let him test first ;) [00:09:17] !log mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/resources/ext.multimediaViewer/ext.multimediaViewer.lightboxinterface.js 'Make VE and MMV work together again' [00:09:33] Logged the message, Master [00:09:48] Three files needed updating [00:09:53] Probably should have sync-dir'd [00:09:56] But whatever [00:09:58] !log mholmquist synchronized php-1.23wmf6/extensions/MultimediaViewer/tests/qunit/lightboxinterface.test.js 'Make VE and MMV work together again' [00:10:12] No worries, just ping me when you're done [00:10:15] Logged the message, Master [00:10:46] Hm, bug's still there. [00:10:54] Probably RL cache. RoanKattouw, you can go. [00:11:02] do you need to touch some other file? [00:11:09] I doubt it [00:11:34] Yeah that should be fine, it'll take 5 minuutes [00:11:36] I'll go [00:12:00] Oh, it's looking good with debug=true [00:12:14] (03PS5) 10Dzahn: role and module structure for ishmael [operations/puppet] - 10https://gerrit.wikimedia.org/r/96403 [00:13:03] marktraceur: good work [00:13:25] * marktraceur notes on bug [00:14:13] (03CR) 10Dzahn: "ori-l: done. that's just site_name,ssl_cert and ssl_ca. the other stuff doesn't need to change between labs and prod" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96403 (owner: 10Dzahn) [00:15:15] so, when do I get to start asking Gage for favors? [00:15:35] only via the RT triage person! [00:15:36] not on his first day? [00:15:38] :) [00:15:49] im sure mutante is going to warn him to never take any task via PM or direct msg [00:15:56] :) :) [00:15:58] he should refer everyone to triage person [00:16:02] I take tasks via PM & direct msgs [00:16:15] yes, but its hard for a new person to tell folks no [00:16:20] also, what's gage's primary role going to be? [00:16:24] i rather they not be placed in that situation [00:16:37] don't worry, I won't corner him or anything [00:16:42] the rt triage person and ops mgmt are in better position to say no to things [00:16:48] greg-g: i know you wont, but others totally will ;] [00:17:04] the goal is to say "yes" and do more, not say no all the time :) [00:17:15] "we just got another person, say no more!" [00:17:16] ;) [00:17:23] You should an an RT field for location of bribes [00:17:24] i'd say first he gets gerrit [00:17:24] im not saying we have to say no more [00:17:28] then he makes his own account [00:17:30] but if thats how its being taken i dont care to discuss [00:17:36] I'm kidding! [00:17:37] throw the new dude to the wolves. [00:17:44] and merges it after review [00:17:57] * Nemo_bis prepares a list of maintenance scripts to be run [00:18:03] but, seriously, what's his specialty/area he'll be focusing on, or more of a generalist? [00:18:16] just curious since that wasn't in the announce email [00:18:23] who knows [00:18:34] i didnt know we were hiring someone until the announcement went out [00:18:36] greg-g: RT and mailman [00:18:57] cool [00:18:59] (i'm just kind of joking:) [00:19:06] mutante: so taking stuff off you're plate ;) [00:19:14] you're? your [00:19:28] greg-g: generalist, I hope [00:19:36] yea, let him pick from the pool [00:20:23] cool cool [00:21:35] would be nice if getting an IRC cloak is kind of quick [00:21:53] mutante: I still haven't received mine :/ [00:21:56] I need to reping [00:22:07] !gc [00:22:30] i bet James_F knows who to annoy for cloaks [00:22:35] !log catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Cherry-picks' [00:22:42] if the online form isnt getting responses that is [00:22:45] OK I'm done [00:22:49] ori-l: You're next up in the LD window [00:22:50] Logged the message, Master [00:22:51] #freenode can get you the unaffiliated cloak, usually takes < 5min [00:22:59] !gc is https://meta.wikimedia.org/wiki/IRC/Group_Contacts [00:22:59] Key was added [00:23:00] longer if you want a fancy cloak :P [00:23:02] yea, but its cool to get the wikimedia one =] [00:23:17] RobH: I actually don't, sadly. [00:23:27] https://spreadsheets.google.com/viewform?hl=en&formkey=dG1FTWV1RnNBVHFOSnExMHF6aUhya2c6MA [00:23:41] so im used to having folks fill out form and they are usually fast, but perhaps those folks are all on wikibreak [00:23:47] https://meta.wikimedia.org/wiki/IRC_channel_cloaks [00:23:58] see link above, there are 3 of them [00:24:13] yep [00:24:23] but i wouldnt contact htem directly until AFTER you submit the form [00:24:26] and dont get a reply [00:24:29] its just more polite. [00:24:45] RoanKattouw: I scheduled the slot optimistically, thinking I'd get https://gerrit.wikimedia.org/r/#/c/100542/ reviewed [00:24:49] the actual answer is, it should be part of on-boarding workflow [00:24:56] I haven't, so I'll return the slot to the pool [00:24:58] that meta.wikimedia.org one wont work, (at least, its not for wmf. Its for wiki editors with >250 edits. I know because i asked and they said no :P ) [00:25:04] ori-l: OK [00:25:09] No one else is waiting I think [00:25:14] ebernhardson: if you are paid by wmf [00:25:15] its enough [00:25:20] it helps identify you for that as well [00:25:22] RobH: i am, and i asked, and they said no [00:25:25] what?!?! [00:25:33] then i am not eligible for the cloak [00:25:37] RobH: they said there is a different thing for wmf [00:25:37] i have like 5 edits [00:25:43] no one tell them [00:25:46] i wanna keep it. [00:25:59] ebernhardson: they are nuts [00:26:09] there is nothing for wikimediafoundation on those wikipages at least [00:26:23] staff have traditionally just been granted wikimedia/ cloaks [00:26:36] make them count wikitech edits [00:26:48] oh if they count wikitech edits im a fucking editor for sure. [00:26:50] and office [00:26:52] heh [00:26:53] /msg MemoServ send wmfgc IRC cloak request [00:26:55] gah [00:26:57] i edit the hell out of those! [00:27:34] but when i did cloak stuff [00:27:38] casey was irc contact for it [00:27:39] greg-g: The FSF member cloak is nice and has the benefit of being trusted to mess with wm-bot [00:28:04] huh [00:28:05] but yea, if you have no cloak, id get unaffiliated at minimum [00:28:07] I dunno about globally, but at least in #mediawiki [00:28:13] its way more secure for channel auth than nick [00:28:28] I have one, just, it's kind of not legit any more (I revoked my own Ubuntu membership a while ago) [00:28:32] (unless you tweak your nick with enforcement) [00:28:36] heh [00:29:29] ori-l: so, no go today? [00:29:30] RobH: Nobody should use nicks for channel auth. Accounts are probably better, because it allows for cloak change without loss of status. [00:29:31] i just said that because i want to get him on all channels on day 1 but not add a non-cloak to access list [00:29:51] mutante: Use accounts instead of hostmasks. [00:29:57] true [00:30:58] greg-g: can still do it, gwicke merged [00:31:04] ori-l: sweet, do it [00:31:09] I'mma gonna head towards home [00:31:47] kk [00:36:33] what's up with /w/index.php?action=raw&ctype=text/javascript&title=MediaWiki:Common.js/IEFixes.js ? [00:36:37] ori-l, V8 is a method-based jit- which explains why the self-executing function wrapper makes a difference for optimizations [00:37:14] optimizations are only disabled for objects defined at the top level of the evaled source [00:38:40] marktraceur: yes, ack [00:39:07] gwicke: hrm, but there are fewer scopes in which to look up names [00:39:37] but i guess that wouldn't make a difference, since the names will be resolved in the same scope anyway [00:39:42] as they were before, I mean [00:40:03] but ReferenceErrors: foo is undefined will be faster! :D [00:40:35] random eval'ed code is just not a unit of optimization in the JIT [00:40:37] I can't run the test today after all, because I don't have a reference start time for the measurement [00:40:43] there's tons of /w/index.php?title=MediaWiki:RefToolbar.js&action=raw&ctype=text/javascript & /w/index.php?action=raw&ctype=text/javascript&title=MediaWiki:Common.js/IEFixes.js hits [00:40:45] it can just be a sequence of statements [00:40:47] I don't think these were there yesterday [00:41:07] gwicke: yeah, though the point about scope is not tied to v8 [00:41:15] but it's moot anyway [00:41:35] paravoid: looking [00:41:36] a jit can optimize away aliasing [00:41:58] it does not need to walk the frame chain every time as an interpreter would [00:42:42] paravoid: which wikis? [00:42:53] 5% of all backend traffic, sigh [00:43:36] RefToolbar.js seems to be eswiki [00:44:17] IEFixes mostly enwiki [00:44:24] neither were updated recently, hrm [00:44:33] maybe I'm wrong about them being recent [00:45:18] let me dig some more [00:48:01] hm, weird, now it doesn't appear at all again [00:48:08] as an outlier [00:49:34] can we switch ganglia-web's default time resolution view to 'day'? [00:49:47] I use hour more than day :) [00:50:08] huh, that's surprises me [00:50:11] * gwicke concurs with paravoid [00:50:52] depends what I'm investigating [00:51:24] right after a change the hourly view shows issues best [00:52:05] Aaron|home: have you seen http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Redis+eqiad&m=cpu_report&s=by+name&mc=2&g=network_report ? [00:52:19] oh my [00:53:13] https://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&c=Miscellaneous+pmtpa&h=hume.wikimedia.org&v=823574&m=Global_JobQueue_length&jr=&js= is also fun [00:54:06] 7 million? [00:54:08] wtf [00:54:19] well, 6, but still [00:54:47] probably an edit to one of the most-used templates [00:54:47] the following wikis have more than 199,999 jobs: , commonswiki (2125013), enwiktionary (3481304), Total (5945186) [00:55:29] or two ;) [00:57:54] those pages are small, so it will drain relatively quickly [01:03:11] everytime I dig in our logs, I just find weird stuff [01:03:19] tons of requests for /w/api.php?action=parse&redirects&prop=text%7Cdisplaytitle&format=xml&page=Orange [01:03:27] 253 TxHeader b User-Agent: Akamai SureRoute [01:03:29] 253 TxHeader b X-Akamai-TestObject: true [01:07:09] gwicke: so, what's the next steps re: nodejs memory leak? [01:07:18] what are even [01:07:49] paravoid: we used to have a cron job that restarted hanging test runners in rt testing once per hour [01:08:05] that likely covered up the leak in rt testing [01:08:28] this is now disabled, so over night we'll see whether we see the same leak in rt testing [01:08:44] if we do, then we can start to track it down there [01:08:49] how is the rt testing performed? [01:08:55] is it client http requests? [01:09:08] not yet (patch for that pending) [01:09:16] or are you reusing the parsing parts of parsoid? [01:09:21] right now it is test clients getting jobs and calling the module directly [01:09:25] right [01:09:32] so the leak might be somewhere else then [01:09:37] in the http codepath [01:09:55] if we don't see it, then it is likely connected to the http part [01:10:08] nod [01:10:24] I have been running rashomon under high load for days though using the 0.10 ppa [01:10:28] no leaks [01:11:16] so I suspect that it might be something specific to the backport [01:13:08] ori-l: match that with mc1001 [01:13:23] that's just the aggregator server being switched (see the SAL) [01:13:29] so nothing crazy there [01:17:55] parsoid seems to request the same URLs over and over and over [01:18:12] e.g. /w/api.php?action=query&format=json&prop=imageinfo&titles=File%3AGamepad.svg&iiprop=size%7Curl&iiurlwidth=80 [01:18:29] I was going to say; if it's the article on suicide or bomb making we should get worried [01:19:08] anyway, I'm going to stop log digging now [01:19:12] it has questionable results anyway :) [01:19:15] paravoid: IIRC within a single request we are caching those, but not across requests [01:19:54] so if a template using such an image was edited and all those page are re-rendered, it will still produce a ton of requests for the same resource [01:26:16] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [01:47:19] (03Abandoned) 10Jforrester: Add "betar" label to VisualEditor links on eswiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98613 (owner: 10Jforrester) [02:16:17] !log LocalisationUpdate completed (1.23wmf6) at Wed Dec 11 02:16:17 UTC 2013 [02:16:35] Logged the message, Master [02:25:16] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [02:30:45] !log LocalisationUpdate completed (1.23wmf5) at Wed Dec 11 02:30:44 UTC 2013 [02:31:01] Logged the message, Master [02:58:40] PROBLEM - Host elastic1007 is DOWN: PING CRITICAL - Packet loss = 100% [03:01:10] RECOVERY - Host elastic1007 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [03:14:00] (03PS1) 10Ori.livneh: Duplicate client-side latency measurements from eswiki to special bin [operations/puppet] - 10https://gerrit.wikimedia.org/r/100736 [03:17:38] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Dec 11 03:17:38 UTC 2013 [03:17:53] Logged the message, Master [03:46:39] (03PS1) 10QChris: Reconfigure and turn on geowiki monitoring again [operations/puppet] - 10https://gerrit.wikimedia.org/r/100747 [03:47:02] (03CR) 10jenkins-bot: [V: 04-1] Reconfigure and turn on geowiki monitoring again [operations/puppet] - 10https://gerrit.wikimedia.org/r/100747 (owner: 10QChris) [03:48:07] (03PS2) 10QChris: Reconfigure and turn on geowiki monitoring again [operations/puppet] - 10https://gerrit.wikimedia.org/r/100747 [03:49:51] (03CR) 10QChris: "recheck" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100747 (owner: 10QChris) [03:50:28] (03PS1) 10Ottomata: analytics1010 is in Row B, need a Gangalia aggregator there [operations/puppet] - 10https://gerrit.wikimedia.org/r/100749 [03:50:41] (03CR) 10QChris: [C: 04-1] "Depends on Ie3b1a4d210a37ab0929b808f27951be52ff8aa26 and $passwords::geowiki" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100747 (owner: 10QChris) [03:51:45] (03PS2) 10Ottomata: analytics1010 is in Row B, need a Gangalia aggregator there [operations/puppet] - 10https://gerrit.wikimedia.org/r/100749 [03:52:33] (03CR) 10Ottomata: [C: 032 V: 032] analytics1010 is in Row B, need a Gangalia aggregator there [operations/puppet] - 10https://gerrit.wikimedia.org/r/100749 (owner: 10Ottomata) [04:28:44] (03PS3) 10QChris: Reconfigure and turn on geowiki monitoring again [operations/puppet] - 10https://gerrit.wikimedia.org/r/100747 [04:30:20] (03PS4) 10Ottomata: Reconfigure and turn on geowiki monitoring again [operations/puppet] - 10https://gerrit.wikimedia.org/r/100747 (owner: 10QChris) [04:30:30] (03CR) 10Ottomata: [C: 032 V: 032] Reconfigure and turn on geowiki monitoring again [operations/puppet] - 10https://gerrit.wikimedia.org/r/100747 (owner: 10QChris) [04:58:17] (03PS1) 10Dzahn: include the bugzilla config in puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/100752 [05:27:26] (03CR) 10Ori.livneh: "> the other stuff doesn't need to change between labs and prod" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96403 (owner: 10Dzahn) [05:59:01] (03CR) 10Ori.livneh: Include redis on logstash servers (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100511 (owner: 10Aaron Schulz) [05:59:22] (03PS3) 10Ori.livneh: Include redis on logstash servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/100511 (owner: 10Aaron Schulz) [05:59:32] (03PS4) 10Ori.livneh: Include redis on logstash servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/100511 (owner: 10Aaron Schulz) [06:00:39] (03CR) 10Ori.livneh: [C: 032] Include redis on logstash servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/100511 (owner: 10Aaron Schulz) [06:01:22] (03PS2) 10Ori.livneh: Duplicate client-side latency measurements from eswiki to special bin [operations/puppet] - 10https://gerrit.wikimedia.org/r/100736 [06:02:35] (03CR) 10Ori.livneh: [C: 032] Duplicate client-side latency measurements from eswiki to special bin [operations/puppet] - 10https://gerrit.wikimedia.org/r/100736 (owner: 10Ori.livneh) [06:05:44] !log silencing global collect paging (GC is having issues) for 1 hour [06:05:58] Logged the message, Mistress of the network gear. [06:06:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:08:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:09:12] oh, yuck [06:09:13] err: /Stage[main]/Redis/File[/a/redis]/ensure: change from absent to directory failed: Cannot create /a/redis; parent directory /a does not exist [06:10:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:11:34] (03PS1) 10Ori.livneh: logstash: specify /var/run/redis as redis $dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/100755 [06:12:39] (03CR) 10Ori.livneh: [C: 032] logstash: specify /var/run/redis as redis $dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/100755 (owner: 10Ori.livneh) [06:12:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:14:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:16:03] (03PS1) 10Ori.livneh: logstash: re-order includes to avoid duplicate def'n [operations/puppet] - 10https://gerrit.wikimedia.org/r/100756 [06:16:21] (03CR) 10Ori.livneh: [C: 032 V: 032] logstash: re-order includes to avoid duplicate def'n [operations/puppet] - 10https://gerrit.wikimedia.org/r/100756 (owner: 10Ori.livneh) [06:16:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:18:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:20:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:22:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:24:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:26:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:28:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:00:22 AM UTC [06:29:16] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Wed Dec 11 06:29:12 UTC 2013 [06:30:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:29:12 AM UTC [06:32:46] PROBLEM - Puppet freshness on mw1035 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 06:29:12 AM UTC [06:59:56] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Wed Dec 11 06:59:45 UTC 2013 [08:01:35] (03PS1) 10ArielGlenn: on puppetmasters, remove default ubuntu logrotate for apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/100758 [08:03:03] (03CR) 10ArielGlenn: [C: 032] on puppetmasters, remove default ubuntu logrotate for apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/100758 (owner: 10ArielGlenn) [08:18:04] (03PS1) 10ArielGlenn: add mw1-16 (pmtpa inactive job runners) to dsh apaches [operations/puppet] - 10https://gerrit.wikimedia.org/r/100759 [08:38:17] (03PS1) 10Matanya: svn: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 [08:38:50] (03CR) 10jenkins-bot: [V: 04-1] svn: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 (owner: 10Matanya) [08:39:51] (03PS2) 10Matanya: svn: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 [08:45:48] !log manually downgraded nodejs on contint (gallium, lanthanum) from 0.10.22 to 0.8.2 (puppet failed to do so) [08:46:04] Logged the message, Master [09:09:00] * apergos eyes robla suspiciously [09:09:07] isn't it a bit late for you? [09:13:16] one could look at it, as being early [09:14:07] there's early and there's ridiculous :-P [09:15:20] and how you know its not future time travelling Rob? [09:16:58] then I would be right to eye him suspiciously [09:18:20] (03PS1) 10ArielGlenn: access to zirconium for bd808, rt #6448 [operations/puppet] - 10https://gerrit.wikimedia.org/r/100762 [09:24:02] (03CR) 10ArielGlenn: [C: 032] access to zirconium for bd808, rt #6448 [operations/puppet] - 10https://gerrit.wikimedia.org/r/100762 (owner: 10ArielGlenn) [09:30:40] apergos: please remind me how do i test a change on labs? any doc on wikirech? [09:31:46] that is a good question (about docs) [09:32:01] apergos: all i know is: https://wikitech.wikimedia.org/wiki/Puppet_usage#labs_testing [09:32:24] that's pretty complete actually [09:32:26] but nothing about how i put my patch in /var/lib/git/operations/puppet [09:33:07] that's step 4 isn't it? [09:33:30] apergos: i want to pull it from gerrit. https://gerrit.wikimedia.org/r/#/c/100760/ [09:33:40] git review -D didn't work [09:33:45] ugh [09:33:50] I don't use git review [09:34:01] just pull? [09:34:40] or cherry pick I guess [09:34:57] depending how current your self hosted puppet branch is [09:38:31] (03PS1) 10ArielGlenn: remove bd808 from restricted, already in mortals [operations/puppet] - 10https://gerrit.wikimedia.org/r/100763 [09:39:01] Hi ops [09:39:30] morning [09:39:38] :) [09:39:41] (03CR) 10ArielGlenn: [C: 032] remove bd808 from restricted, already in mortals [operations/puppet] - 10https://gerrit.wikimedia.org/r/100763 (owner: 10ArielGlenn) [09:39:41] Is this the right place to ask about a Gerrit/jenkins test failure I don't understand? [09:39:46] https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-qunit-mobile/803/console [09:40:41] hashar would be the expert on contint [09:40:50] AndyRussG: i guess hashar would be able to help here [09:41:05] hello [09:41:08] morning [09:41:11] Morning hashar [09:41:23] How's it going? [09:41:49] AndyRussG: so seems some page is timing out in the QUnit javascript tests [09:41:54] AndyRussG: if you go on the job page at https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-qunit-mobile/803/ [09:42:25] AndyRussG: you will find the full debug logs for CLI script and web queries (respectively: mw-debug-cli.log.gz and mw-debug-www.log.gz ) [09:42:32] that corresponds to $wgDebugLogFile [09:43:08] AndyRussG: there is a stack trace :( [09:43:36] [exception] [fa6da801] /jenkins-mwext-MobileFrontend-qunit-mobile-803/index.php?title=Special:JavaScriptTest/qunit&useformat=mobile Exception from line 112 of /srv/ssd/jenkins-slave/workspace/mwext-MobileFrontend-qunit-mobile/includes/WikiPage.php: Invalid or virtual namespace -1 given. [09:43:46] AndyRussG: have a look at http://paste.debian.net/70323/ [09:43:50] yw, sorry for long spam :-] [09:44:53] Wow, thanks [09:45:06] As far as I can see that has nothing to do with my patch [09:45:19] Which was just a one-liner [09:45:25] hashar: while you're here, I downgraded nodejs on gallium/lanthanum, it had been upgraded for parsoid but they rolled that back [09:45:35] just in case there are any issues, hopefully not [09:45:41] Ah no, I'm wroing [09:45:46] apergos: ah thank you to have taken care of the downgrade! [09:45:52] It *does* go through my patch [09:46:12] apergos: I have noticed it happened in production (ensure latest -> present) but haven't thought about downgrading on jenkins slaves :( [09:46:30] yes, well you have ensure latest which makes sense [09:46:48] AndyRussG: something worth mentioning is that we use MediaWiki master branch AND your patch against MobileFrontend is merged with the tip of the MobileFrontend master branch [09:46:49] OK, I think I got it [09:46:51] and ordinarily latest will be later :-D [09:47:18] OK [09:47:22] AndyRussG: so sometime it might be an issue in a change introduced in the extension master branch, though that is really unlikely [09:47:46] Yeah, no, I think I know what it might be, the stack trace is a huge help [09:47:53] hashar: thanks so much!!! [09:48:29] who do we think is the nominal manager for fundraising tech? is that jeff? [09:59:47] katie? [10:06:34] (03PS1) 10Mark Bergsma: Revert "Take mark off the SMS list" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100765 [10:06:42] (03CR) 10jenkins-bot: [V: 04-1] Revert "Take mark off the SMS list" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100765 (owner: 10Mark Bergsma) [10:07:29] (03Abandoned) 10Mark Bergsma: Revert "Take mark off the SMS list" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100765 (owner: 10Mark Bergsma) [10:07:51] (03PS1) 10Mark Bergsma: Put mark back in the SMS list [operations/puppet] - 10https://gerrit.wikimedia.org/r/100766 [10:08:54] (03CR) 10Mark Bergsma: [C: 032] Put mark back in the SMS list [operations/puppet] - 10https://gerrit.wikimedia.org/r/100766 (owner: 10Mark Bergsma) [10:11:49] mark: welcome back :-] [10:12:05] thanks :) [10:12:49] apergos: yup Katie Horn is the lead software dev, happily dispatching fundraising tech issues [10:13:04] great, thanks [10:13:18] apergos: I don't think she is actually managing Matthew and Adam, merely doing tech leading [10:13:22] oh [10:13:25] hm [10:13:38] she is most probably part of the scrum of scrum as well [10:14:01] anywa,y you can't be wrong pinging her. At worth she will route your request to the appropriate person [10:14:15] good point [10:26:26] dns katie [10:29:16] (03PS1) 10Hashar: beta: adapt $wgParsoidCacheServers [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100769 [10:29:31] (03CR) 10Hashar: [C: 032] beta: adapt $wgParsoidCacheServers [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100769 (owner: 10Hashar) [10:29:42] (03Merged) 10jenkins-bot: beta: adapt $wgParsoidCacheServers [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100769 (owner: 10Hashar) [10:32:06] ah elastic search is paying off [10:32:18] they added thumbnails of pictures in Special:Search http://commons.wikimedia.beta.wmflabs.org/w/index.php?title=Special%3ASearch&profile=default&search=jpg&fulltext=Search [10:34:15] mark: welcome back!!! [10:34:26] hey [10:34:36] did you have a nice time ? [10:34:41] very :) [10:35:27] * akosiaris envious [10:35:32] thanks for keeping wikivoyage & wikipedia up during my travels ;) [10:35:50] they were rather useful from time to time hehe [10:36:20] those thanks should mostly go to paravoid. At some point he had a streak of responding to 15 outages in a row. [10:38:31] so I read [10:42:03] i guess he's sleeping it off now ;-p [10:43:51] (03PS1) 10Hashar: beta: get rid of lucene search configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/100771 [10:51:26] (03CR) 10Akosiaris: [C: 032] Template ferm's defs.production [operations/puppet] - 10https://gerrit.wikimedia.org/r/99705 (owner: 10Akosiaris) [11:04:12] PROBLEM - Host elastic1007 is DOWN: PING CRITICAL - Packet loss = 100% [11:07:17] RECOVERY - Host elastic1007 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [11:08:50] (03PS1) 10Akosiaris: Replace +,- chars with _ in ferm defs [operations/puppet] - 10https://gerrit.wikimedia.org/r/100776 [11:10:02] akosiaris: i'm testing my svn module and get: err: Failed to apply catalog: Could not find dependent Service[apache2] for File[/etc/apache2/sites-available/svn] at /etc/puppet/modules/svn/manifests/server.pp:36 [11:10:11] how can i debgu this? [11:11:00] the changeset is at https://gerrit.wikimedia.org/r/#/c/100760/ [11:11:03] matanya: the error pretty much says all you need to know. The File resource depends on a Service resource that is not defined [11:11:45] you need to look carefully and decided whether you really need the dependency (the require parameter probably) [11:12:59] how are you testing ? [11:13:13] thanks akosiaris may question was regarding the need, not the error itself. [11:13:24] i'm running it in a labs instance [11:13:50] oh sorry, I did not understand that from the question [11:14:56] in that case, this requires a more thorough examination of the manifest which I can do right now. I look into it in about 30 mins, ok ? [11:15:03] yeah, sorry, bad word choosing. [11:15:10] I can not, i meant [11:15:23] sure 30 mins. thanks [11:15:38] (03CR) 10Akosiaris: [C: 032] Replace +,- chars with _ in ferm defs [operations/puppet] - 10https://gerrit.wikimedia.org/r/100776 (owner: 10Akosiaris) [11:20:29] hmmm there seems to be an ordering issue here. Minor but /etc/ferm/conf.d/00_defs changes in every run (lines get rearranged). I wonder if it's puppet's fault or ERB's... [11:26:52] you need to sort any hashes, right [11:28:21] meh.. :-( [11:28:57] ruby hashes are not stable, I should sort by key i suppose [11:35:30] yep [11:39:06] heeeeeelloo [11:39:31] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [11:39:51] hi [11:40:39] grumble, that check firing just as soon as I say hello [11:41:06] I was about to tell you that, then I said.. let him be :P [11:41:13] too early in the morning for him [11:41:16] ok, false positive [11:41:24] POST http://he.wikipedia.org/ [11:41:28] empty POSTs, someone [11:43:45] (03PS1) 10Akosiaris: Sort ruby hashes before interpolation in ferm defs [operations/puppet] - 10https://gerrit.wikimedia.org/r/100778 [11:46:28] (03CR) 10Akosiaris: [C: 032] Sort ruby hashes before interpolation in ferm defs [operations/puppet] - 10https://gerrit.wikimedia.org/r/100778 (owner: 10Akosiaris) [12:24:33] matanya: those svn manifests are so weird to begin with, that it is not easy to answer your question. You can a) remove the notify and be done with it (it is a legacy service after all), b) include webserver::apache to satisfy the dependency (hosts in site.pp either do it directly or through some other class), c) refactor the entire thing so much that the dependency no longer makes sense. This class is a hybrid between a role class a [12:25:53] thanks akosiaris. i think i'll do a in this patch and then c in a seperate patch. what do you think? [12:27:02] matanya: sounds like it has a good chance of working. [12:27:48] (03CR) 10Akosiaris: [C: 04-1] include the bugzilla config in puppet (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100752 (owner: 10Dzahn) [12:29:27]