[00:26:55] !log integration Greg Grossmeier added and made project admin [00:26:58] Logged the message, Master [00:28:14] (bd808: thanks) [00:28:15] yw [00:28:15] You barely beat me to hitting the second submit button [01:19:46] I want to test out some queries that might be a bit taxing. Is there a good way for me to monitor DB load? [01:35:09] 3Wikimedia Labs / 3deployment-prep (beta): Beta Labs: API PrefixSearch is taking a very long time to return, and returns nothing when it does - 10https://bugzilla.wikimedia.org/72332#c4 (10James Forrester) 5ASSI>3NEW a:3None Moving to Beta Labs. [06:09:37] 3Wikimedia Labs / 3wikistats: Respect wikisite and EditThis throttling - 10https://bugzilla.wikimedia.org/59742#c11 (10Nemo) 5ASSI>3RESO/FIX Seems to have worked well! Only a few dozens wikis errored out now. http://wikistats.wmflabs.org/display.php?t=et&s=http_desc http://wikistats.wmflabs.org/display.p... [06:17:29] !ping [06:17:29] !pong [06:22:48] !ping [06:22:48] !pong [06:33:53] 3Wikimedia Labs / 3deployment-prep (beta): Beta Labs: API PrefixSearch is taking a very long time to return, and returns nothing when it does - 10https://bugzilla.wikimedia.org/72332#c5 (10Sam Reed (reedy)) Didn't Chad remove numerous search instances from beta? [08:55:09] 3Wikimedia Labs: wildcard release parameter fails with jsub - 10https://bugzilla.wikimedia.org/72248#c1 (10merl) 5NEW>3RESO/FIX I found a solution: wildcard still does not work with jsub, but regex -l release=trusty|precise (without quoting) at script header solves my problem. qsub also accepts single or d... [13:35:59] !ping [13:35:59] !pong [14:00:51] !PING [14:01:02] BOT IS NOT CAPSLOCK COMPATIBLE :( SUCH A PITY [14:01:17] !PING is !ping [14:01:17] Key was added [14:01:21] !PING [14:01:21] !ping [14:01:35] !PING del [14:01:35] Successfully removed PING [14:01:40] !PING Is !pong [14:01:45] !PING [14:01:47] .... [14:01:53] !PING ALIAS !ping [14:01:59] !PING alias !ping [14:01:59] You are not authorized to perform this, sorry [14:02:04] LIKE HELL I'M NOT [15:57:02] YuviPanda: Have you seen https://gerrit.wikimedia.org/r/#/c/167852/ ? [16:05:55] Coren: oooh, nice! [16:05:59] I'll take a look tomorrow... [16:37:47] <^d> YuviPanda: Yo, so statsd plugin is ancient. [16:37:59] <^d> Designed for ES 0.90.x! [16:38:07] <^d> I'm trying to hack it into shape for 1.x+ [16:45:09] ^d: aaah, right [16:45:11] ok [16:46:04] <^d> yay all compiled & passing tests [16:49:22] !ping [16:49:22] !pong [16:54:38] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079 (10Greg Grossmeier) [16:54:41] 3Wikimedia Labs / 3deployment-prep (beta): Automatic mobile redirects for MobileFrontend on Betalabs - 10https://bugzilla.wikimedia.org/46620 (10Greg Grossmeier) [16:55:09] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c8 (10Greg Grossmeier) a:5Sam Reed (reedy)>3Brion Vibber Hey Brion, Reedy suggested you taking a look into this one (Apple devices not redirecting to the mobile domai... [16:56:37] 3Wikimedia Labs / 3deployment-prep (beta): Automatic mobile redirects for MobileFrontend on Betalabs - 10https://bugzilla.wikimedia.org/46620#c2 (10Sam Reed (reedy)) 5NEW>3RESO/FIX This seems to work fine for me on my Android at least. Bar bug 71079 at least (don't have an apple device to test on) [16:58:22] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c9 (10Brion Vibber) If I go to http://en.wikipedia.beta.wmflabs.org on my iPad Air (iOS 8.1) I get redirected to http://en.m.wikipedia.beta.wmflabs.org and I see the mobi... [17:00:16] <^d> Hmm, plugin loaded. [17:00:21] <^d> Nothing coming out tho. [17:10:23] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c10 (10Sam Reed (reedy)) (In reply to Brion Vibber from comment #9) > If I go directly to http://www.en.m.wikipedia.beta.wmflabs.org then I get > "mobile domains are not... [17:13:54] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c11 (10Brion Vibber) Logging in keeps me on en.m.wikipedia.beta.wmflabs.org; I cannot reproduce the bug except by manually typing in the wrong domain. [17:38:41] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366 (10Chris McMahon) 3NEW p:3Unprio s:3normal a:3None I am seeing a number of 503s from beta labs but upon checking, everything looks normal. Here is an example: https://integration.w... [17:40:08] hello, are issues with labs en enwiki data solved? [17:40:24] there were reports couple days ago that data was missing [17:42:38] 3Wikimedia Labs / 3deployment-prep (beta): beta labs goes 503 for a short time - 10https://bugzilla.wikimedia.org/72366#c1 (10Chris McMahon) adding Ori in case this might be a consequence of HHVM [17:57:23] 3Wikimedia Labs / 3deployment-prep (beta): Mobile redirect goes to wrong domain name on beta labs - 10https://bugzilla.wikimedia.org/71079#c12 (10Greg Grossmeier) 5NEW>3ASSI Kaldari: Can you still reproduce your issue here? What are the exact steps that Brion can try? [18:12:39] Coren: virt1005 has a whole lot of things like "lldp_send: unable to send packet on real device for vnet21: No buffer space available" in the syslog. I've verified that instances running on virt1005 do have network connectivity, so I guess it's just dropping the occasional packet... [18:12:57] Is that likely just because something is running amuck on that host and using too much bandwidth? [18:14:00] Maybe not even all that much bandwidth, depending on how big the buffer space is - occasional spikes might do it. [18:14:20] But if it's only a couple/minute it's not a concern I'd think. [18:14:56] I've got a mail today from an institution. They used to load some files from tool labs (from one of my tools) every six hours. Now they get a 403 from their particular IP. Sounds like a block. Where can I see, which IPs are blocked and why? Who is the correct person to contact? [18:16:14] apper: I would be, but I'm going to guess that if it started a couple weeks ago it's because whatever they use to fetch the data does not send a valid User-Agent header. [18:17:29] Coren: thanks, sounds reasonably [18:17:46] Coren: ok, I will mentally file that as not a problem, but it's probably a good place to look if you seen network quality degrading. [18:17:47] I'll write them and will come back if this is not the reason. [18:27:41] andrewbogott: Also, there may be a tunable we can crank up to reduce the impact of occasional spikes in network activity by beefing up the queue length. [18:50:46] !ping [18:50:46] !pong [19:24:22] !ping [19:24:22] !pong [19:59:37] andrewbogott: I'd have prefered it if YuviPanda|zzzz had the time to check on https://gerrit.wikimedia.org/r/#/c/167852/ but it's kinda blocking me in the resolution of Magnus's issues atm (because I need the new instance types for him) [20:00:09] And I expect I'll need a couple of round of fixes once I try it live too. [20:00:24] Would you like me to merge it right now? It looks fine to me. [20:01:05] I can merge it, I just wanted someone else to +1 it. I'm not averse to self-+2 on trivial labs patches but that one definitely doesn't qualify as trivial. :-) [20:02:24] Thankee. :-) [20:39:16] The lack of external resources /really/ hurts. [21:10:23] !log deployment-prep updated OCG to version e977e2c8ecacea2b4dee837933cc2ffdc6b214cb [21:10:27] Logged the message, Master [21:42:46] I've never really hated puppet despite its warts, but no external resources really cripples it. [21:43:11] Coren: I don't know for certain that we can't use them, you could run some tests to see if our puppetmaster is up to the task. [21:43:22] Or make a separate master just for tools... [21:43:33] They're disabled, but for a good security reason. [21:43:43] Well, 'decent' security reason. [21:44:28] Ah, I'd always heard it chalked up to performance, but I can think of plenty of security reasons too [21:45:15] I don't think they have significant performance impact unless you use them, really. [21:45:47] But yeah, turning them on would mean we'd have to be really paranoid when doing puppet code review. [21:46:46] But the cost is high. Monitoring would be a *lot* easier to do with external resources, for one, and so would most "here is a bunch of things I'd like to make a list of" [21:47:36] I'll cope. It just makes gridengine especially ugly to puppetize and makes it rely on a shared filesystem which is teh suck. [21:48:24] The only alternative I can think of is have labs use its own puppet manifest and puppetmaster, which sucks for different reasons. [21:50:32] <^d> We tried having labs on its own branch before. [21:50:37] <^d> That ended in pain. [21:52:19] separate master wouldn't necessarily mean runnign out of a different repo. [21:52:25] OTOH, I'm not entirely convinced that allowing exported resources is all that problematic. [21:52:27] but, it would still be a lot of trouble [21:53:23] It would mean that we'd have to carefully review anything that exports resources, obviously. [21:54:32] Ima write to ops-l about this; because while the gunk I have to write now for gridengine will /work/, it'll be brittle. [22:21:37] 3Wikimedia Labs: Signup broken - FancyCaptcha.class.php: Ran out of captcha images - 10https://bugzilla.wikimedia.org/71587#c2 (10Matthew Flaschen) 5NEW>3RESO/FIX Fixed. I forgot to copy in the CAPTCHA images. For future reference: sudo mkdir /mnt/fancycaptcha sudo cp /data/project/fancycaptcha_images/*... [22:21:53] 3Wikimedia Labs: Signup broken - FancyCaptcha.class.php: Ran out of captcha images - 10https://bugzilla.wikimedia.org/71587 (10Matthew Flaschen) a:3Matthew Flaschen [22:42:10] 3Wikimedia Labs / 3(other): Can't create account on phab-01 - 10https://bugzilla.wikimedia.org/72403 (10Matthew Flaschen) 3NEW p:3Unprio s:3normal a:3None I can't create an account at https://phab-01.wmflabs.org/auth/register/ . It throws an exception: Unhandled Exception ("AphrontQueryException")... [22:43:08] 3Wikimedia Labs / 3(other): Can't create account on phab-01: 'worker_activetask' is full - 10https://bugzilla.wikimedia.org/72403 (10Matthew Flaschen) [22:44:52] 3Wikimedia Labs / 3(other): Can't create account on phab-01: 'worker_activetask' is full - 10https://bugzilla.wikimedia.org/72403#c1 (10Chad H.) Yeah, it's been wonky all day. I think the disk is full. [23:18:37] 3Wikimedia Labs / 3deployment-prep (beta): Beta Labs: API PrefixSearch is taking a very long time to return, and returns nothing when it does - 10https://bugzilla.wikimedia.org/72332#c6 (10Greg Grossmeier) (In reply to Sam Reed (reedy) from comment #5) > Didn't Chad remove numerous search instances from beta... [23:29:07] 3Wikimedia Labs / 3deployment-prep (beta): Beta Labs: API PrefixSearch is taking a very long time to return, and returns nothing when it does - 10https://bugzilla.wikimedia.org/72332#c7 (10Chad H.) deployment-elastic01 through 04 don't exist anymore. They're replaced with deployment-elastic05 through 08. T... [23:32:08] 3Wikimedia Labs / 3deployment-prep (beta): Beta Labs: API PrefixSearch is taking a very long time to return, and returns nothing when it does - 10https://bugzilla.wikimedia.org/72332#c9 (10Sam Reed (reedy)) Config is still on the old servers.... https://noc.wikimedia.org/conf/highlight.php?file=CirrusSearch-... [23:47:09] !paste [23:47:09] http://tools.wmflabs.org/paste/ [23:49:36] wow @ "trending pastes" [23:50:21] afraid the fact that phab has built-in pastebin too might be tempting to just use that one. but tools pastebin is certainly pretty advanced, wowww [23:50:35] syntax highlighting ..sweet