[00:03:29] !ping [00:03:29] !pong [02:54:56] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-exec-14.diskspace._var.byte_avail.value (11.11%) [03:01:16] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [03:16:06] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-exec-14.diskspace._var.byte_avail.value (10.00%) [09:30:28] !help [09:30:29] !documentation for labs !wm-bot for bot [09:30:33] !wm-bot [09:30:33] http://meta.wikimedia.org/wiki/WM-Bot [09:54:43] 3Tool Labs tools / 3Commons Delinker: Error: InnoDB - 10https://bugzilla.wikimedia.org/73353 (10Steinsplitter) 3NEW p:3Unprio s:3normal a:3None /data/project/delinker/cdh/mysql_autoconnection.py:56: Warning: InnoDB: WITH CONSISTENT SNAPSHOT was ignored because this phrase can only be used with REPEAT... [10:11:44] !log deployment-prep cherry pick https://gerrit.wikimedia.org/r/#/c/172967/1 to test https://bugzilla.wikimedia.org/show_bug.cgi?id=73263 [10:11:46] Logged the message, Master [10:30:55] Having trouble trying to stop a job with jstop. Is this due to the system crash? [10:59:40] 3Tool Labs tools / 3Commons Delinker: Error: InnoDB - 10https://bugzilla.wikimedia.org/73353#c1 (10Andre Klapper) Why does the bug summary say "Error" when the comment says "Warning" only? [11:28:43] 3Wikimedia Labs / 3deployment-prep (beta): deployment-parsoidcache02 is not using betalabs puppetmaster - 10https://bugzilla.wikimedia.org/73357 (10Yuvi Panda) 3NEW p:3Unprio s:3normal a:3None It is still using virt1000, which is the general labs puppetmaster. I don't think this is intentional :) Thi... [11:29:03] Hi everybody. I'm trying to kill and re-start a job on tools, but it's not responding to jstop. Any ideas? Or do I just need to wait? [12:14:52] PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-exec-14.diskspace._var.byte_avail.value (100.00%) tools.tools-exec-15.diskspace._var.byte_avail.value (100.00%) tools.tools-exec-13.diskspace._var.byte_avail.value (100.00%) tools.tools-webgrid-04.diskspace._var.byte_avail.value (100.00%) tools.tools-exec-11.diskspace._var.byte_avail.value (100.00%) tools.tools-exec-09.diskspace._var.byte_avail.value [12:15:58] wtf [12:16:00] coredumps [12:16:02] I bet [12:51:42] 3Wikimedia Labs / 3tools: imagemapedit folder on http://tools.wmflabs.org does not open hence problem accessing files - 10https://bugzilla.wikimedia.org/73361 (10Umesh Patil) 3UNCO p:3Unprio s:3major a:3Marc A. Pelletier Hi, My instance of Wiki trying to access below two files, http://tools.wmflabs... [13:42:10] 3Tool Labs tools / 3Commons Delinker: Warning: InnoDB @ cdh - 10https://bugzilla.wikimedia.org/73353 (10Steinsplitter) [13:43:26] 3Tool Labs tools / 3Commons Delinker: COMMONSDELINKER-20 Add a diff link on delinker log - 10https://bugzilla.wikimedia.org/61655 (10Steinsplitter) s:5major>3enhanc [13:44:56] 3Tool Labs tools / 3Commons Delinker: COMMONSDELINKER-12 Disable pywikpedia throttle file - 10https://bugzilla.wikimedia.org/61676 (10Steinsplitter) a:5Bryan Tong Minh>3None [14:26:23] is betalabs up to date schema-vise? [14:26:28] Function: LinksUpdate::incrTableUpdate [14:26:28] Error: 1054 Unknown column 'pl_from_namespace' in 'field list' [14:42:32] also, it seems the last update to ZeroPortal extension happened is ef58f57, 4 days ago [15:40:04] RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK [16:01:33] Hi again everybody [16:02:07] I'm still not able to kill a job on the labs cluster [16:02:33] I tried jstop this morning [16:03:04] qstat is still showing it as "dr", now about 7 hours later [16:04:01] jheald: It's possible - if your job is relatively old (<~ 2 weeks or so) that it was started while there was a bug; you'll have to kill the processes yourself. :-( [16:04:21] I started it last night [16:04:37] Hm. Odd. Lemme me go look at it. What's its job ID? [16:04:37] it needs to be stopped & restarted every 12 hours [16:04:58] 5612633 [16:05:36] (a session key it uses for Flickr expires & has to be re-updated) [16:06:03] jheald: Also (unrelated) you shouldn't be running a bot under your own account. [16:06:37] So I need to register a tool account for it ? [16:06:44] Yep. [16:06:53] But that shouldn't be related. [16:08:23] That's odd. I see no reason why deleting the job wouldn't have worked. [16:13:55] At any rate, I've killed it "by hand" as it were. [16:20:29] Thanks. I'll restart with today's session key now. [16:21:25] I'll get a tool account set up by the end of the weekend -- I'm at an airport at the moment, so it is a bit distracting just now. [16:22:54] All working now. Thanks again! [16:47:18] 3Wikimedia Labs / 3Other: Cannot switch to VE mode, "Edit beta" tab missing in betalabs - 10https://bugzilla.wikimedia.org/73375 (10Ritu Swain) 3UNCO p:3Unprio s:3major a:3None Created attachment 17119 --> https://bugzilla.wikimedia.org/attachment.cgi?id=17119&action=edit screenshot 1> Link to my... [16:47:25] 3Wikimedia Labs / 3Other: Cannot switch to VE mode, "Edit beta" tab missing in betalabs - 10https://bugzilla.wikimedia.org/73375 (10Ritu Swain) [17:36:12] 3Wikimedia Labs / 3deployment-prep (beta): Bits broken on beta labs: ReferenceError: mw is not defined - 10https://bugzilla.wikimedia.org/73377 (10James Forrester) a:3None [17:42:41] 3Wikimedia Labs / 3deployment-prep (beta): Bits broken on beta labs: ReferenceError: mw is not defined - 10https://bugzilla.wikimedia.org/73377#c3 (10etonkovidova) Failed to load resource: the server responded with a status of 503 (Service Unavailable) http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.o... [18:19:41] bd808: or Coren or someone, any idea what happened to bits in beta labs? https://bugzilla.wikimedia.org/show_bug.cgi?id=73377 it makes VE not work at all [18:20:42] something borked in RL? [18:21:32] * bd808 sees the 503 now [18:21:36] bd808: dunno. Roan doesn't seem to think so, but I don't have a good sense of why bits would be 503 [18:21:55] RL = prod ? [18:22:01] I'll look at the hhvm servers and see if I can spot the problem [18:22:06] mutante: naw, just beta [18:22:31] just bits on beta [18:22:51] ok, btw, i reported an issue with it last night.. [18:22:59] chrismcmahon: Fatal error: Class undefined: ResourceLoaderTemplateModule in /srv/mediawiki/php-master/includes/resourceloader/ResourceLoader.php on line 506 [18:23:08] in phabricator.. hope that was right [18:23:43] oh, closed by Yuvi, ok :) [18:23:55] 3Wikimedia Labs / 3deployment-prep (beta): Bits broken on beta labs: ReferenceError: mw is not defined - 10https://bugzilla.wikimedia.org/73377#c4 (10Bryan Davis) Nov 13 18:22:44 deployment-mediawiki01 hhvm: #012Fatal error: Class undefined: ResourceLoaderTemplateModule in /srv/mediawiki/php-master/includes/... [18:24:00] "(Betalabs has no component here, beta features is something else)" [18:24:06] should it? [18:24:14] It will after the migration [18:24:17] ok [18:24:52] They have been trying not to make projects in Phab that will transition over from bugzilla [18:26:16] Yuvi told me he uses phab, not RT, so i made phab, rejected because tracked in BZ :) [18:26:24] gotcha, ok [18:26:59] the madness will soon end, we hope :) [18:27:17] * bd808 can't wait for not needing to figure out what to put in RT [18:28:31] just everything you now have to put into one of the multiple Ops projects :) [18:30:21] but yea, people will triage "orphaned" tasks into the right projects [18:32:10] 3Wikimedia Labs / 3deployment-prep (beta): Bits broken on beta labs: ReferenceError: mw is not defined - 10https://bugzilla.wikimedia.org/73377#c5 (10Bryan Davis) This commit is breaking ZeroBanner 3Wikimedia Labs / 3deployment-prep (beta): request administrator rights on metawiki beta for user Jmorgan (WMF) - 10https://bugzilla.wikimedia.org/73382 (10Jonathan Morgan) 3NEW p:3Unprio s:3normal a:3None Hi, can I get administrator rights on http://meta.wikimedia.beta.wmflabs.org/ ? This account wi... [19:54:42] 3Wikimedia Labs / 3deployment-prep (beta): request administrator rights on metawiki beta for user Jmorgan (WMF) - 10https://bugzilla.wikimedia.org/73382#c1 (10John F. Lewis) 5NEW>3RESO/FIX a:3John F. Lewis Done. [20:00:10] andrewbogott: ping? little more help with wikitech interface? (my phone arrives tomorrow) [20:00:25] andrewbogott: 1. allocate public IP for shinken project, 2. assign one to shinken-server-01? [20:00:25] YuviPanda: what's up? [20:00:34] ok, one moment... [20:00:34] required for the IRC bot [20:01:50] YuviPanda: actually, you can do step one yourself… log into virt1000 [20:02:10] andrewbogott: aaah, moment [20:02:19] and then # nova quota-update --floating_ips 1 shinken [20:02:49] andrewbogott: yay, done [20:03:04] ok, I will assign [20:03:45] YuviPanda: what DNS name would you like? Or, don't need one? [20:03:50] andrewbogott: no, don't need one [20:03:53] ok. 208.80.155.195 [20:04:06] andrewbogott: hmm, so new connections to the wide internet go through that? [20:04:10] * YuviPanda tries anyway [20:04:15] dunno about outbound [20:06:54] andrewbogott: hmm, do you know anything about ircecho? [20:06:59] not really [20:07:03] hmm [20:07:06] Except, it works ok from tools [20:07:13] hehe [20:07:13] yeah [20:07:14] I think [20:07:17] the package isn't in there for trusty [20:07:18] only precise [20:07:22] can you re-upload? [20:07:27] I think it should 'just work' [20:07:32] or point me to instructions... [20:08:10] https://wikitech.wikimedia.org/wiki/Reprepro [20:08:14] On Carbon [20:08:31] Oh, uh, "Copying between distributions" [20:08:34] should be a one-liner [20:10:22] andrewbogott: ah, yes, I see it [20:10:32] What Could Go Wrong?™ [20:11:02] Coren: are we still having issues? [20:11:14] Betacommand: Not as far as I know. [20:11:55] Coren: see PM [20:12:03] I dont want to publish the IP [20:12:07] on that note, how bad can I fuck it up if I do fuck it up? :) [20:14:40] 3Wikimedia Labs / 3deployment-prep (beta): request administrator rights on metawiki beta for user Jmorgan (WMF) - 10https://bugzilla.wikimedia.org/73382#c2 (10Jonathan Morgan) Thank you, John! [20:15:15] andrewbogott: hmm, so [20:15:15] root@carbon:/srv/wikimedia# reprepro copy precise-wikimedia trusty-wikimedia ircecho [20:15:18] Will not copy as not found: ircecho. [20:15:22] but reprepo ls does list it [20:16:46] YuviPanda: lemme look... [20:17:28] YuviPanda: any chance you have source/dest reversed? [20:17:41] andrewbogott: copying from precise to trusty, no? [20:17:49] andrewbogott: oh, wait... [20:18:27] andrewbogott: yeah, you're right [20:18:30] andrewbogott: that ordering is stupid [20:18:38] andrewbogott: but yeah, done now :) thanks [20:19:01] I think that it source/dest must not be intuitive since there's no consistency in tools about what order they come in... [20:19:26] usually I find it to be source -> dst [20:19:34] Yeah, that seems intuitive to me too [20:19:43] But I'm giving all these other tool writers the benefit of the doubt [20:34:12] 3Wikimedia Labs: add user filter to status page - 10https://bugzilla.wikimedia.org/73384 (10Betacommand) 3NEW p:3Unprio s:3normal a:3None http://tools.wmflabs.org/?status should include a method for filtering results per tool. [20:38:47] !log tools stopping puppet on tools-dev to test shinken [20:38:49] oh wait [20:38:50] Logged the message, Master [20:38:51] that's stupid [20:38:58] !log tools didn't actually stop puppet, need more patches [20:39:00] Logged the message, Master [20:39:18] Need moar patches!!1!one! [20:41:25] 3Wikimedia Labs / 3deployment-prep (beta): deployment-parsoidcache02 is not using betalabs puppetmaster - 10https://bugzilla.wikimedia.org/73357 (10Greg Grossmeier) p:5Unprio>3Normal [21:05:20] Coren | YuviPanda: can I ask you for a favor? [21:05:28] gwicke: Sure, what be up? [21:05:53] we are looking into wikibase indexing, and one of the options we are looking at is Magnus' tool at http://wdq.wmflabs.org/ [21:06:08] could you check how much memory that daemon is currently using? [21:06:30] it's probably in tool labs, which I'm not a member of [21:07:19] No, it has to be its own project given the name. Lemme see what I can dig up. [21:07:43] cool, thanks! [21:11:27] It runs on, as far as I can see, a cluster of 5 instances; it's not obvious that the memory usage of a particular daemon is going to be useful as a metric to anything. From what I can see, it relies on a cassandra backend. [21:11:53] whoa [21:11:54] !log tools disable puppet on tools-dev to check shinken [21:11:57] Logged the message, Master [21:12:01] that's very different from what I had heard so far [21:12:57] You may want to ask manybubbles; he's actually logged into one of those boxen so I trust he has some idea what he's doing. :-) [21:13:01] we all thought it was a single-machine in-memory db [21:13:30] oh, okay -- maybe he's just looking into the same thing [21:13:45] or.. you could also be looking at Nik's titan test cluster [21:13:48] rather than wdq [21:14:00] that's using Cassandra as a backend [21:14:36] which project is this? [21:15:16] Well, the instance that wdq.wmflabs.org points to is named wdq-titan1. The project is 'wikidata-query' [21:15:50] I see.. he probably used the same name then [21:15:57] Coren: When you're ready, I need some help working out why pywikibot in a virtualenv is not found in a cgi even though the hashbang is set. [21:16:23] w930913: I'm pretty much the worse person to ask for python help, though I'll try to figure it out. [21:16:53] Coren: if Nik has access to the host backing wdq.wmflabs.org, then I'll bug him [21:16:58] didn't know that he had [21:17:03] thanks for your help! [21:17:07] Coren: I don't think this is a python problem, that's why I asked. [21:17:09] gwicke: Well, he's currently logged into it right this minute. :-) [21:17:22] okay [21:17:38] is there any huge memory user on that box? [21:18:56] gwicke: There's a pretty big java vm is all [21:19:06] okay, thanks! [21:19:42] w930913: Point me at your problem and I'll see what I can glean. [21:21:08] Coren: /data/project/cluestuff/public_html/wdt/onthisday.py [21:21:42] Which executes fine. [21:21:58] But not when invoked from the web. [21:23:43] Actually, http://tools.wmflabs.org/cluestuff/wdt/onthisday.py has python itself running fine, it complains about not finding the pywikibot module. [21:24:12] : No module named pywikibot [21:24:57] Coren: Yes, but why can't it find it? [21:25:11] It finds it fine when executed normally. [21:25:35] So what changes when it's executed from the web? [21:26:07] w930913: You may be, on the command line, relying on some environment variables set by your shell. [21:27:17] Hmm. [21:28:59] Coren: Reconnected to labs, ran and worked fine. qlogin and ran, worked fine too. [21:29:38] Right, you probably have something required set it your .bashrc or .profile [21:29:54] You'll get those interactively, but the web server won't. [21:30:29] Coren: Those files don't exist. [21:33:13] Hmm, they exist for my personal user, but not the tool. [21:36:18] w930913: Sorry, when you reach the subtleties of venv and python, you're walking outside my reach. That said, : No module named pywikibot [21:36:18] might provide useful guidance, I see it has a section on 'install as a module - virtualenv' [21:37:34] Erm. [21:37:38] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Developing#Setup_pywikibot_on_Labs_.28locally.29 [21:37:42] I mean this ^^ [21:43:49] Coren: It's ignoring the hashbang when it's being executed from the web. [21:44:02] Any reason it would be doing that? :/ [21:47:56] gwicke: wdq-titan1 is mine. I didn't know what magnus's stuff was in wdq as well. I'm should have guessed. [21:49:01] gwicke: Coren was looking at my stuff. wdq.wmflabs.org sure shouldn't be pointing my wdq-titan1. [21:51:33] w930913: Oh! Yes, of course - you haven't actually configured your lighttpd to use /your/ python have you? [21:53:38] w930913: you might be able to get away with this stanza if you always want to use your virtualenv: [21:54:55] https://tools.wmflabs.org/paste/view/4b54a2a6 [21:57:56] manybubbles, gwicke: Oh. Yes. I was confused and looked at the wrong row of a long table. :-( [21:58:07] ah! [21:58:58] Coren: I could do, but surely that's the purpose of the hashbang? [21:59:23] Or is that what the cgi.assign(".py"=>"") does? [22:00:00] w930913: That will have this effect if that's what you want. [22:00:17] w930913: =>"" means "don't invoke an interpreter, call the script itself as an executable" [22:00:32] Which may have security impacts, though it's probably okay. [22:00:52] (In that anything named .py will get silently executed regardless of what it is) [22:01:16] andrewbogott: how would i know who owns this machine: eployment-eventlogging02 [22:01:27] andrewbogott: sorrry: deployment-eventlogging02 [22:01:38] andrewbogott: the code deployed is quite old [22:01:40] nuria__: ownership is an amorphous concept :) But I can check and see who created it. Just a second... [22:01:47] Hmm, I suppose I could just rename it to .cgi, which is already set up for that. [22:02:18] andrewbogott: it is pulling code from http://deployment-bastion.eqiad.wmflabs/eventlogging/EventLogging/. [22:02:32] It was created by hashar [22:02:49] Coren: Well, that will work until I break it. Much thankings. [22:02:55] andrewbogott: which i am not sure is right, code is pretty old, i just updated teh checkout but looks like it is not getting new deployments [22:02:56] Oh, gah the wikitech interface is teh suk. [22:03:09] andrewbogott: could we reboot it? [22:03:23] * w930913 tries to remember what he was trying to do that caused all the problem in the first place. [22:03:47] nuria__: I don't know what it's for or what it's doing or if it's important. You'll have to ask hashar or someone else in the know regarding deployment-prep [22:04:04] andrewbogott: it's running eventlogging for beta env [22:04:25] nuria__: When you said 'could we reboot it' do you mean is it possible? Or advisable? [22:04:26] andrewbogott: rather ... trying to do that ....but not quite... succeeeding [22:04:37] It's certainly possible [22:04:52] andrewbogott: possible, cause i know teh code there and processes are from august [22:04:53] But, I'm not involved in that project so I generally keep my hands out of it. [22:05:13] Are you an admin in deployment-prep? [22:05:26] andrewbogott: so machine is not working properly [22:05:43] andrewbogott: i think so [22:06:17] https://wikitech.wikimedia.org/wiki/Special:NovaInstance [22:06:24] There should be a 'reboot' link next to every instance... [22:07:09] andrewbogott: thank you [22:07:13] np [22:08:12] gwicke: Ah! Found the real one. It's running on wikidata-wdq-mm of wikidata-dev. [22:09:07] gwicke: The server is wd_server and seems to be using roughly 5G of ram [22:09:58] (Well, it's got a vmem footprint of ~9.5g about 5.7 of which are resident) [22:29:50] andrewbogott: can i get more than 8 cpu machine in labs ? [22:30:56] matanya: I'm not sure. It might be possible to temporarily set up a bigger instance type [22:31:09] Why do you ask? [22:31:40] I would like to boost my encoding box with more cpu power [22:32:36] Hm… since it's virtualized, I'm not positive that more vcpus = more actual power [22:33:50] andrewbogott: when i run avconv with 1,2,4,8 cpu i see a differance in execution speed [22:33:54] and a major one [22:38:46] matanya: try now? [22:39:36] yes, i see m.fast :) [22:39:47] specs ok for your purposes? [22:39:59] need, more ram too :) [22:40:08] how much? [22:40:20] 32GB if possible [22:41:00] Our cluster is… just about full already, one like that might cause a host to OOM [22:41:15] We're getting more hardware in a few weeks, then it might be possible. [22:41:34] so 16GB for now andrewbogott ? [22:42:14] Even that is pushing our luck. But, create it and then tell me when it's up and I'll migrate it to virt1006 which has space (but has slightly suspect stability) [22:42:38] And then you're going to clean up your former 16g instance? [22:42:49] Any chance you can do that first, so there's room? [22:43:41] matanya: what project is this, btw? [22:44:20] video project [22:44:34] what's the actual project name? I'm making the flavor project-specific. [22:44:45] just 'video'? [22:45:11] https://wikitech.wikimedia.org/wiki/Nova_Resource:Video [22:45:35] I can delete that project, if you need the resources [22:45:44] *instance [22:46:50] Yeah, if it won't break anything, best to delete the old one first [22:47:33] ok, project 'video' now has a custom flavor type, m1.fast. [22:47:35] 3Wikimedia Labs / 3deployment-prep (beta): deployment-parsoidcache02 is not using betalabs puppetmaster - 10https://bugzilla.wikimedia.org/73357#c1 (10Antoine "hashar" Musso (WMF)) 5NEW>3ASSI a:3Antoine "hashar" Musso (WMF) Indeed: deployment-parsoidcache02:~$ grep server /etc/puppet/puppet.conf serve... [22:47:43] i'll just backup the list of packages to make sure i have all my deps [22:47:46] Let me know when it's up and running and I'll move it to a host where it'll have plenty of elbow room. [22:49:03] my ~ doesn't go away if i delete the instance right andrewbogott ? [22:49:04] andrewbogott: Come April 1st, can we have a photo of a goat in the data centre, with the caption " installing more RAM"? [22:49:30] matanya: it shouldn't, but best to make sure it's mounted on a shared volume [22:49:35] * matanya would just be happy with a GPU :) [22:49:49] w930913: why wait until April? Photoshop! [22:50:12] it is on labstore [22:50:22] seems to me like a shared volume [22:50:48] yep, that's safe then [22:51:00] /data/project/video also [22:52:02] encoding01 was recliamed [22:52:22] thanks [22:53:12] andrewbogott: Because it makes for an appropriate April fools. :) [23:00:53] matanya: looks like your new instance is puppetized… mind if I move it now? [23:01:07] It may freeze for a minute or so while in transit [23:01:12] no, feel free. does it matter if i'm in ? [23:01:21] It's ok if you're logged in [23:01:30] please do :) [23:01:33] Actually, this is a new thing which I haven't done much of yet, interested to see how it works for you [23:01:40] I figure if the instance dies… you're probably not attached. [23:03:00] feel free to use me as a beta tester [23:03:09] as long as my data is safe ... [23:04:27] 3Wikimedia Labs / 3deployment-prep (beta): deployment-parsoidcache02 is not using betalabs puppetmaster - 10https://bugzilla.wikimedia.org/73357#c2 (10Antoine "hashar" Musso (WMF)) I have upgraded varnish while at it (apt-get upgrade + restart of the varnish* services): Unpacking varnish (3.0.6plus~x-wm3)... [23:13:36] matanya: it still says 'migrating' -- this is slow! [23:13:39] Is the instance still responsive? [23:13:54] no, for several minutes now [23:14:14] if you had horizon you could see the progress ... [23:15:50] maybe :) [23:26:26] matanya: ok, migratin is done. Instance look ok? [23:26:33] I bet your ssh session timed out... [23:26:38] but in theory the instance doesn't know it was moved [23:26:57] channel 0: open failed: connect failed: No route to host [23:26:57] stdio forwarding failed [23:26:58] ssh_exchange_identification: Connection closed by remote host [23:27:38] well that's not right [23:31:39] dang. I"m going to try a reboot [23:34:41] ok [23:35:04] Seems to have survived the reboot. Maybe I was just too impatient. [23:35:26] anyway, you should be in business -- let me know if you run into trouble [23:35:49] I'm in! thank you! :) [23:36:19] 3Wikimedia Labs / 3deployment-prep (beta): deployment-parsoidcache02 is not using betalabs puppetmaster - 10https://bugzilla.wikimedia.org/73357#c3 (10Antoine "hashar" Musso (WMF)) 5ASSI>3RESO/FIX deployment-parsoidcache02 is now using betalabs puppetmaster and salt master. Varnish has been upgraded. I... [23:37:18] 3Wikimedia Labs / 3Other: Cannot switch to VE mode, "Edit beta" tab missing in betalabs - 10https://bugzilla.wikimedia.org/73375#c1 (10etonkovidova) 5UNCO>3RESO/DUP Was resolved as bug 73377 *** This bug has been marked as a duplicate of bug 73377 *** [23:40:54] Coren: thanks! [23:42:34] gwicke: Sorry I led you astray. We don't have an interface to do the reverse mapping so I had to look inside the redis db, and I msiread the first time (made understandable because it also made sense that wdq.* would proxy for wikidata-query) [23:55:35] Coren: no worries ;) [23:56:09] added this data in https://www.mediawiki.org/wiki/Wikibase/Indexing [23:56:42] huge change, thank you andrewbogott ! [23:57:52] my metrics show about 1 frame/cpu/sec, so i just jumped for 8 frames a sec to 12 frames a sec [23:57:59] *from [23:58:59] !seen sandaru [23:58:59] you probably wanted to use @seen [23:59:03] @seen sandaru [23:59:03] mutante: Last time I saw sandaru they were leaving the channel #mediawiki at 11/12/2014 10:30:52 AM (1d13h28m11s ago) [23:59:23] too bad mutante :) [23:59:41] do you know sandaru? [23:59:49] nope [23:59:53] https://wikitech.wikimedia.org/wiki/User:Sandaru