[00:00:50] (03PS11) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [00:04:46] TimStarling: have you considering using --bwlimit with no -F limit? [00:06:12] there could be other limitations, like memory usage [00:06:16] it's otherwise annoying to handle server-specific scap tasks that take a bit of time [00:06:53] if you run sync-common manually on a single server, presumably you wouldn't want --bwlimit, right? [00:07:47] it could be an argument...would be easier if it wasn't bash [00:08:33] feel free to run it with high fanout to see what happens [00:10:26] TimStarling: F60? ;) [00:11:38] well, wc -l mediawiki-installation is 431 [00:12:05] so -F108 would let you do it in 4 batches... [00:12:27] -F216 in two batches... [00:13:22] what I would want to know is: [00:13:38] * what fanout do you need to saturate the network [00:13:40] (03PS12) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [00:13:51] * what fanout do you need to saturate the proxy CPU [00:15:46] <^d> AaronSchulz: Do abandoned jobs keep things from claiming new ones to run? [00:16:00] no [00:16:16] <^d> http://p.defau.lt/?k4D3qKJTqm1fTJ7bHghDYw - current jobs for enwiki [00:16:59] (03PS13) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [00:17:24] TimStarling: for later: http://www.psc.edu/index.php/hpn-ssh (""OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links. [00:17:25] "Modifying the ssh code to allow the buffers to be defined at run time eliminates this bottleneck. We have created a patch that will remove the bottlenecks in OpenSSH and is fully interoperable with other servers and clients." [00:17:54] ori: I think I just finished addressing yours and faidon's comments; so if you (when you have a chance) take a peek at https://gerrit.wikimedia.org/r/102352 to see what other silly stuff I did; it would be much appreciated [00:18:20] mwalker: ok, looking [00:21:02] TimStarling: I found out about it via "High Performance Bulk Data Transfer", a presentation from the Lawrence Berkeley National Laboratory in the US. It has some useful info about rsync's limitations and what to do about them. [00:21:10] er, link: http://fasterdata.es.net/assets/fasterdata/JT-201010.pdf [00:26:26] mwalker: Could not find template 'ocg/mw-ocg-service.js.erb' [00:26:34] it's mw-ocg-frontend.js.erb [00:27:06] and this is why I shouldn't change my mind about what to call things 15 times [00:27:35] yes, change your mind either 14 or 16 times, but not 15 [00:27:46] hehe; the evenness is important :) [00:28:22] (03PS14) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [00:28:29] ^ see, even patch number! :p [00:32:46] !log kaldari synchronized php-1.23wmf9/extensions/MobileFrontend 'updating MobileFrontend in wmf9' [00:32:53] Logged the message, Master [00:33:50] (03PS1) 10Springle: warm up db1055 and db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107274 [00:34:28] (03CR) 10Springle: [C: 032] warm up db1055 and db1023 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107274 (owner: 10Springle) [00:35:39] !log springle synchronized wmf-config/db-eqiad.php 'warm up db1055 and db1023' [00:35:45] Logged the message, Master [01:19:52] !log kaldari synchronized php-1.23wmf10/extensions/MobileFrontend 'updating MobileFrontend in wmf10' [01:19:59] Logged the message, Master [01:29:10] mwalker: sorry, I got side-tracked. I'll finish reviewing later. [01:29:33] ori: no worries! I appreciate your time when you can give it [01:30:15] if you want to test it more thoroughly yourself, my strategy is simple: i comment out the few wmf-specific things (monitor_group, system_role) and cp -R the module directory to puppet/modules in the vagrant repo [01:31:01] then i edit puppet/manifests/roles.pp to add https://dpaste.de/bLn5/raw [01:31:13] then 'vagrant enable-role ocg' followed by 'vagrant provision' [01:32:17] file { $temp_dir: path => $temp_dir } is awkward, btw. the resource name is the path by default, so you don't need to specify it manually. and, more importantly, it's bizarre to make the temp dir configurable [01:32:36] just make it /var/run/ocg or something and force people to live with it; it's not the sort of configurability that people actually need IMO [01:32:38] anyways, back later [01:32:43] ^ mwalker [01:33:09] I'm pretty sure I tried just file { $temp_dir } and it complained; and I made it configurable because the temp dir should live on an SSD; so it would be nice to configure that without playing with mount points [01:35:09] (03CR) 10Mwalker: "(05:32:17 PM) ori: file { $temp_dir: path => $temp_dir } is awkward, btw. the resource name is the path by default, so you don't need to s" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [02:20:29] (03PS1) 10Springle: depool db1040 and db1049 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107310 [02:22:43] (03PS2) 10Springle: depool db1040 and db1049 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107310 [02:23:14] (03CR) 10Springle: [C: 032] depool db1040 and db1049 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107310 (owner: 10Springle) [02:23:23] (03Merged) 10jenkins-bot: depool db1040 and db1049 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107310 (owner: 10Springle) [02:24:22] !log springle synchronized wmf-config/db-eqiad.php 'depool db1040 and db1049' [02:25:24] !log LocalisationUpdate completed (1.23wmf9) at Tue Jan 14 02:25:24 UTC 2014 [02:29:22] PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:30:49] (03CR) 10MZMcBride: "Thanks for including example output in a comment, Nemo." [operations/puppet] - 10https://gerrit.wikimedia.org/r/106892 (owner: 10Tinaj1234) [02:38:12] RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.194 second response time [02:38:50] !log wikitech down. restarted apache on virt0 [02:39:11] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [02:39:21] !log synchronized wmf-config/db-eqiad.php [02:39:41] still doesn't seem happy [02:41:21] PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:43:52] !log wikitech down. restarted apache on virt0. phusion_passenger exception + MaxClients hit [02:44:02] * springle waits hopefully [02:44:11] RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.160 second response time [02:46:16] looks to me like a lot of puppet calls [02:48:14] It's kind of painful to watch people try to log info to wikitech when wikitech is down. [02:48:39] yep [02:48:48] we should really split the labs puppetmaster apart from wikitech [02:48:56] really it would be nice for wikitech to just be in the cluster [02:49:43] also, the way we log is fucking stupid [02:49:59] !log LocalisationUpdate completed (1.23wmf10) at Tue Jan 14 02:49:59 UTC 2014 [02:49:59] we have logstash and kibana now, don't we? [02:50:00] Yes. :-) [02:50:07] I file a bug about that general issue. [02:50:09] why don't we log to that, and have a canned search? [02:50:19] You must be new around here [02:50:26] :) [02:50:30] Who's kibana? New intern? [02:50:40] No [02:50:42] kibana is an interface for logstash [02:51:04] Let me find the bug. [02:51:11] I suspect when the infrastructure is a little bit more stablised, it shouldn't be too much work to improve the situation greatly [02:51:14] https://bugzilla.wikimedia.org/show_bug.cgi?id=57343 [02:51:33] Gloria: yeah, I saw the bug [02:51:48] If you think it should use kibana and logstash, you should say so. ;-) [02:51:52] Direction helps a bug like that. [02:55:35] ok, i'm not crazy... [02:55:40] can't log into wikitech [02:58:31] springle has been poking at it, he might need some help. I'm reluctant to wake people up for it though [02:58:43] poking is the right word [02:59:08] ah [02:59:15] /a is full on virt0 [02:59:57] I'm deleting some backups [03:00:08] seems keeping 7 days of backups isn't doable anymore [03:00:22] thanks Ryan_Lane :) [03:00:30] that's mostly thanks to our stupid logging strategy [03:00:47] since we have like 5G of logs [03:01:02] yw [03:04:25] !log synchronized wmf-config/db-eqiad.php 'depool db1040 and db1049' [03:04:29] * springle tries again [03:04:33] Logged the message, Master [03:04:38] \o/ [03:08:34] !log xtrabackup clone db1050 to db1049 [03:08:40] Logged the message, Master [03:11:44] andrewbogott: howdy. any idea when you want to deploy the changes I pushed in for OSM? [03:11:59] it looks like most of it was reviewed? [03:12:45] Ryan_Lane: You're talking about https://gerrit.wikimedia.org/r/104144 and https://gerrit.wikimedia.org/r/104129? [03:12:49] Or did I miss some merges? [03:14:44] and https://gerrit.wikimedia.org/r/#/c/104320/ [03:14:58] ah. missed you as a reviewer [03:15:50] (03PS1) 10Springle: switch db1049 to innodb_file_per_table=1 during reclone [operations/puppet] - 10https://gerrit.wikimedia.org/r/107315 [03:17:07] (03CR) 10Springle: [C: 032] switch db1049 to innodb_file_per_table=1 during reclone [operations/puppet] - 10https://gerrit.wikimedia.org/r/107315 (owner: 10Springle) [03:17:25] * andrewbogott reads [03:17:55] Ryan_Lane: I can schedule a roll-out for later this week… do you want to be around to watch when that happens? [03:18:08] yep [03:18:25] it does look like my keystone change could be fixed a little [03:18:30] ok… what time of day would suit you? [03:19:04] after 6pm PDT is best for me [03:19:10] I can do other times if necessary [03:19:35] Great -- I'm in an exceptionally difficult TZ right now (+8) so SF evening is ideal for me. [03:20:02] ah. great [03:21:55] greg-g, you still working? If so can you walk me through scheduling a labs deployment? [03:24:21] heh. I never scheduled any of that :D [03:25:21] Me neither but I caused a brief wikitech outage last week due to incompatible extension versions… so I'm a bit edgy just now. [03:25:53] heh [03:26:05] I always update to the newest branch [03:26:28] Not true -- we're using an old version of SMW and related [03:26:33] ah. true [03:26:38] we should probably upgrade SMW [03:26:39] The tip is incompatible with the core tip [03:27:00] Or was as of last week, I haven't followed up yet. [03:27:39] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Jan 14 03:27:39 UTC 2014 [03:27:44] Logged the message, Master [03:37:48] we "fixed" core by making the variable public again... [03:39:03] Reedy: Yeah, that fixed backwards compatibility with the old version of SMW. [03:39:11] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [03:39:17] I don't remember what the story was with updating SMW… maybe just a dependency cascade [04:57:32] springle, this TZ is so much quieter than CST! Do you read the whole US day's IRC transcripts when you wake up, or just live in comfortable ignorance? [04:59:11] i usually page back a ways, but ignorance is nice too :) [05:00:59] Perhaps my coding productivity will skyrocket with things so peaceful? [05:01:01] Perhaps not. [05:01:16] where are you? [05:01:45] Singapore. Still very far from you, but you're my closest Ops neighbor :) [05:02:31] 2h away [05:02:47] Plus many many degrees of latitude. [05:03:03] :) [05:03:27] wow, 4000 miles, that's /way/ more than I would've guessed. [05:04:06] big desert in between [05:04:17] yep [05:10:57] * springle plays whac-a-mole with slow queries [05:20:48] !log xtrabackup clone db1022 to db1040 [05:20:54] Logged the message, Master [05:36:57] (03CR) 10Faidon Liambotis: [C: 04-1] "More comments :) It's shaping up nicely though!" (0319 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [05:38:20] mwalker: ^ [05:38:32] yep yep; wandering through the comments now [05:41:19] paravoid: the reason I'm requiring the package install to notify on the service is because it doesn't seem safe to me to start a service without it's dependencies [05:42:14] that's a require/before relationship [05:42:48] before... that's probably the keyword I was looking for [05:43:38] http://docs.puppetlabs.com/learning/ordering.html & http://docs.puppetlabs.com/puppet/2.7/reference/lang_relationships.html [05:43:50] also; I gave the logs the wikidev group so that I can read them for debug [06:18:13] (03CR) 10Mwalker: Collection Renderer (Now a module!) (0319 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [06:19:29] (03PS15) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [06:37:40] (03CR) 10Ori.livneh: Collection Renderer (Now a module!) (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [06:38:43] paravoid: proposed convention: one 'include' per line [06:38:53] yeah, it makes for less dirty diffs [06:39:33] adding to style guide [07:06:15] (03CR) 10Mwalker: Collection Renderer (Now a module!) (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [07:06:51] (03PS16) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [07:21:20] mwalker: Invalid parameter user at /tmp/vagrant-puppet-1/modules-0/ocg/manifests/init.pp:103 [07:21:25] should be 'owner' [07:21:51] mwalker: also: warning: Dynamic lookup of $hostname at /tmp/vagrant-puppet-1/modules-0/ocg/manifests/init.pp:14 is deprecated. Support will be removed in Puppet 2.8. Use a fully-qualified variable name (e.g., $classname::variable) or parameterized classes. [07:21:57] should be '$::hostname' [07:22:35] ori, how well do you know XSS? [07:22:59] i know the basics, not some of the fancy new techniques [07:23:13] i need to evaluate if something is a live XSS [07:24:27] Krinkle is in charge of front-end security, I think? Anyways, feel free to PM. An e-mail to security@ is usually a good idea if the suspicion is credible. [07:35:41] (03CR) 10Ori.livneh: [C: 04-1] Collection Renderer (Now a module!) (038 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [07:37:13] yurik: I can look at it if you need help [07:37:36] csteipp, i'm about to email to to security@ [07:37:43] is that the right approach? [07:37:55] i can cc you [07:37:58] If you're pretty sure it's an issue, yet [07:38:00] yep [07:38:07] csteipp, nope, i'm not :) [07:38:09] I'm on security@, so just that is fine [07:38:32] If not, email me directly or open a bug is best [07:38:47] csteipp, ori , just sent it to you two [07:55:59] (03CR) 10Mwalker: Collection Renderer (Now a module!) (038 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [07:56:25] (03PS17) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [07:56:53] ^ shoot; it's an odd PS number; that definitely means that ori is going to find more bugs :[ [07:59:07] mwalker: if you insist: config.redis.port is a string, but config.frontend.port is an int [07:59:38] but the modules applies correctly [08:00:11] yay! thanks :) also -- how hard would it be to recreate your vagrant stuff for ubuntu 13.10; paravoi_ has given us a tentative green light to deploy on that [08:00:41] (to avoid having to backport lots of stuff) [08:01:26] (03PS18) 10Mwalker: Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 [08:01:51] mwalker: you'd have to change these two lines in Vagrantfile: [08:01:54] config.vm.box = 'precise-cloud' [08:01:54] config.vm.box_url = 'https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-amd64-vagrant-disk1.box' [08:02:14] there are 13.10 images: https://cloud-images.ubuntu.com/vagrant/saucy/current/ [08:02:31] ok! task for tomorrow right there! (also; that is significantly easier than I thought you were going to say...) [08:02:57] * mwalker adds to list of things to do [08:03:14] let me know if you run into issues, i haven't tried provisioning on saucy [08:03:45] (03CR) 10Ori.livneh: [C: 031] Collection Renderer (Now a module!) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [08:04:21] will do! /me *crosses fingers* for no big huge issues [08:04:50] I suspect it's going to come down to if our production puppet makes 12.04 specific assumptions [08:05:22] we'll figure it out [08:05:27] we need to anyway [08:08:52] paravoid: since you're around; either myself or max will play with the vagrant side of things tomorrow -- but who do I need to talk to to get a saucy install on rhodium? [08:09:04] or; what's the process for that? [08:09:12] jeff [08:09:31] saucy ships with puppet 3, interesting [08:09:39] he has offered to help with the ops side of things for this I think [08:09:57] he has; he's just a bit swamped with some fundraising stuff at present [08:11:07] but, that's good to hear; I was worried I was going to have to take some of Chris's time for an install in front of the console in the DC [08:13:09] ideally we'd start with a labs instance [08:13:18] but testing on rhodium is also possible [08:13:21] I don't particularly mind [08:14:45] ori: yeah, that's going to be fun, esp. since you're supposed they promise compatibility only for master > client scenarios [08:14:54] we might have to do puppet 3.x on the server first [08:14:59] akosiaris: weren't you working on this? [08:15:23] and apergos finding/fixing scoping issues I think? [08:15:39] max tested briefly today in labs on 13.10; but it wasn't the most ideal install (he started from 12.04 and then do-release-upgrade'd to 13.10 [08:15:39] more or less [08:16:02] i guess it could easily be added to the set of images available on labs [08:16:21] yes, that's right [08:17:21] mwalker: what's the instance name? [08:18:27] ori: not sure; it doesn't look like he created it in the collection_alt_renderer project [08:19:32] he also said he only got a couple minutes of playtime on it before it bricked [08:19:40] (03PS1) 10Matanya: dynamicproxy: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107339 [08:20:42] ... so according to https://wikitech.wikimedia.org/wiki/OpenStack#Building_new_images someone with shell on the openstack controller would need to at least import the image -- and most likely to have created it... [08:33:38] (03PS1) 10Hydriz: Move WikimediaIncubator extension call to be after Scribunto [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107340 [08:34:31] (03PS1) 10Matanya: download: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107341 [08:35:13] (03CR) 10Hydriz: "See also: If0f030674c728d39876a251a7c35613b6f88d2bf" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107340 (owner: 10Hydriz) [08:35:43] akosiaris, apergos, so, what's the progress then? :) [08:37:42] on the puppet 3 front ? On my part stalled due to OSM. [08:38:14] ok [08:39:13] I'm going to ask someone else to review https://gerrit.wikimedia.org/r/#/c/97007/ (I need to know if this is the right approach before I go on and apply it to several other changes) [08:39:21] ryan has been too busy or whatever [08:39:35] ori, good perf chat, listening to it now :) [08:39:36] suggestions for who to poke? [08:40:10] no that's not great [08:40:21] but leave ryan/mike to handle the openstack refactoring [08:40:48] the openstack classes are a bit complicated, I wouldn't dare to touch them [08:41:23] well I still need someone to say 'yes this is a good way to fix that sort of scope issue' or 'no, this is a better way' (even if they just comment on the changeset) [08:41:54] it's not a good way to fix this scope issue [08:42:05] a better way would be to refactor the openstack classes significantly [08:42:11] I see [08:42:51] well there will be a few more of those [08:43:47] apergos: why does the openstack version impact the scope? [08:45:29] I don't quite follow the question [08:56:44] getting "Pool queue is full" fairly regularly on the JSON based search API. [08:56:55] Not sure if I should file a bug for that or just mention it here. [08:57:17] as I doubt its a bug in the software, probably more to do with capacity. [08:58:13] sdehaan: file it anyway. Worst case it gets closed [08:58:18] we're hitting that API harder than usual with the Wikipedia Text app in Kenya because the local mobile network operator did a big media push. [08:58:21] yuvipanda: thanks [08:58:34] yuvipanda: http://bugzilla.wikimedia.org/? [08:58:40] or is there a more relevant one? [08:59:01] sdehaan: that's good enough. [09:05:44] yuvipanda: thanks, https://bugzilla.wikimedia.org/show_bug.cgi?id=60032 filed. [09:06:34] sdehaan: sweet. let do a bit of fiddling so that the proper people are notified [09:07:18] yuvipanda: thanks, appreciate it. fwiw our side of the issue is here https://github.com/praekelt/vumi-wikipedia/issues/28 [09:07:28] sdehaan: :) [09:07:57] sdehaan: there were some issues with the poolcounters yesterday, this might be the cause of that [09:08:00] err [09:08:03] that might be the cause of this [09:08:42] kk, I have a graphite graph showing the timeouts & problems. Has UTC timestamps, let me know if that's helpful to attach. [09:12:20] sdehaan: might as well do it :) [09:12:26] sdehaan: more info is usually nice anyway [09:12:31] sdehaan: the search folks are all asleep now (SF time) [09:15:22] done [09:25:06] (03PS1) 10Matanya: deployment: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107343 [09:58:39] (03PS1) 10Matanya: cpufrequtils: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107346 [10:14:10] (03PS1) 10Matanya: contint: lint packages.pp before spliting up [operations/puppet] - 10https://gerrit.wikimedia.org/r/107347 [10:27:07] (03CR) 10Faidon Liambotis: [C: 032] Varnish: don't mobile redirect www.$project.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/89879 (owner: 10JanZerebecki) [10:27:19] (03CR) 10Faidon Liambotis: [C: 032] varnish: simplify the mobile redirect regexp [operations/puppet] - 10https://gerrit.wikimedia.org/r/106669 (owner: 10Faidon Liambotis) [10:28:13] sdehaan: what wiki? [10:28:28] Nemo_bis: english [10:28:32] the normal/old search constantly gives pool queue full errors [10:28:38] English Wikipedia I suppose you mean [10:29:09] I don't know if there's a parameter to get the Cirrus search results via API too [10:29:58] Nemo_bis: cirrus isn't supposed to be used in production yet, I think [10:30:02] Nemo_bis: yeah English Wikipedia [10:30:19] yurik mentioned something about a new search backend, is that the cirrus stuff? [10:30:25] yuvipanda: no idea what you mean in production, it's already default on at least 2 top10 Wikipedias [10:30:31] yes [10:31:15] well, not the default for enwiki, at the least :) [10:31:22] has the search api itself changed due to the backend changes? [10:31:30] the old search is haunted (quot.), can't really expect it to work [10:31:48] no, the search API won't be affected by the backend [10:31:50] well, it 'mostly' works :) [10:31:54] but yeah, haunted [10:33:32] sdehaan: I suggest you to watchlist https://www.mediawiki.org/wiki/Search#Wikis to be notified when the backend changes on en.wiki too [10:34:43] thanks [10:39:15] gah [10:39:24] (03PS1) 10Faidon Liambotis: Fix mobile redirect breakage [operations/puppet] - 10https://gerrit.wikimedia.org/r/107349 [10:39:48] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Fix mobile redirect breakage [operations/puppet] - 10https://gerrit.wikimedia.org/r/107349 (owner: 10Faidon Liambotis) [11:13:13] (03Abandoned) 10Matthias Mullie: Fix notice due to incorrect capitalization [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107186 (owner: 10Matthias Mullie) [11:39:59] (03PS1) 10Springle: warm up db1049 in s1 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107354 [11:40:35] (03CR) 10Springle: [C: 032] warm up db1049 in s1 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107354 (owner: 10Springle) [11:40:43] (03Merged) 10jenkins-bot: warm up db1049 in s1 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107354 (owner: 10Springle) [11:41:39] !log springle synchronized wmf-config/db-eqiad.php 'warm up db1049' [11:41:45] Logged the message, Master [11:45:26] hrmm.. too warm too quick [11:49:33] When is the first dump of 2014 ready for Wikidata ? [11:55:44] springle: saw my mail? [11:56:36] paravoid: yep. sorry, meant to reply earlier [11:56:47] no worries, it's nothing urgent obviously [11:59:48] (03PS1) 10Matanya: base: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107355 [12:00:22] (03CR) 10jenkins-bot: [V: 04-1] base: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107355 (owner: 10Matanya) [12:00:42] the good news it can actually be found :) http://dumps.wikimedia.org/wikidatawiki/20140106/ [12:01:48] (03PS2) 10Matanya: base: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107355 [12:02:23] (03CR) 10jenkins-bot: [V: 04-1] base: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107355 (owner: 10Matanya) [12:05:46] Nemo_bis: sorry, I'm going to need some explanation on how to watchlist [12:07:01] (03PS3) 10Matanya: base: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107355 [12:07:07] typos :/ [12:11:27] sdehaan: login/register and click the star [12:11:34] doh [12:11:54] https://meta.wikimedia.org/wiki/Help:Watching_pages [12:14:17] thanks [12:29:15] (03PS1) 10Matanya: bacula: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107356 [12:39:31] (03CR) 10Alexandros Kosiaris: [C: 032] bacula: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107356 (owner: 10Matanya) [12:44:05] (03PS1) 10Matanya: apt: lint clean [operations/puppet] - 10https://gerrit.wikimedia.org/r/107358 [12:45:00] and now my project is offically over. all modules are linted :) [12:45:51] what's next?:P [12:46:08] reforming webserver [12:46:46] i would like some ops input before. got from andrewbogott and i ran into some issues [12:47:02] but that is the plan [13:05:24] (03CR) 10Alexandros Kosiaris: [C: 031] "LGTM, I also like the new format. The flag is existent at least as far back as hardy so it is supported across the cluster. I personally c" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106892 (owner: 10Tinaj1234) [13:09:50] akosiaris: should manifests/mail.pp become a module? [13:23:11] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [14:00:08] (03PS1) 10Faidon Liambotis: Varnish: don't inadvertently convert 500s to 503s [operations/puppet] - 10https://gerrit.wikimedia.org/r/107364 [14:00:40] mark: ^^^ we might need to tune this further -- I think 4 is too much [14:00:50] I'll merge for now [14:01:06] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Varnish: don't inadvertently convert 500s to 503s [operations/puppet] - 10https://gerrit.wikimedia.org/r/107364 (owner: 10Faidon Liambotis) [14:07:47] (03PS1) 10Faidon Liambotis: Varnish: brown paper bag fix for return(restart) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107365 [14:09:02] (03PS2) 10Faidon Liambotis: Varnish: brown paper bag fix for return(restart) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107365 [14:09:40] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Varnish: brown paper bag fix for return(restart) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107365 (owner: 10Faidon Liambotis) [14:11:49] (03PS1) 10Springle: This was turned off for wikidata queries on wb_terms. However it caused a parformance hit for an ipblocks query on enwiki. Latter trumps former. Revert and find another way... [operations/puppet] - 10https://gerrit.wikimedia.org/r/107366 [14:13:17] (03CR) 10Springle: [C: 032] This was turned off for wikidata queries on wb_terms. However it caused a parformance hit for an ipblocks query on enwiki. Latter trumps for [operations/puppet] - 10https://gerrit.wikimedia.org/r/107366 (owner: 10Springle) [14:17:06] (03PS1) 10Faidon Liambotis: gdash: remove logbase from reqstats.5xx [operations/puppet] - 10https://gerrit.wikimedia.org/r/107367 [14:17:36] (03CR) 10Faidon Liambotis: [C: 032 V: 032] gdash: remove logbase from reqstats.5xx [operations/puppet] - 10https://gerrit.wikimedia.org/r/107367 (owner: 10Faidon Liambotis) [14:20:11] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [14:23:08] paravoid: looks like only bugzilla still talking to db9 [14:23:20] oh dear [14:23:46] that migration is in progess though, afaik. mutante may know more [14:23:58] yeah [14:24:38] (03CR) 10Nikerabbit: [C: 031] Changed date format in l10nupdate-1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/106892 (owner: 10Tinaj1234) [14:29:24] (03PS1) 10Springle: db1049 to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107370 [14:29:54] (03CR) 10Springle: [C: 032] db1049 to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107370 (owner: 10Springle) [14:30:02] (03Merged) 10jenkins-bot: db1049 to full steam [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107370 (owner: 10Springle) [14:30:55] !log springle synchronized wmf-config/db-eqiad.php 'db1049 to full steam' [14:31:01] Logged the message, Master [14:32:44] hashar: hello there [14:33:06] i was wondring if it is possible to add puppet-lint to jenkins tests [14:33:34] I doubt all of our manifests pass that [14:34:07] Jenkins tests can be marked "non-voting". [14:34:12] I know [14:34:15] I fail to see their point [14:34:53] i guess most of them don't but i found out not everybody know about them [14:35:00] (03PS1) 10Springle: reassign db1040 to s4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/107371 [14:37:21] (03CR) 10Springle: [C: 032] reassign db1040 to s4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/107371 (owner: 10Springle) [14:37:41] matanya: there is a rake target for puppet-lint [14:37:55] matanya: try rake -T [14:37:55] [14:38:03] rake lint [14:38:14] (03PS1) 10Aude: Enable Wikibase Client on Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107372 [14:38:20] it calls puppet linter and disable a bunch of tests [14:38:31] !log xtrabackup clone db1042 to db1040 [14:38:37] Logged the message, Master [14:39:06] I think they're useful for when you don't want to enforce puppet-lint yet, but want to migrate step by step. [14:39:21] (03CR) 10Aude: [C: 04-1] "*important* verify wikisource has sites table (and populated) before deploying this!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107372 (owner: 10Aude) [14:42:15] thanks hashar [15:04:16] !log Jenkins: uninstalled phpunit has provided by PEAR (pear uninstall pear.phpunit.de/PHPUnit) We are using Wikimedia deployment system now (integration/phpunit) [15:04:23] Logged the message, Master [15:15:28] (03PS1) 10Nemo bis: [Planet] Fix Guillaume's blog feed URL [operations/puppet] - 10https://gerrit.wikimedia.org/r/107376 [15:15:37] (03CR) 10jenkins-bot: [V: 04-1] [Planet] Fix Guillaume's blog feed URL [operations/puppet] - 10https://gerrit.wikimedia.org/r/107376 (owner: 10Nemo bis) [15:50:28] more poolcounter fun: https://bugzilla.wikimedia.org/show_bug.cgi?id=60032 [16:04:37] just what I wanted. [16:06:09] added yurik because he told me about that issue on Friday [16:29:51] (03PS1) 10Ottomata: Adding parameters to set root database (mysql) user and pass [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/107387 [16:30:40] (03CR) 10Ottomata: [C: 032 V: 032] Adding parameters to set root database (mysql) user and pass [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/107387 (owner: 10Ottomata) [16:40:51] (03PS4) 10Ottomata: added basic hbase support [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [16:43:12] (03CR) 10Physikerwelt: "by the way you can see how to set up zookeeper here" [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [17:03:51] ksnider, Coren: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration [17:10:49] (03CR) 10Ottomata: "Yeah the zookeeper issue is a tough one. Our problem is that we prefer to use Debian/Ubuntu packages where possible. The configuration s" [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/99381 (owner: 10Physikerwelt) [17:22:45] !log reedy updated /a/common to {{Gerrit|I8d1013484}}: db1049 to full steam [17:22:49] (03PS1) 10Reedy: All non wikipedias to 1.23wmf10 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/107390 [17:22:51] Logged the message, Master [17:32:27] hi, just saw the mail about wikitech outage. looks like we have no disk space monitoring for that node in icinga? [17:33:07] also i don't understand why wikitech is hosted on a node called 'virt0'? [17:34:18] jgage, because it's tightly related to labs? [17:35:09] jgage: 1) we should, 2) it's on a virtual machine just so it is separate from the main cluster (ie: if the main cluster goes down, we still want to see the Server Admin Log and other documentation) [17:35:29] it's not on a virtual machine [17:35:32] oh [17:35:37] well, a separate machien [17:35:41] virt0 is the labs "master", used to be labsconsole [17:35:41] (why call it virt?) [17:35:45] ahhh [17:35:53] labsconsole merged with wikitech [17:35:57] * greg-g nods [17:36:13] it still is the frontend to openstack nova [17:36:53] i see, ok. i will educate myself about how to add a disk space check. [17:36:54] virt0 (in contrary to virtN, N > 1) doesn't do virtualization at all actually, it's just control stuff [17:37:02]