[00:07:38] jeremyb: [00:07:44] LeslieCarr: [00:07:46] oops, forgot to type the content - figured it out :) [00:08:02] ooooh, works [00:08:04] what was it? [00:08:51] so apache2 wasn't set to subscribe to that file [00:09:01] so it didn't know to reload when the file changed [00:09:07] ahhh [00:09:10] so i should fix that [00:14:09] mutante: I assigned a bug to you. [00:14:25] You have two Bugzilla accounts, I guess. [00:15:00] Brooke: the ssl think? i was thinking about taking it [00:15:08] If you want it, go for it. [00:15:37] https://bugzilla.wikimedia.org/show_bug.cgi?id=31369 [00:15:41] i researched it a month or two ago and then never did it. /me digs up the research again [00:16:01] I think you can just use relative URLs? [00:16:35] how do you mean? [00:16:57] hashar made a comment that suggested that... [00:16:58] RewriteRule ^/(.*)$ //www.mediawiki.org/$1 [R=301,L] [00:16:59] would work. [00:17:18] huh [00:17:21] Can Location: headers be protocol relative? [00:17:25] i can test that [00:17:27] I assume so, but I have no idea. [00:17:30] uhhh, idk [00:17:48] but maybe apache will absolutify it anyway [00:19:00] Is absolutify a word? [00:19:08] i was assuming not [00:19:18] but you can let me know [00:19:29] I'm not sure what it means in context. [00:19:48] to make absolute [00:24:11] notpeter: you there? [00:24:38] New review: preilly; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/13287 [00:30:22] New patchset: Jeremyb; "notify Service[apache2] when its conf changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13288 [00:30:31] LeslieCarr: ^ [00:30:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13288 [00:31:47] Brooke, did you read on #wikimedia the validation doubts on the feeds wrt protocol-relative URLs? [00:32:02] No. [00:32:07] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/13288 [00:32:10] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13288 [00:32:13] it wasn't just protorel [00:32:13] LeslieCarr: can you merge and push this https://gerrit.wikimedia.org/r/#/c/13287/ ? [00:32:16] cool, that should work :) [00:32:22] wow, surprisingly fast ;) [00:32:26] LeslieCarr: I can't find Asher or Peter at this time [00:32:28] preilly: you guys are still doing that ? [00:32:30] hopefully preilly's change will test mine ;) [00:32:47] seems unrelated though ;( [00:32:50] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/13287 [00:32:58] Brooke, what about the broken mingle redirect? [00:33:15] is that included in that bug? I don't remember [00:33:20] Does anyone use mingle? [00:33:29] I see the scrollback you're talking about from #wikimedia. [00:33:29] apparently yes [00:33:31] File a bug? [00:33:36] describe the brokeness? where to where? [00:33:45] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/13287 [00:33:48] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13287 [00:33:54] Nemo_bis: Brooke: it wasn't just protorel it was also multiple entries in the feed using the same ID [00:34:09] So file two bugs, then. [00:34:46] (well protorel was about the ID) [00:34:51] * Nemo_bis is not going to file bugs about things he doesn't understand and also needs sleep [00:35:11] Nemo_bis: Don't know what mingle issue you're talking about. [00:35:25] yeah, I'm looking for the link [00:35:40] oh, I did file it https://bugzilla.wikimedia.org/show_bug.cgi?id=34160 [00:36:05] haha [00:36:14] unfortunately i think that's not puppetized [00:36:31] LeslieCarr: thanks [00:36:33] andrew_wmf: ping? [00:36:53] !b 34160 | andrew_wmf [00:36:53] andrew_wmf: https://bugzilla.wikimedia.org/34160 [00:40:19] preilly: done [00:41:14] LeslieCarr: thanks again you rock! [00:53:27] Brooke: protorel most certainly isn't working. unless there's some config change to make it work [01:10:37] there's 2 labs project creations waiting if someone wants to do them [01:10:42] (/j #wikimedia-labs) [01:54:35] so, i realized for 31369 (SSL redirect) I did a bunch of unnecessary research ;( [01:54:45] s/unnecessary/irrelevant/ [01:55:57] and squid configs are still not in public VCS somehow?!! [01:56:07] * jeremyb goes to look at the beta setup [02:33:10] any bored root want to test something for me? ;) [02:58:19] New patchset: Jeremyb; "redirects.conf: mk whitespace consistent" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/13290 [03:13:40] New patchset: Jalexander; "Adjust target for WikimediaShopLink to test" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13291 [03:14:51] damnit, Jamesofur you trying to merge conflict me? ;) [03:15:10] jeremyb: well, duh, who wasn't? [03:15:10] :p [03:15:17] gerrit's being slow [03:15:26] for me [03:15:36] me as well [03:15:41] Jamesofur: do some code review for me? ;) [03:15:54] when it's back [03:16:30] jeremyb: going to be leaving the office in a minute but if it's not reviewed when I get home I will [03:16:42] (review, I can't push out on that branch) [03:16:55] sure [03:16:58] of course you can't [03:17:25] there it comes [03:18:35] did anyone just fix gerrit ? [03:18:37] (got paged) [03:19:00] LeslieCarr: saw it broke, saw it fixed [03:19:04] no one said a thing here [03:19:05] cool [03:19:05] yay [03:19:08] hehe [03:19:30] LeslieCarr: james and I have some patches coming if you're in a reviewing mood ;) [03:19:39] (james alexander) [03:19:44] i'm running a d&d game right now [03:19:48] hah [03:19:49] gonna check out the box real quick then go [03:20:07] well how did it page you but not notify the channel? [03:20:09] was it nagios? [03:20:24] weird, don't see anything [03:20:25] watchmouse [03:20:36] though nagios should have caught it [03:20:43] ok, off to kill my party with more evil humans [03:20:44] :) [03:21:03] * Jamesofur shakes his head sadly [03:21:08] well.. not so sadly [03:21:12] obvious followup: can we get watchmouse reporting in here? ;) [03:21:15] quite a nice CPU spike: http://ganglia.wikimedia.org/latest/?c=Miscellaneous%20eqiad&h=manganese.wikimedia.org&m=cpu_report&r=hour&s=descending&hc=4&mc=2 [03:21:33] New review: Jalexander; "(no comment)" [operations/apache-config] (master) C: 1; - https://gerrit.wikimedia.org/r/13290 [03:21:50] also load [03:22:48] yeah… doesn't look like we've seen either cpu or load get that high over the past week or so [03:23:19] it was 80legs again [03:23:39] ugh [03:23:40] I deployed the robots.txt, I guess they didn't get the message [03:23:58] ok, time to leave the office [03:24:00] * Jamesofur will be back [03:24:10] and yeah… if they keep it like that they're going to have to get blocked.. [03:27:33] New patchset: Jeremyb; "comment out redirect cfp.wikimania.wm" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/13292 [03:33:18] New patchset: Jeremyb; "bug 31369 - make redirects protorel where possible" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/13293 [03:34:11] * jeremyb wonders what 80legs was doing? something that could be done another way? (like from a git clone) [03:34:19] local git clone* [03:40:49] jeremyb: 80legs is a spider-as-a-service, so people can write their own custom spider jobs and run it on 80legs. so the quality of the 80legs api in combination with the quality of the dev who wrote the job determines how good or bad the spider behaves [03:41:12] New review: Jeremyb; "haven't tested much (and not in the WMF environment)." [operations/apache-config] (master) C: 0; - https://gerrit.wikimedia.org/r/13293 [03:41:32] drdee: ewwww [03:41:46] drdee: does it follow robots.txt by default? [03:42:01] i believe it does, but not 100% sure [03:42:12] because 28 03:23:39 < TimStarling> I deployed the robots.txt, I guess they didn't get the message [03:42:34] hold on [03:42:44] I'm blocking it [03:43:05] it says on their website that it supports robots.txt, but you have to wait a while for updates to propagate [03:43:12] a day is too long, our service has to be up [03:43:17] so I'm blocking it by UA [03:43:35] yes a day is way too long [03:43:45] should like every 5 mins or at least 1 hr [03:44:04] (well that depends on our expires header i guess? ;P) [03:44:09] it does say it respects robots.txt and that it should read it within 3 hours http://wiki.80legs.com/w/page/1114616/FAQ [03:45:07] we serve last-modified but not expires [03:45:09] sounds right to me [03:46:40] New patchset: Tim Starling; "Block 80legs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13294 [03:47:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13294 [03:47:19] New review: Tim Starling; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/13294 [03:47:22] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13294 [03:53:52] New patchset: Jeremyb; "bug 31369 - make redirects protorel where possible" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/13293 [03:56:03] New review: Tim Starling; "Won't this break Squid caching? The HTTPS proxy is in front of Squid." [operations/apache-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/13293 [03:56:08] New review: Jeremyb; "PS2 is just adding the RTs from bugzilla to the commit msg" [operations/apache-config] (master) C: 0; - https://gerrit.wikimedia.org/r/13293 [03:56:59] TimStarling: so what then? set a Vary header? [03:57:49] TimStarling: how do we already fragment or not for HTTPS or not? [03:58:10] i guess the main goal of protorel is to not fragment [03:58:14] I guess we could set a vary header [03:58:46] yes, the way we do it is to use protocol relative URLs [03:58:55] that's the only reason for them as far as I'm concerned [03:59:42] TimStarling: can you tell me the prod apache version? [04:00:48] 2.2.14-5ubuntu8.9 [04:01:00] danke [04:01:07] people have been adding protocol relative URLs to external links [04:01:13] it just seems ridiculous to me [04:01:21] hah [04:01:24] why? [04:01:26] if HTTPS is a good idea then you should use it, if not, then don't [04:01:34] well sure [04:01:49] but if it's not crucial for a given external site but they do offer it? [04:01:58] surfing WP with an http:// URL isn't an expression of a user preference [04:02:13] make a decision [04:03:42] New patchset: Jeremyb; "bug 31369 - make redirects protorel where possible" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/13293 [04:03:59] if the protocol in use is not a user preference, if it's just an accident because the search engines happen to index http:, then it shouldn't influence the decision about whether to use https for a given external link [04:04:15] protocol URLs are a nasty obscure hack that breaks clients [04:04:56] which clients? [04:06:41] IE7 and 8, according to Krinkle a couple of hours ago [04:06:46] huh [04:07:17] there are other clients that are broken in more obvious ways, we see them request incorrect URLs with double slashes in the path [04:07:41] don't know what their names are [04:07:53] haha [04:08:03] and also domain in the path? [04:09:24] * jeremyb kinda wants a list of staff (or even just tech) who's coming to wikimania [04:09:35] of course i could just look in the registration system ;) [04:10:03] yes, we see log entries for things like http://en.wikipedia.org//en.wikipedia.org/wiki/Foo [04:10:13] right [04:11:18] TimStarling: Great, it turns out that if stylesheets with having a protocl-relative url are downloaded twice in IE7 and IE8. A fine browser bug :) [04:12:55] twice but at the right URL?! [04:12:59] Yes [04:13:13] someone should find out whether it uses the content from the first or the second fetch ;) [04:13:36] I think (not tested) either both apply (like stylesheets do), or which ever last arrives [04:13:55] huh [04:14:18] if it were the first one then it would just be extra network but not extra waiting for paint [04:14:35] but its not much of an issue since ResourceLoader doesn't use or encourage this kind of loading anyway [04:14:55] Except for skin css fallback, we load everything through module packages that are inserted as CSS text directly, not through a url. [04:17:03] jeremyb: TimStarling: Redirects at the apache level (e.g. wikipedia.com > org, .mw.org > ww.mw.org etc.) - are those cached in squid, and if so cached by request url with or without protocol? I recall something about a shared cache, but not sure if that just applies to mw-parser cache or also to things like squid > apache directly. [04:17:36] i don't think squid has a shared cache [04:17:47] anyway, this is cached by squid [04:18:12] oh, let me read the back scroll first [04:18:15] see the patch. right above the Vary header is the cache-control [04:18:21] I see a comment from tim mentioning this somewhere up there [04:18:31] \ Tim Starling; "Won't this break Squid caching? The HTTPS proxy is in front of Squid." [04:19:03] keep going ;) [04:19:15] sorry, I'm going for lunch [04:19:17] ah, we have HTTPS > squid > apache ? interesting, I didn't know that [04:19:23] TimStarling: bon appetit [04:19:40] I thought we had http apaches and https apaches and squid in front both regardless. [04:19:48] Krinkle: right. IPv6 and HTTPS are both nginx. which then hit squid and then apache [04:20:06] apaches don't even have certs [04:20:10] rigiht [04:21:06] jeremyb: so the https proxy is just for the cert then, right? I mean the rest is shared/compatible [04:21:23] ummm, idk what shared/compatible means [04:21:49] squid has a header that indicates if it's HTTPS or not. but besides that extra header there's no difference [04:21:58] (well maybe more than one extra [04:22:00] ) [04:22:02] all the headers and response body can be forwarded without modification of any kind [04:22:06] damn xkcd.com/859/ [04:22:48] in that mediawiki apaches don't react differently. Ah, wait they do. MediaWiki does redirects as well and uses the current protocol. [04:23:41] what's an example mediawiki generated redirect? [04:24:39] Special:MyPage or saving an edit (post-get-redirect) [04:24:53] lots of redirects going on [04:25:41] post-redirect-get* whatever, the last two belong together [04:26:12] $ curl -sv -4 https://en.wikipedia.org/wiki/m:foo 2>&1 >/dev/null | egrep -e '^< (Vary|Location):' [04:26:15] < Vary: Accept-Encoding,Cookie,X-Forwarded-Proto [04:26:17] < Location: https://meta.wikimedia.org/wiki/foo [04:26:36] looks good to me [04:27:21] (although that's not direct from a backend... but probably fine) [04:27:28] Krinkle [04:27:51] jeremyb: What do you mean by looks good? good in what way? [04:28:02] X-Forwarded-Proto is present in the Vary header [04:29:01] (Mind you, I'm an operations noob) for which node in the communication chain is that header intended (X-Forwarded-Protocol) [04:29:17] sent by MediaWiki? Or from elsewhere? For the squid? [04:29:35] by MediaWiki i hope [04:30:37] $ git grep X-Forwarded-Proto | wc -l [04:30:37] 8 [04:30:44] that's mediawiki core [04:31:19] for any caching proxy. squid and nginx and varnish everything else that might cache it [04:31:29] within WMF it's just squid i guess [04:31:39] but that's just because nothing else caches [04:33:32] I've always been confused by the Vary header. [04:34:15] So using Vary: X-Forwarded-Protocol in redirects.conf would not make the redirect un-cacheable. It tells the requestor (e.g. a proxy) that this response may only be re-used if the value of the header name(s) in Vary are also the same [04:34:22] http://tools.ietf.org/html/rfc2616#section-14.44 ;) [04:34:33] I was at http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.44 [04:35:18] I'm afraid my abstraction level of http is higher than the one assumed in the context of that spec [04:35:29] heh [04:36:09] we could discuss over beer in ~12 days ;) [04:36:18] and with whiteboard! [04:36:45] almost there [04:36:47] So that also means that without a Vary header, the proxy will re-use the response as cache no matter the headers (only based on url?) [04:37:01] idk [04:37:09] what about content-language? [04:37:33] and accept-* headers? [04:37:45] The fact that we're using *reverse* proxies also makes some assumptions made in documentation for proxies a bit complicated (i.e.g who the "client" is) [04:37:55] some may be there implicitly? also depends on the configuration [04:38:06] we can add headers to always vary on. idk if we have [04:38:15] jeremyb: afaik those are ignored by proxies unless the server (e.g. apache/mediawiki) response with a Vary header. [04:38:24] Some APIs work that way, which is nice. [04:38:41] i don't follow [04:39:20] * jeremyb needs to sleep fairly soon [04:39:22] Instead of doing something like /w/api?format=json&action=foo or /api/foo.json it could just be /api/foo and based on the accept header the web application can use a format [04:39:34] sure [04:39:55] without the Vary header in such response stuff would get messed up :P [04:40:07] yes [04:40:45] but when ignoring the Accept: headers (which afaik most request in mediawiki do, they don't care about the accept header, stuff is motley all is in the url), sending a Vary header would unneededly fragment the cache [04:40:55] mostly* [04:41:40] brb later, and about that beer! [05:03:20] !log fixed fatal.log on fenari, socat was writing to a deleted file [05:03:27] Logged the message, Master [05:03:59] who keeps screwing up fatal.log on fenari, by attempting to rotate it but failing? [05:05:33] cp fatal.log{,.1} && cp /dev/null fatal.log; ? [05:05:54] or just send it a HUP or boot [05:06:17] * jeremyb really should sleep [05:06:19] bye! [05:06:57] yes, those are some of the ways it has been screwed up [05:08:14] !log srv266 was flooding the fatal error log, complaining about a missing file. Killed apache and ran sync-common. [05:08:20] Logged the message, Master [06:21:29] New review: Jalexander; "yeah, old and obsolete" [operations/apache-config] (master) C: 1; - https://gerrit.wikimedia.org/r/13292 [08:35:12] New patchset: Dereckson; "(bug 37674) Adding standard logo for na.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13301 [08:40:44] New patchset: Dereckson; "(bug 37674) Adding standard logo for na.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13301 [09:38:11] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [09:38:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [11:10:07] New patchset: Hashar; "static-master for bits docroot" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13316 [12:38:51] New patchset: Hashar; "(bug 37245) makes labs use bits.beta.wmflabs.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13322 [12:40:06] New patchset: Hashar; "(bug 37245) docroot 'static-master' for beta bits" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13316 [12:40:24] New review: Hashar; "patchset 2 add bug number" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/13316 [12:42:34] New patchset: Hashar; "(bug 37245) docroot 'static-master' for beta bits" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13316 [12:43:26] New review: Hashar; "Patchset 3 update topic to reflect bug number and reworded commit message." [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/13316 [12:43:30] /clear [12:46:04] paravoid: can we possibly take care of https://gerrit.wikimedia.org/r/#/c/12178/ ? [12:46:15] to fix two classes conflicting on installing /etc/sudoers [12:46:33] that prevents me from running puppet on the apaches / jobrunner boxes on 'beta' [13:18:53] New patchset: Demon; "(bug 35802) Gerrit email title truncation over eager" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13325 [13:19:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13325 [13:19:47] hashar: looking [13:21:00] paravoid: I have installed it on psm-precise instance, seems to work [13:21:08] though I am not really sure what need to be checked [13:23:50] New review: Jens Ohlig; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/8344 [13:31:50] New review: Demon; "(no comment)" [operations/mediawiki-config] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12990 [13:31:52] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/12990 [13:33:56] New review: Demon; "(no comment)" [operations/mediawiki-config] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12583 [13:33:58] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/12583 [13:35:17] New review: Demon; "(no comment)" [operations/mediawiki-config] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/13322 [13:35:19] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/13322 [13:47:01] New review: Faidon; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/12178 [13:47:04] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/12178 [13:48:20] New patchset: Ottomata; "filters.oxygen.erb - adding filter for Wikipedia Zero Grameenphone Bangladesh provider" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13327 [13:48:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13327 [13:53:48] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [13:54:21] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [13:54:29] New review: Hashar; "Patchset 2 is a rebase I sent by mistake." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [13:57:22] the good thing with the new way of doing things is that I can be confident it works [13:57:27] the bad thing is that I merge into production :-) [14:03:15] oh you merged my /etc/sudoers change \O/ [14:04:15] ohh generic::geoip::files disappeared [14:07:22] New patchset: Hashar; "disable geoip on labs for now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13329 [14:07:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13329 [14:07:55] paravoid: https://gerrit.wikimedia.org/r/13329 to disable misc::geoip something, that got rewritten / replaced. I don't need geoip in labs for now. [14:08:38] New review: Faidon; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/13329 [14:08:41] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13329 [14:09:01] danke :) [14:13:17] New patchset: Ottomata; "filters.emery.erb - adding Arabic Wikipedia banner page filter" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13331 [14:13:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13331 [14:13:55] heh [14:14:33] New review: Ottomata; "Should this be pipe 10? Asking Diederik..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13331 [14:15:16] !log restarting gerrit [14:15:22] Logged the message, Master [14:15:46] !log restarting apache on manganese [14:15:51] Logged the message, Master [14:20:57] hi ops! [14:21:01] this should be an easy one: [14:21:05] if anyone is around to help me out: [14:21:05] https://gerrit.wikimedia.org/r/#/c/13327/ [14:24:58] !log manganese dist-upgrade [14:25:03] Logged the message, Master [14:25:10] hey hashar [14:25:18] hello :-] [14:25:23] to get geoip on labs [14:25:25] all you ahve to do [14:25:25] is [14:25:27] include geoip [14:25:36] oh but it relies on private repo [14:25:38] i seeee [14:25:39] ummmm [14:25:41] are you the one that rewrote the class ? [14:25:49] thanks for doing that :-] [14:25:59] yeah the private repo probably mean that it is not going to work on labs [14:26:10] we have a public private repo though [14:26:11] yeah hmmm [14:26:17] ahha, oh yeah? [14:26:24] yeah I rewrote it [14:26:40] a public private repo? [14:26:52] hmm, the goeip stuff is funny though [14:26:57] do we also have a private public repo :D [14:26:58] well hmm maybe it is private after all [14:27:24] ahey actually [14:27:47] ATTENTION!! rebooting maganese (aka gerrit) for dist-upgrade [14:27:51] so labs got it is own private [14:27:52] does labs talk to sockpuppet, or ummmm wherever the volatile repo is? [14:27:53] that's the problem [14:27:58] we don't need the private repo [14:28:04] because we don't want the GeoIP creds on labs [14:28:13] but by default it doesn't need it [14:28:19] if you just include geoip [14:28:24] will need to try it out so :-] [14:28:26] it will try to copy the files from volatile [14:28:33] puppet:///volatile/GeoIP [14:28:37] i have no idea where that is [14:28:41] though, or if labs has access to that [14:28:42] neither do i [14:28:48] aye hm [14:31:31] bugzilled it https://bugzilla.wikimedia.org/show_bug.cgi?id=38027 [14:33:30] New patchset: Ottomata; "misc/statistics.pp - adding -t flag to rsync log command to preserve mod times" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13334 [14:34:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13334 [14:37:58] !log allocating yttrium to payments rack per rt 1227 [14:38:03] Logged the message, RobH [14:38:29] !log manganese rebooted for kernel update [14:38:31] !log dist-upgrading formey (svn/gerrit), rebooting soon [14:38:34] Logged the message, Master [14:38:40] Logged the message, Master [14:40:13] !log svn server is rebooting.brb [14:40:19] Logged the message, Master [14:40:20] heh [14:41:15] and back [14:41:58] * jeremyb wonders if the elements have CNAMEs for their atomic #? [14:42:08] hrmmmm.... [14:42:28] and for that matter whose idea was elements [14:42:42] ottomata1: why not just rsync -a ? [14:42:55] i don't want to preserve ownership [14:43:00] huh [14:43:31] -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) [14:43:37] yeah [14:43:38] -o, --owner preserve owner (super-user only) [14:44:17] -g, --group preserve group [14:44:22] aye [14:47:27] jeremyb, do you have merge powers? [14:47:31] no [14:47:35] ayyyeee [14:47:59] ottomata: hashar: the labs puppetmaster is virt0. virt0 is in turn also puppetized pointing at sockpuppet. i think. you could include the geoip file on virt0 to pull from sockpuppet's volatile and then use that in labs? [14:48:01] I have 5 simple commits waiting for approval [14:48:28] yeah totally, as long as puppet://volatile is reachable and configured on labs [14:48:29] I don't have access to virt0 :D [14:48:29] it will work [14:48:37] but I will definitely try out the geoip class on some instance [14:48:40] ohohoh [14:48:48] hm, i see, more than just including it [14:48:50] oh hmmmm [14:48:56] * hashar is not an ops so got no root / fancy access [14:49:07]