[00:02:42] Krinkle can you pull that list actually? [00:02:56] git pull against gerrit is timing out for me [00:03:00] anyone else having issues? [00:03:39] ToAruShiroiNeko: https://github.com/wikimedia/operations-mediawiki-config/ [00:03:57] https://github.com/wikimedia/operations-mediawiki-config/blob/master/all.dblist [00:03:58] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 00:03:54 UTC 2013 [00:04:00] or which list did you mean? [00:04:28] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [00:10:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [00:15:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) [00:20:10] RECOVERY - check_mysql on payments1001 is OK: Uptime: 5545520 Threads: 4 Questions: 4453825 Slow queries: 21421 Opens: 627 Flush tables: 1 Open tables: 61 Queries per second avg: 0.803 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [00:24:14] Krinkle brilliant [00:24:15] thanks [00:34:40] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 00:34:35 UTC 2013 [00:35:20] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [01:04:50] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 01:04:48 UTC 2013 [01:05:20] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [01:30:03] Krinkle here: https://www.mediawiki.org/wiki/Requests_for_comment/URL_shortener/WikiMap [01:30:06] just an idea [01:30:40] it lacks some wikis such as non-wmf operated chapter wikis, bugzilla etc [01:31:51] perhaps community input isnt needed as its rather sitraight forward [01:33:25] is that unacceptable for you? :/ [01:34:35] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 01:34:33 UTC 2013 [01:35:13] ToAruShiroiNeko: Where is the wiki-id column (the actual wiki database name or domain name like from all.dblist) [01:35:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [01:36:03] ToAruShiroiNeko: www | commonswiki seems odd [01:36:58] don't split 2015 from wikimania etc. [01:38:01] looks like a plausible implementation [01:38:03] its kind of for human use [01:38:14] I put www as its commons.wikimedia [01:38:24] www isn't even valid, even for humans. [01:38:36] I am open to suggestions :) [01:38:56] 2015 wikimania wiki doesnt exist yet I think [01:38:57] I'd use base36 | family (e.g. label of first character of base36) | wikiid or dbname or domainname [01:39:44] those 3 columns. Don't tear apart the supposed prefix, they aren't prefixes and would make it hard to analyse or map back to actual domain names (which is the main purpose of the map for the shortener) [01:40:54] project name is commonswiki and wikimania2015, not www or 2015. Only the family is virtual, the project could be real. That saves having to back fill missing values that we don't have (and can't have). [01:41:25] the list can expand with time [01:41:59] M0C would be wikimania2015 wiki [01:42:18] it doesnt exist yet [01:42:39] I think there's a couple ways to go about it, this is one of them. Though this is maybe not the easiest one to maintain with semi-automated mapping to how families are configured on the cluster. Either way, it'd be a trivial part of the URL shortener implementation. [01:42:58] yeah [01:43:20] fast majority would be simple to maintain like the main big projects [01:43:24] Not something I hope will be much discussion about (eg.. bikeshedding). This is a good start. [01:44:54] the real challenges at this point are 1) the domain name, 2) the open bugs for the ShortUrl extension (if we end up using it); 3) whether or not to drop this format alltogether and use full url shortening instead [01:45:12] the latter would fit equally (if not, better) in the QR encryption format. [01:45:59] the advantage of having (limited) readbility of the short url is something that hasn't been considered before so it's going to be hard to justify I think. [01:47:09] we'd sell it on being deterministic with static data (as opposed to being impossible to calculate, as the full url shortener would be, that would require an http or db request to register it and get the new id) [01:47:24] e.g. given the wiki id and shorturl id, we can create the url cheaply [01:51:20] * Krinkle runs \|\|(.+)\|\|(wikibooks|wikimedia|wiktionary|wikinews|wikiquote|wikisource|wikiversity|wikivoyage|wikipedia)$ on it [01:51:46] Krinkle I added a database column [01:52:09] Krinkle I think it answers to a flaw they mentioned [01:52:19] "Obfuscation and mis-use" [01:53:28] if I see an short/M url I know I am going to a Wikimania wiki [01:53:39] and not to a penis picture [01:54:28] the project column is now obsolete, only contains confusing values [01:54:51] and the family column contains various db name that aren't families [01:55:06] I suppose [01:55:12] editing [01:55:17] please do :) [01:55:35] it was helpful to seperate them for sorting to generate the base 36 [01:55:45] but thats been done already :p [01:56:04] what does 'J' stand for? [01:56:10] arbcom wikis [01:56:15] Judicial? :p [01:56:27] could be stuffed with X wikis I suppose [01:56:49] I dont think we will see loads of tiny URLs to those restricted wikis [01:57:06] board could be bundled with capters [01:59:39] WMFfoundationwiki should be foundationwiki [01:59:45] letting you edit to avoid EC [02:00:07] there is no WMFfoundationwiki [02:00:18] yes [02:00:26] I can fix it too [02:01:08] * Krinkle is done editing [02:01:28] board wikis are not chapter wikis, per wmf-config saying so. [02:01:33] foundationwiki is not a chapter wiki [02:02:14] no but it manages chapters, kinda [02:02:15] no? [02:03:04] practical purpose is relevant, but not as much as the configuration categories are relevant. [02:03:05] https://github.com/wikimedia/operations-mediawiki-config/blob/master/wikimedia.dblist [02:03:08] those are chapter wikis only [02:03:21] very well it can be moved :) [02:03:26] should I do it? [02:03:41] go ahead :) [02:03:55] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 02:03:49 UTC 2013 [02:04:19] these are all the X wikis: https://github.com/wikimedia/operations-mediawiki-config/blob/master/special.dblist. Could make commons, wikimania* and wikidata and wikispecies separate, but I'm not sure that is clear. [02:04:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [02:04:56] perhaps chapters can be given a country ID [02:05:00] two letter ISO codes [02:05:40] would that make sense? [02:06:18] I'm going to stop working/discussing this further. I'm not disagreeing, just don't think it is useful to discuss further on this detail given the state of the RFC. This could be all wasted if we decide not to use this specific part of the RFC, and specifying it now doesn't make it more liely to be used either. [02:06:46] sure [02:08:35] I am probably more enthusiastic than most :p [02:09:49] !log testing MHA on pmtpa db37, db56, db58, db68. disabled their notifications' [02:10:05] Logged the message, Master [02:13:22] !log LocalisationUpdate completed (1.22wmf18) at Thu Sep 26 02:13:22 UTC 2013 [02:13:36] Logged the message, Master [02:14:23] Krinkle so for implementation domain name will be key [02:14:28] was there any progress on that? [02:15:38] not that I know of [02:15:42] springle: MHA? [02:16:24] jeremyb: http://code.google.com/p/mysql-master-ha/ [02:16:36] interesting... [02:17:02] Krinkle I hope there is some progress on that eventually [02:18:10] @replag 10.64.16.13 [02:18:11] Krinkle: [db1024: s7] 10.64.16.13: 0s [02:20:27] @replag db50 [02:20:28] Krinkle: [db50: s7] db50: s [02:20:32] @replag s7 [02:20:32] Krinkle: [s7] db1041: 0s, db1007: 0s, db1024: 0s, db1028: 0s [02:21:02] right, db50 isn't in rotation for the main api endpoint that the bot uses [02:21:40] ugh, long running queries, i hates (my own instances) [02:24:19] springle: i wonder if i could pick your brain a little [02:24:38] my mysql can get pretty rusty sometimes [02:25:15] !log LocalisationUpdate completed (1.22wmf17) at Thu Sep 26 02:25:14 UTC 2013 [02:25:26] Logged the message, Master [02:25:44] jeremyb: go ahead [02:25:52] i'll try to answer :) [02:25:52] seconds behind master is increasing, thread is waiting for next event, both threads (sql+io) are running. why might nothing be happening? [02:26:43] (i.e. waiting for master to send) [02:26:47] waiting for next event is io thread. what's sql thread doing? [02:27:43] grrr, so confusing. show processlist has 2 threads. i *think* innotop is combining them into one? [02:27:54] is this a box i can access? [02:28:38] nope, sorry :) [02:29:02] i think i need to implement some time limits. unattended hour long queries (from a webapp) are no good [02:29:12] in this case DELETE [02:29:29] thanks anyway [02:29:47] in any case, if seconds behind master is increasing, probably sql thread is lagging due to long-running transaction, or blocked by something else locally, or perhaps it's a big write txn blocked on io [02:30:13] big delete? [02:30:26] hour-long will suck :) [02:33:23] jeremyb: http://dev.mysql.com/doc/refman/5.5/en/slave-sql-thread-states.html might help [02:33:55] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 02:33:51 UTC 2013 [02:34:25] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [02:36:40] springle: thanks for jogging my memory :) [02:36:55] yw [02:45:41] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Sep 26 02:45:41 UTC 2013 [02:45:51] Logged the message, Master [03:06:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [03:29:12] @info db50 [03:29:12] Krinkle: [db50: ?] 10.0.6.60 [03:33:12] @replag s2 [03:33:13] Krinkle: [s2: zhwiki] db1034: 0s, db1036: 0s, db1002: 0s, db1009: 0s, db1018: 0s [03:34:51] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 03:34:48 UTC 2013 [03:35:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [04:06:51] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 04:06:41 UTC 2013 [04:07:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [04:35:41] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 04:35:36 UTC 2013 [04:36:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [04:59:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [05:00:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 16.133 second response time [05:04:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [05:05:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 29.774 second response time [05:07:11] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 05:07:02 UTC 2013 [05:07:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [05:10:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [05:15:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 9.770 second response time [05:35:18] (03CR) 10MZMcBride: "Greg: would it be okay to amend this changeset to deploy the MassMessage extension to the three phase 0 wikis (test2wiki, testwiki, and me" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85944 (owner: 10Legoktm) [05:36:41] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 05:36:37 UTC 2013 [05:37:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [05:37:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [05:38:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 21.161 second response time [05:38:16] (03CR) 10MZMcBride: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85944 (owner: 10Legoktm) [05:56:39] (03CR) 10Legoktm: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85944 (owner: 10Legoktm) [05:59:14] (03PS3) 10Legoktm: Enable MassMessage extension on test2.wikipedia.org [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85944 [06:06:21] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 06:06:19 UTC 2013 [06:06:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [06:14:23] (03CR) 10MZMcBride: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85944 (owner: 10Legoktm) [06:14:44] oops [06:14:47] i a word there. [06:15:04] I almost that joke in the comment. [06:15:11] Heh. [06:15:27] Typing is the worst. [06:16:02] (03PS4) 10Legoktm: Enable MassMessage extension on test2.wikipedia.org [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85944 [06:37:11] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 06:37:01 UTC 2013 [06:37:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [07:00:11] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours [07:02:32] (03PS1) 10Faidon Liambotis: varnish: remove X-CS/Analytics tag from vcl_fetch [operations/puppet] - 10https://gerrit.wikimedia.org/r/86079 [07:14:41] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 07:14:33 UTC 2013 [07:15:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [07:34:51] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 07:34:42 UTC 2013 [07:35:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [07:58:08] paravoid, yei!!! :) [07:58:48] paravoid, is it going to instantly start producing correct results? Or we need to wait until cache flushes? [08:02:33] sry, never mind, you already answered that in the patch notes. Great to hear that this issue is solved! [08:03:51] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 08:03:46 UTC 2013 [08:04:41] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:46] yurik: do you set X-Analytics anywhere in mediawiki? [08:16:56] paravoid, nope [08:17:09] its done only in the varnish [08:17:17] ok, good [08:18:37] paravoid, while you are looking at varnish, could you help with enabling ESI? I would really like to start live tests with it [08:18:51] i sent an email yesterday outlining test process [08:19:23] i will be traveling early next week, so won't be as easy to deal with pottential issues [08:20:12] has it been tested in labs? [08:21:32] yes, on one instance [08:21:58] at this point ESI won't be enabled unless there is a X-FORCE-ESI header [08:22:06] I haven't read your mail yet [08:22:37] so the only requirement from varnish is to enable tag search. The tag won't be there unless the header is set [08:22:56] any vanish magicians about? (it might not be a vanish issue) https://bugzilla.wikimedia.org/show_bug.cgi?id=54417#c4 [08:23:06] btw, how's the homepage changes coming along? [08:23:11] you've promised those a while ago :) [08:23:28] now that netmapper is finally going in prod, I'd like us to get rid of DfltLang and all those redirects altogether [08:23:45] we really need to make an effort simplifying the VCLs gradually [08:24:15] paravoid, most of the rewrite has already happened and is in production - all redirects are done by the special page now [08:24:20] as well as home page generation [08:25:05] once ESI is done, I can fairly easily enable dflt lang stuff (but you will have to remind me what exact behaviour is needed - my memory is shaky :( [08:25:15] PROBLEM - Puppet freshness on mw1072 is CRITICAL: No successful Puppet run in the last 10 hours [08:25:24] what does ESI has to do with anything? [08:25:45] because that's what Dan wants to be done first :) [08:26:00] yes, but you've promised that the homepage fix for many months first :) [08:26:16] (netmapper too) [08:26:23] heh, i knew there was something i forgot - pls remind me excatly what needs to happen [08:26:27] netmapper is not on my plate :( [08:26:37] so, with netmapper, tag_carrier will get simplified a lot, right? [08:26:47] it took me a few hours to get all the needed ips info for Brandon [08:26:50] all those set X-Cs will go [08:26:59] set req.http.X-CS etc. [08:27:18] correct - it will be X-CS = netmapper (ip) in a loop - to deal with proxies (opera / ssl) [08:27:28] popping off the XFF values [08:27:37] but you still have X-DfltLang [08:27:41] for certain carriers [08:28:00] and then issuing a redirect [08:28:00] if (req.http.X-CS) { [08:28:00] // Carrier detected [08:28:01] error 666 "http://" + req.http.X-DfltLang + ".m.wikipedia.org" + req.http.X-DfltPage; [08:28:08] } else if (req.http.host == "zero.wikipedia.org") { [08:28:08] // All ZERO requests should go to the Special:ZeroRatedMobileAccess, even for unknown carrier [08:28:11] error 666 "http://" + req.http.X-DfltLang + ".zero.wikipedia.org" + req.http.X-DfltPage; [08:28:14] etc. [08:28:36] we've talked a bunch of times (there's even a mail from me about it if you search your inbox) about a proper solution for this [08:28:55] which would be for mobile to have its own homepage that m.wikipedia.org would go to, similar to www.wikipedia.org [08:29:11] a mobile version of www if you'd like [08:29:15] that could be varied on X-CS [08:29:29] and redirect or give different content to different carriers [08:29:38] searching, i think i remember now - should be easy to fix (assuming we never want to allow en.m.wikipedia.org/wiki/Special:ZeroSpecial if the default language is fr [08:30:09] that page simply shows a list of available language [08:30:17] the point is, we shouldn't have a default language in varnish at all [08:30:26] or care about language subdomains in general [08:30:54] paravoid, the easiest solution is to send everyone to english special page, which will redirect to the proper language depending on the carrier [08:30:54] or redirect [08:30:57] no [08:31:14] these redirects are for when users go to "m.wikipedia.org" or "zero.wikipedia.org". [08:31:29] this needs to be a separate, cross-language page IMHO [08:31:32] like www.wikipedia.org is [08:31:42] ignore zero for a moment [08:32:03] right now if you go to m.wikipedia.org from your non-zero browser, you'll get redirected to en.m [08:32:08] that's wrong by itself [08:32:47] doesn't that page (desktop ver) support mobile? [08:32:50] we just need a language-agnostic homepage/landing page; that could also handle zero/welcome $carrier customer [08:32:54] i remember there was some CSS discussion [08:33:00] which one? [08:33:06] main www [08:33:12] no idea [08:33:13] non language specific [08:33:51] hm, seems to [08:33:55] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Thu Sep 26 08:33:50 UTC 2013 [08:33:55] paravoid, just spoofed user-agent -- no redirect to m [08:34:13] www.wikipedia seems to be mobile-enabled [08:34:20] yep [08:34:21] so shouldn't m.wikipedia.org just use that then? :) [08:34:28] instead of redirecting to en.m [08:34:29] try hitting Ctrl+Shift+M in firefox [08:34:35] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [08:34:38] and set smaller res [08:34:50] seems to be properly scaling [08:35:06] yes, but this is about users going to "m.wikipedia.org" or "zero.wikipedia.org" [08:35:23] I propose(d) that we should just present them the same page as www [08:35:30] max has been talking about killing m & zero outright :) [08:35:38] and you should make that page zero-aware [08:35:45] add the banners there or whatever [08:36:01] and kill special:zerospecial and the dfltlang redirects [08:36:30] i am not sure i can just copy that page -- carriers whitelist specific languages, and moreover, they decide which languages they want to show for their locale [08:36:46] if you look at the meta pages, they list "prefered" languages in their order [08:36:58] fix that homepage to have that [08:37:09] to have the preferred languages on top, for example [08:37:17] and Vary: X-CS [08:37:40] it's crazy that we do preferred languages in varnish [08:37:42] so the special:zero shows that list, followed by a dropdown with all langs. The www page AFAIK is static, defined by users [08:37:57] i totally agree that it shouldn't be in varnish [08:38:30] we need a landing page and I don't see why zero's landing page belongs under Special: tbh [08:38:55] and the www page being static, this can change you know :) [08:39:31] that's why i said - the easiest solution is to simply redirect to the correct language's special, and then decide how to change the whole system of main page based on carrier [08:39:41] there is no correct language, is there [08:39:53] there is -- each carrier defines their prefered language [08:40:14] language*s*, for starters [08:40:23] right [08:40:30] but the main one is first in the list [08:40:45] plus, the fact that the carrier provides zero access to some languages, doesn't mean that users may not be interested in other languages and pay for them [08:40:59] yes - that's why we have a dropdown [08:41:06] or that the users speak the language you chose in the first place [08:41:06] with all langs [08:41:14] and no, i don't think its a good solution :) [08:41:29] if I go travel in russia and access m.wp.org I'll get a page in russian, that's not what we do in desktops [08:41:30] (the dropdown) [08:41:48] I'll get a page in russian if I'm accessing it via beeline, but the english wikipedia if I access it from wifi [08:41:51] crazyness :) [08:42:18] paravoid, you are asking for 2 things: 1) remove defalt langs from varnish - i can easily do that, 2) change the behavior of zero sites - you will have to get Dan to approve that [08:42:27] indeed [08:42:34] so let me do #1 [08:42:47] the one I care most about is cleaning up varnish [08:42:48] and we will deal with #2 after discussion with the biz ppl :) [08:43:06] but I do think it's crazy how we do landing pages and I care in general sense :) [08:43:08] it is their "productio" - something that they "sell" to carriers [08:43:10] in the* [08:43:21] and I agree with you :) [08:43:47] so anyway, will fix redirect, thx for reminding! [08:43:57] pls enable request's esi in the mean time [08:43:59] thank you :) [08:44:01] would be great if we can demo it :) [08:44:13] and start measuring impact [08:44:21] (will probaly need your help tracking graphs) [08:44:29] pls take a look at my email - its very short [08:44:34] let's wait for mark though :) [08:44:39] heh :( [08:44:43] i meant :) [08:47:01] (03PS1) 10Yuvipanda: Add labsvagrant helper script to help manage labsvagrant roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/86083 [08:47:27] mmmm! [08:47:37] uhoh [08:47:44] ori-l: you're still uP! [08:48:29] ori-l: I'm now putting this in ops/puppet.git, since it has 'labs' in its name - but it is sortof 'inserted' into the vagrant repo anyway. I'll be happy to move it to the vagrant repo if you're okay with it [08:48:33] run now ori-l, while you still have a chance [08:49:17] ori-l, :( https://dpaste.de/CdMwp/ [08:49:27] i mean :) [08:49:33] and i saw the same issue on mac :) [08:49:56] wait, what? [08:49:59] * yurik throws inflatable duck to ori to prevent him from drowning [08:50:26] that's your roundabout way of saying you still need a varnish role, yes? [08:50:40] i didn't actually accidentally commit some half-written varnish module, did i? [08:50:44] yurik: pull to latest? [08:50:58] yep [08:51:04] I had a similar problem [08:51:05] iirc [08:51:12] just did a git pull [08:51:37] there's no varnish role [08:51:38] i think it still runs though [08:51:58] yurik, it's 2 AM, don't fuck with my head! :P [08:52:11] * YuviPanda orders ori-l a tuna sandwich [08:52:12] hehe, its 5 am here :-P [08:52:24] I actually the subway peppers. [08:52:36] and paravoid just did that with mine, so i pass it on :-P [08:52:38] ironic, considering subway is nicer in the US than here. [08:52:53] ori-l: hmm, mind if I put labsvagrant.rb in mediawiki/vagrant.git [08:52:54] ? [08:52:55] subway the sandwitch place? bleh [08:53:30] the trailing whitespace wasn't confidence-inspiring [08:53:40] there's trailing whitespace? [08:53:50] line 63 [08:53:56] ori-l: gah, I no longer check for it since i've git set to strip it out [08:53:59] apparently it failed this time [08:54:11] yes, you can commit it [08:54:21] as long as it doesn't impact the desktop stuff [08:54:30] it doesn't [08:54:33] and sweet :D [08:54:41] I'll test it for a bit more and then commit [08:54:58] ori-l: it's just a standalone script, doesn't really do anything else. Most of it is sadly copy-pasted off mediawiki-vagrant.rb [08:55:22] i'm an awful rubyist [08:55:30] you should not imitate me [08:55:47] but it's your project, so gl;hf [08:56:03] this is the first time I'm writing ruby [08:56:04] so [08:56:11] (not counting attempts to do so about 4-5 years ago) [08:57:24] ah, hmm. [08:57:25] key_verify failed for server_host_key [08:57:31] (03PS2) 10Yuvipanda: Add labsvagrant helper script to help manage labsvagrant roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/86083 [08:57:37] this is gerrit, yes? [08:57:39] yeah [08:57:44] ok, paravoid [08:57:47] that's #3 [08:57:47] there's a bug filed, IIRC. [08:57:57] do you remember where? [08:58:02] i don't know wtf is up with that server [08:58:13] ok, i might have just sleepily read IRC convos [08:58:17] checking anyway [08:58:26] what is? [08:58:40] ori-l: https://bugzilla.wikimedia.org/show_bug.cgi?id=53895 [08:58:58] been around for a while looks like [08:59:07] ori-l: ? [08:59:25] people keep getting key_verify failed for server_host_key for gerrit [08:59:34] that's the worst bug title ever, it has nothing to do with translatewiki [08:59:44] heh [08:59:45] it usually clears itself up on a second try [09:00:16] e.g.: [09:00:19] 10:10 Krinkle: Hm.. Pushing to gerrit gives me: [09:00:19] 10:10 Krinkle: hash mismatch [09:00:19] 10:10 Krinkle: key_verify failed for server_host_key [09:00:21] 10:10 Krinkle: fatal: Could not read from remote repository. [09:00:23] 10:10 Krinkle: Trying again worked though [09:00:40] that's about 16 hours ago on #wikimedia-dev [09:01:11] and on 9/21: [09:01:14] Krenair: [16:57:17] has gerrit's key changed? I'm getting key_verify failed for server_host_key [09:01:14] Krenair: [16:58:08] .. and it's now working. weird. [09:01:37] (03PS3) 10Yuvipanda: Add labsvagrant helper script to help manage labsvagrant roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/86083 [09:05:37] paravoid: any idea? [09:06:37] Coren: commit 367294d573db933b934c8b9433dd2acb3b0c3c8d [09:06:56] Coren: on ops/dns, "Add dickson to DNS (new freenode server)", very misleading commit message [09:07:19] Coren: you removed manganese in the same commit, didn't mention anything in the message, I got very confused [09:07:52] ori-l: it's hard to debug retroactively [09:08:11] ori-l: we should go log digging the moment it happens [09:08:46] let me see if i can trigger it [09:08:57] i'll pull in a loop with a 5 second sleep [09:11:35] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [09:12:46] paravoid: unrelated, but follow on from the 'Mediawiki Labs' conversation between you, me and Ryan_Lane from the office - https://wikitech.wikimedia.org/wiki/Projects/mediawiki_Labs_project [09:18:13] (03PS4) 10Yuvipanda: Add labsvagrant helper script to help manage labsvagrant roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/86083 [09:24:01] paravoid: haven't been able to trigger it so far :/ it must be an IP conflict somewhere, no? [09:33:55]