[00:45:38] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:05:09] New review: MZMcBride; "Hi Ryan." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/57649 [01:35:03] New patchset: MZMcBride; "noc: Refactor highlight.php to be simpler and ~~less~~ more secure" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59034 [01:35:30] New review: MZMcBride; "This was a wonderful commit message to read. Thanks for working on this, Krinkle. :-)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59034 [01:37:30] TimStarling: If you have a minute, I'd like you to do a sanity check on g 59034 (in case you haven't already). [01:38:28] * jeremyb_ has just starred 59034 [01:38:54] +1 for the same reasons as Susan but i haven't looked at the actual changes [01:39:48] wow, that's a real essay on g 57649 [01:42:03] https://gerrit.wikimedia.org/r/#/c/57649/ [01:42:16] Gerrit isn't conducive to essays. [01:42:20] But I don't let that stop me. [01:42:53] Susan: contact and contact-url probably come from the ContactPage extension [01:43:39] Contact-url isn't defined. [01:43:42] But Contact, probably. [01:43:51] Though that'd be pretty nasty for an extension to not prefix its messages. [01:44:18] morebots still missing. [01:44:19] Grrrr. [01:44:41] Seems someone updated the log, though. [01:44:46] That makes me happy. [01:49:15] "attempt to write a readonly database" [01:49:20] So that's why jenkins is failing [01:49:37] https://integration.wikimedia.org/ci/job/mediawiki-core-qunit/2091/console [01:49:42] PleaseStand is wonderful. [01:50:44] I let him know. [01:51:29] Susan: Nice work on publishing your bots on github [01:51:33] I didn't know toolserver/reba was yours. [01:56:31] Thanks. :-) [01:56:36] Yeah, it's a really simple/stupid bot. [01:56:42] It just polls Gmail every minute or something. [01:57:02] There's a much smarter way to do e-mail --> IRC involving a mail filter or something, I think, but this was simpler for me at the time. [02:03:47] Krinkle: doing it now [02:16:20] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:55] Krinkle: maybe it should just redirect to gitweb, since we have the gitweb URL there already [02:16:59] !log LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 02:16:58 UTC 2013 [02:17:17] Susan: i'm surprised you havn't just rewritten a moresbot replacement yet [02:17:20] that would make it somewhat easier to review [02:18:28] p858snake: Yeah... I suppose I could. [02:18:44] It wouldn't tweet, but it could post to the wiki page. [02:19:17] TimStarling: That doesnt have syntaxhighlighting though, and the inteface doesn't have a blame url in the same interface. [02:19:18] Susan: I don't think many people would pay that much attention to the twitter feed most likely [02:19:23] Perhaps redirect to github? [02:19:40] why does it need those things? [02:20:24] well, it doesn't need anything. The information is already public in the repos. the whole portal has become redundant. [02:20:50] Susan: Actually instead of posting directly to the page, You could have it go to a subpage (eg: SAL/Year-Month) which wouuld resolve the issue of the page having thousands of revs and easier to archive the logs away (but just removing their trunsucation from the SAL page) [02:20:50] but the way it is, it is easier to use. [02:20:58] having those things adds to that. [02:21:07] I personally use noc regularly. [02:21:14] Particular the raw text files. [02:21:25] Syntax highlighting with a large file is really slow to load and search. [02:21:32] Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/59034 [02:21:51] I'm not sure if gitweb supports in-browser raw file viewing. [02:21:53] Susan: What if noc.wikimedia.org/conf/ would redirect to https://github.com/wikimedia/operations-mediawiki-config [02:21:53] GitHub might. [02:22:07] I merged it, but I think it has outlived its usefulness [02:22:11] github does, their raw. subdomain enforces text/plain [02:22:23] And then click wmf-config? Maaaaaaaybe. [02:22:29] TimStarling: I noticed that the cdb file with .txt still triggers a download [02:22:35] TimStarling: Any idea what's up with that? [02:22:52] those arguments about extra features in highlight.php could apply to any of our code [02:23:06] Sure [02:23:07] the reason highlight.php is limited to configuration is because the rest of the code was in version control [02:23:15] Click wmf-config, click InitialiseSettings.php, then maybe go to the raw version. [02:23:20] I agree, we should phase it out. [02:23:59] I'll start on a patch that reroutes it to version control viewers. And then we can make the individual redirects for the symlinks to the raw view of whatever viewer we use (not the ones via highlight.php) [02:24:06] what URL is triggering a download? [02:24:15] Some people also rely on the .txt files as an API of sorts. [02:24:17] the "raw text" for the cdb files in noc/conf [02:24:37] I used to have a tool to search CommonSettings.php/InitialiseSettings.php. [02:24:48] I still might. [02:24:49] https://toolserver.org/~mzmcbride/splarka/ [02:24:53] you mean http://noc.wikimedia.org/conf/interwiki.cdb.txt ? [02:24:57] Yes [02:25:00] Heh, this was before the .txt versions existed. [02:25:05] binary data [02:25:08] ha [02:25:09] I used lynx to strip the syntax highlighting. ;-) [02:25:36] it doesn't trigger a download for me [02:25:45] it just shows the binary garbage in the browser [02:25:51] Triggers a download for me. [02:25:57] In Chrome/OS X. [02:26:18] that is not surprising, I guess [02:26:39] sub.Popen('lynx -dump -width=99999 -display_charset=utf-8 %s%s.php' % (url, file) [02:26:43] :D [02:26:45] so highlight.php would redirect to the 'blob' action for gitweb or github ( https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/StartProfiler.php ) and the raw one'd direct to the plain/raw action https://raw.github.com/wikimedia/operations-mediawiki-config/master/wmf-config/StartProfiler.php [02:26:46] you know IE's FindMimeFromData(), which I disassembled and wrote a blog post about [02:26:53] and the portal / index can go to the tree view [02:27:08] that would ignore text/plain if there was binary data in the file [02:27:36] and we knew at the time that safari emulated IE's behaviour in some ways [02:27:55] so it's not surprising that other webkit clones would do it also [02:28:11] http://tstarling.com/blog/2008/12/secure-web-uploads/ [02:28:14] Susan: Also a download for this one? https://raw.github.com/wikimedia/operations-mediawiki-config/master/wmf-config/interwiki.cdb [02:28:23] Yes. [02:28:29] Not here [02:28:30] Triggers a download of interwiki.cdb. [02:28:36] As opposed to interwiki.cdb.txt. [02:28:45] Looks like github is using something that chrome makes it happy [02:28:51] Susan: Sure [02:28:56] it uses the file name [02:29:08] https://noc.wikimedia.org/conf/interwiki.cdb.txt and https://noc.wikimedia.org/conf/interwiki.cdb [02:29:11] respectively [02:29:35] Right. [02:29:58] TimStarling: The one thing it does add it centralise apache-config.git puppet:////lucene and mediawiki-config together [02:30:05] though that's a biased subset of course [02:30:39] I added repo links for those as well a few months back [02:30:48] I'm not sure the maintenance burden of noc/conf outweighs the cost of deprecating it. [02:31:34] brb later [02:31:34] index.php can do that, with deep links to gitweb or whatever [02:35:00] !log LocalisationUpdate completed (1.22wmf2) at Tue Apr 16 02:34:59 UTC 2013 [02:35:55] wow is morebots still dead? :( [02:48:53] hmmm [02:57:09] closedmouth: Yes, it seems my bug was ignored. And a number of people apparently performed actions today without logging them. [02:57:23] :(( [02:57:26] Which you'd think would upset more people. Though I don't think many people noticed. [03:04:23] New patchset: Tim Starling; "Enable terbium cron jobs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59370 [03:06:18] I can't fix it anymore [03:07:14] Oh, because it's on the old wikitech? [03:07:15] I'm nearly done with a clone. [03:07:32] Kind of insane that it wasn't moved to the new host when the wiki was moved... [03:07:38] Though I imagine there was a reason. [03:07:39] no, it's on a new linode [03:07:54] hang on, I found the root password for it [03:07:58] morebots is on wikitech-static. [03:08:17] (I'm not sure what host that is at this point.) [03:08:32] wikitech-static is the new linode [03:08:43] Okay. [03:09:41] just setting it up properly... [03:16:15] except: [03:16:15] logging.warning(sys.exc_info) [03:16:20] WARNING:root: [03:17:15] not very helpful [03:17:37] python? [03:17:40] where is that code? [03:17:45] ori-l: morebots. [03:18:05] haven't found its version control yet [03:18:07] https://wikitech.wikimedia.org/wiki/Morebots is where it should be... [03:18:08] logging.exception('An error occurred') will automatically attach exception info / traceback to the log output [03:18:30] I feel like it was in SVN... may be moved to Git. [03:18:41] provided it is invoked from within an 'except:' block [03:19:21] https://github.com/search?q=morebots&ref=cmdform&type=Code [03:19:28] of course just adding parens to that line, so that it reads 'logging.warning(sys.exc_info())' will do the trick [03:19:31] I see a puppet file... [03:19:33] but it's not very pythonic [03:19:51] trunk/tools/adminlogbot [03:20:08] https://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/adminlogbot/ [03:20:24] ok, fixing [03:20:26] no adminlogbot project in git [03:20:50] ori-l: I doubt the version in SVN is up-to-date. [03:20:57] Tim can probably pastebin the current version. [03:21:12] this looks like the latest, no: https://gerrit.wikimedia.org/r/gitweb?p=operations/debs/adminbot.git;a=tree [03:21:25] Oh, nice. [03:22:57] https://wikitech.wikimedia.org/wiki/Morebots <-- I added that link there. Thanks for dinging that. [03:23:00] Finding. [03:23:07] It's time for a break from typing. :-) [03:23:16] * legoktm finished his clone [03:23:22] !log test [03:23:32] oh [03:23:38] is anonymous editing disabled on wikitech? [03:23:42] Yes. [03:23:45] blah [03:23:53] as on all labsconsole wikis [03:23:54] I think account creation is also restricted. [03:23:58] I have an account [03:24:28] no_morebots [03:24:38] account creation isn't [03:24:44] but it requires many fields to be filled in [03:24:50] and $wgConfirmEmailToEdit is true [03:25:10] It's rejecting my password... [03:25:19] Via index.php or api.php? [03:25:42] index.php works in my browser, but wikitools (api.php) isn't. [03:25:53] What'd you set as the API URL? [03:25:59] I don't think a bot on nightshade is really a great replacement [03:26:03] >>> site = wikitools.Wiki('https://wikitech.wikimedia.org/w/api.php') [03:26:04] that's not going to be easier to restart, is it? [03:26:25] TimStarling: It's a single python script, I can just as easily move it to labs [03:26:31] legoktm: You know, wikitools logging in broke on the TS wiki ages ago. I never figured out why. [03:26:36] I just edit anonymously there. [03:27:12] Well thats not an option here :/ [03:27:16] TimStarling: With my IRC bots, I usually have code in them that pings the server every minute and then the bot will kill itself (and restart through some other process) if it doesn't get a pong. [03:27:34] It can help with netsplits and other funkiness. [03:27:42] And my bots are just a copy of Susan's :P [03:27:52] Frightening. [03:28:17] legoktm: Are you getting an error? Like WrongPass? [03:28:30] yeah [03:28:32] thats the error [03:28:34] You can print the raw API output as well. [03:28:43] in wikitools? [03:28:43] Yeah... it may be the same issue as the Toolserver wiki. [03:28:51] With wikitools. [03:28:54] You may have to hack it up a bit [03:28:55] . [03:28:59] But it's pretty simple. [03:30:37] {u'login': {u'result': u'WrongPass'}} [03:30:41] legoktm: Yeah, I'm getting WrongPass/False. [03:30:58] When I try to execute login(). [03:31:31] If you pass a made-up username, you get WrongPluginPass. [03:31:34] Pywikipediabot is complaining that it was getting a non-json response, but that has so many random quirks. [03:31:53] I feel like the login code is probably a little broken. [03:32:18] I bet Pywikipedia still has the screen-scraping ability somewhere. [03:32:32] Heh. [03:32:37] * legoktm looks [03:32:44] ori-l: are you going to submit any patches against it? [03:33:54] anyways, here's my morebots code http://dpaste.de/6QQj4/raw/ [03:34:54] legoktm: What would it take to get you to figure out why logging in doesn't work? :-) [03:35:11] I'm already working on that :P [03:35:13] I can mail you a prize. [03:36:47] erhmmm [03:36:58] My custom script can log in. [03:37:18] this is the library it's using: https://bitbucket.org/jaraco/irc/src/5ddfc868d51e096435088b59e22df0c94abea839/?at=version_0_4_8 [03:39:19] Lets see if it can edit... [03:40:04] you see that library has irclib and ircbot [03:40:17] ircbot has automatic reconnection [03:40:25] irclib does not, and morebots uses irclib [03:40:39] bleh. it didnt actually work. [03:40:42] so it just connects once and then goes into an infinite loop of doing nothing [03:42:16] Susan: I can't figure out what could be wrong. [03:42:26] so if someone could change it to use ircbot instead, then it would just work [03:42:29] I think its a bug in one of the login extensions wikitech is using. [03:43:16] I would be happy to review such a patch [03:43:38] legoktm: You should fix the real® morebots first. :-) [03:43:42] but obviously I'm not going to let a second bot be in here, writing duplicate messages to the SAL [03:43:57] And perhaps file a bug about wikitools + Labs. [03:44:07] TimStarling: That makes sense, but right now neither is here :P [03:44:32] easily fixed [03:44:45] but that is just temporary, until it next gets disconnected [03:45:18] :) [03:45:28] its https://gerrit.wikimedia.org/r/gitweb?p=operations/debs/adminbot.git;a=blob;f=adminlogbot.py;h=3bea8a53f2157f8441a0ef5f4e905ebbfede93d8;hb=HEAD right? [03:45:59] yes [03:49:00] eww tabs :( [03:50:03] Let me PEP8-ify it first, then I'll try and convert it to ircbot [03:55:17] I'm filling a bug for this [03:55:21] New patchset: Legoktm; "Convert all tabs to 4 spaces, and other minor PEP8 fixes." [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/59371 [03:55:26] mostly for somewhere to put my own notes [03:55:31] I'll assign it to you [03:56:20] sure [03:56:56] New patchset: Ori.livneh; "Make logging to file correct and configurable" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/59372 [03:57:32] TimStarling: oh, sorry, I missed the rest of the conversation [03:58:38] ori-l: should I just re-do my tabs-->spaces commit on top of yours? [03:59:23] legoktm: no, that seems unfair to you. if you're going to do the irc library change, I can just abandon my change and you can just pick out the things you want from it (if anything) [03:59:48] or I can re-base my change on top of your change, if you prefer [04:00:38] Well I'm about to fix a few other files in that director so the PEP8 test will pass [04:00:53] either way works for me, since its one click in my IDE to fix the tabs [04:00:58] filing, directory * [04:01:17] thanks Susan [04:01:17] :P [04:01:42] No problem. [04:02:04] legoktm: I think this should be your project :P [04:02:14] it sounds up your alley [04:02:19] so I'm withdrawing my change [04:02:30] You'd be a good reviewer, though. :-) [04:02:46] ok :) [04:02:51] Change abandoned: Ori.livneh; "(no reason)" [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/59372 [04:23:10] Is officewiki https only? And if so, where is that set? [04:23:52] Yes, and it's here: https://noc.wikimedia.org/conf/remnant.conf [04:28:47] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [04:39:22] * legoktm pokes ori-l [04:39:33] How do you split this comment into 2 lines? [04:39:34] # user_query = "[[IRC Nick::+]] or [[IRC Cloak::+]]|?IRC Nick|?IRC Cloak|limit=500|format=json" [04:39:42] That's the only non-pep8 thing left in the repo. [04:41:06] Well, it's a comment, so it's really up to you, but if you want the comment to remain valid python code for some reason, I'd go with: [04:41:12] # user_query = ("[[IRC Nick::+]] or [[IRC Cloak::+]]|?IRC Nick|?IRC Cloak" [04:41:12] # "|limit=500|format=json") [04:42:04] Does the pep8 unit test thing check against comments? Mine does, but I haven't touched any of the settings [04:43:22] yes, it does (as it should, since pep8 includes guidelines about comment style) [04:50:43] New patchset: Legoktm; "Convert all tabs to 4 spaces, and other minor PEP8 fixes. All PEP8 tests should pass now." [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/59371 [05:01:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:02:16] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59370 [05:02:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.164 second response time [05:04:07] !log moving most cron jobs from hume to terbium [05:04:15] Logged the message, Master [05:28:21] TimStarling: it was already broken before today, but are the moves to terbium what broke https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20pmtpa&h=hume.wikimedia.org&v=823574&m=Global_JobQueue_length&r=hour&z=default&jr=&js=&st=1365625056&z=large ? [05:29:12] probably not [05:41:44] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [05:41:44] New patchset: Tim Starling; "Move broken cron jobs back to hume and note what is wrong" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59374 [05:43:09] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59374 [05:51:49] Nemo_bis: I don't think wmf1 has c3b315b8405f0626bcf6cf0ae612d01135a21e50 [05:51:57] so I'd expect a flat graph [05:52:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:52:28] I guess that will work wen [05:52:40] Wednesday that is [05:53:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [06:26:36] PROBLEM - Host search23 is DOWN: PING CRITICAL - Packet loss = 100% [06:26:52] hmmmmmm [06:28:30] 4.5 days since that box was reinstalled [06:29:16] RECOVERY - Host search23 is UP: PING OK - Packet loss = 0%, RTA = 26.50 ms [06:31:26] PROBLEM - search indices - check lucene status page on search23 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:32:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:33:16] RECOVERY - search indices - check lucene status page on search23 is OK: HTTP OK: HTTP/1.1 200 OK - 269 bytes in 0.055 second response time [06:33:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [06:36:47] Aaron|home: I didn't find c3b315b8405f0626bcf6cf0ae612d01135a21e50 but thank you, I'll wait hopeful :) [06:43:54] PROBLEM - NTP on search23 is CRITICAL: NTP CRITICAL: Offset unknown [06:47:54] RECOVERY - NTP on search23 is OK: NTP OK: Offset 1.835823059e-05 secs [07:19:42] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [07:31:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:32:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [07:57:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [08:02:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:03:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [08:03:49] mark: hey :] Will you will be available this afternoon to review some varnish changes I have made for beta? [08:06:22] i can review them now, this afternoon will be spotty [08:06:32] that works for me :-] [08:06:35] ok [08:06:49] * hashar moves the pile of accounting related papers to another desk. [08:07:03] :) [08:07:13] any particular preference/order? :) [08:07:28] I have rebased them yesterday [08:07:41] hopefully under a common topic [08:07:54] https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/puppet+branch:production+topic:54864,n,z [08:08:26] the first one would be https://gerrit.wikimedia.org/r/#/c/54864/ which make it so the mobile class uses role::cache::configuration instead of lvs::configuration::lvs_service_ips [08:08:37] since we do not have LVS in beta [08:08:52] that made the varnish mobile cache to points to some pre reserved IP address which simply does not serve any traffic :- ] [08:09:23] the trouble I have is finding out whether it will cause any madness on production :(( I should probably create a fake production varnish instance on my laptop to test that out [08:09:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [08:11:47] New patchset: Mark Bergsma; "mobile always uses role::cache::configuration" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54864 [08:13:03] New review: Mark Bergsma; "Note that the hierarchy is different for production and labs, and also for the new backends you adde..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54864 [08:14:13] New review: Mark Bergsma; "(2 comments)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54864 [08:14:19] almost merged it hashar [08:14:25] but have a few comments :) [08:14:31] goood :-] [08:15:07] * mark watches his car being serviced [08:15:07] oh site [08:15:17] are you working in a garage right now? [08:15:18] remote/mobile working is awesome [08:15:21] yes [08:15:28] I love it every single day [08:15:45] whenever it is sunny outside, I got to the park, sit next to a tree and enjoy a fresh coke while coding [08:15:51] yep [08:15:57] some people come to me asking what I am doing: "I am working for Wikipedia" :-D [08:16:05] hehehe [08:16:13] look on their face: priceless [08:16:13] and then they think you're writing it [08:16:18] yeah [08:16:22] that is a common misconception [08:16:37] that is a common trick at my coworking place [08:16:46] I am presented as the guy who wrote the french encyclopedia [08:16:49] ;] [08:16:52] i'm sure you are ;) [08:17:52] go forth and live up to the expectation? [08:21:43] mark: mind if I had the $::site level to backends in a different patch. That would require me to change the bits class too [08:21:51] or I can do it first and rebase my changes on that [08:23:27] yeah please rebase [08:23:38] for upload we also need to find a different way [08:23:44] because it's still special casing labs [08:24:26] okk [08:25:55] probably also working through role::cache::configuration more [08:26:02] i'm looking again at the mount stuff [08:26:14] so does every instance have a /dev/vdb available that can be used this way? [08:26:30] yes [08:26:54] the distrib is mounted on /dev/vda and /dev/vdb has most of the available disk space which is mounted on /mnt by default [08:27:05] I guess /dev/vda is usually 10GB [08:27:17] if you create a 320G instance, /dev/vdb would have 310G [08:27:36] ok [08:31:28] New patchset: Hashar; "beta: adds ::site to bits_appservers backends" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59380 [08:31:49] New review: Hashar; "(2 comments)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54864 [08:33:44] New patchset: Mark Bergsma; "(bug 44041) adapt role::cache::mobile for beta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44709 [08:36:50] New patchset: Mark Bergsma; "(bug 44041) adapt role::cache::mobile for beta" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44709 [08:37:41] damn I am lost [08:38:02] you're in france [08:38:20] you can actually get people to help you there ;p [08:40:44] Change abandoned: Mark Bergsma; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59136 [08:41:21] New patchset: Hashar; "mobile always uses role::cache::configuration" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54864 [08:41:53] heh [08:42:09] so https://gerrit.wikimedia.org/r/59380 adds ::site to the labs backends [08:42:25] I guess the bits server is broken right now [08:42:28] (on labs) [08:42:51] New review: Mark Bergsma; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54864 [08:42:53] alllmost [08:43:13] hmm [08:43:35] ah yeah if you want :-] [08:44:07] sorry, should have spotted that earlier ;) [08:44:13] otherwise it can be a bit confusing [08:44:41] New review: Hashar; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54864 [08:44:48] New patchset: Hashar; "mobile always uses role::cache::configuration" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54864 [08:45:20] mark: I fully agree that makes more sense. I have originally cut and paste the line without actually thinking about it [08:45:20] :D [08:45:35] me too [08:45:41] (in my head ;) [08:45:58] brb [08:47:39] back [08:47:43] let's see if we can merge htis... [08:50:12] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59380 [08:50:31] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54864 [08:52:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:20] hashar: if you rebase 44709 I think it can go in too [08:53:52] ah needs another upload change first [08:54:02] I am rebasing another change [08:54:07] https://gerrit.wikimedia.org/r/#/c/54863/ [08:54:10] for the upload cache [08:54:13] ok [08:54:28] Fatal error: require() [function.require]: Failed opening required '/usr/local/apache/common-local/php-1.22wmf2/redirect.php' (include_path='.:/usr/share/php:/usr/local/apache/common/php') in /usr/local/apache/common-local/w/redirect.php on line 3 [08:54:40] New patchset: Hashar; "upload cache in labs now uses role::cache::configuration" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54863 [08:54:58] Did someone delete redirect.php? [08:55:33] seems so :-D [08:55:36] Looks to be missing [08:55:40] Who/why/where/when/how? [08:56:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.146 second response time [08:56:20] Reedy: use your git skills : git lg -- redirect.php --stat [08:56:21] hashar: let's just move "backend" into role::cache::configuration as "swift" [08:56:25] Reedy: * f9e5457 - Remove completely unused $wgRedirectScript/redirect.php (2 weeks ago) [08:56:25] varnish doesn't exist in tampa (and won't) [08:56:47] mark: it does in tampa, on the beta cluster :-] [08:56:54] hashar: I'm currently getting ready heading out of the door ;) [08:56:57] mark: we don't have swift either yet. [08:57:08]