[00:03:44] !log testing test [00:04:24] !log testing test [00:05:31] !log testing test [00:08:02] !log testing test [00:08:27] !log testing test [00:08:30] Logged the message, Master [00:08:41] !project testing [00:08:41] https://labsconsole.wikimedia.org/wiki/Nova_Resource:testing [00:08:55] muahahahahaha [00:09:45] !log testing test [00:10:00] well, that's a fail [00:10:16] !log testing test [00:10:24] hm. infinite loop? [00:10:31] !log testing test [00:10:36] nope [00:10:41] not reporting correctly, though [00:10:52] Not a trusted nick or cloak. This is just a warning, for now. Please add your nick or cloak added to the trust list or your user page. [00:10:52] Logged the message, Master [00:10:52] Not a trusted nick or cloak. This is just a warning, for now. Please add your nick or cloak added to the trust list or your user page. [00:10:52] Logged the message, Master [00:10:52] Not a trusted nick or cloak. This is just a warning, for now. Please add your nick or cloak added to the trust list or your user page. [00:10:59] ah [00:11:04] that's incredibly slow [00:11:20] I need to cache that search [00:21:27] !log testing test [00:22:04] !log testing test [00:22:33] !log testing test [00:24:10] !log testing test [00:25:17] !log testing test [00:26:24] !log testing test [00:29:28] !log testing test [00:30:03] !log testing test [00:31:24] !log testing test [00:31:24] Not a trusted nick or cloak. This is just a warning, for now. Please add your nick or cloak added to the trust list or your user page. [00:31:25] Logged the message, Master [00:31:28] !log testing test [00:31:28] Not a trusted nick or cloak. This is just a warning, for now. Please add your nick or cloak added to the trust list or your user page. [00:31:29] Logged the message, Master [00:31:32] \o/ [00:31:52] !log testing test [00:31:52] Not a trusted nick or cloak. This is just a warning, for now. Please add your nick or cloak added to the trust list or your user page. [00:31:52] Logged the message, Master [00:32:11] !log testing test [00:32:12] Logged the message, Master [00:32:15] sweet [00:36:44] !log testing test [00:36:46] Logged the message, Master [00:37:07] !log testing test [00:37:07] Not a trusted nick or cloak. Was your nick or cloak added to the trust list or your user page? [00:44:19] !log testing test [00:44:21] Logged the message, Master [00:46:28] !log bots updated labs-morebots to lower the cache from 30 minutes to 5 minutes [00:46:30] Logged the message, Master [00:47:20] !log bots added feature to labs-morebots to check users against a trust list; we need user profiles before we can realistically enable this. [00:47:21] Logged the message, Master [02:41:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:59:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 15% free memory [03:40:13] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [03:41:01] !log Testing My first attempt [03:41:01] Testing is not a valid project. [03:48:12] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Warning: 16% free memory [04:05:13] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 22% free memory [04:13:12] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f.pmtpa.wmflabs output: Critical: 4% free memory [04:18:12] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f.pmtpa.wmflabs output: OK: 95% free memory [04:50:58] 10/12/2012 - 04:50:57 - Created a home directory for madman in project(s): cvresearch [04:56:00] 10/12/2012 - 04:55:59 - User madman may have been modified in LDAP or locally, updating key in project(s): cvresearch [05:28:52] PROBLEM Disk Space is now: WARNING on echo-xmpp i-00000351.pmtpa.wmflabs output: DISK WARNING - free space: / 571 MB (5% inode=91%): [05:33:53] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [06:28:53] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 22% free memory [06:53:52] RECOVERY Disk Space is now: OK on echo-xmpp i-00000351.pmtpa.wmflabs output: DISK OK [07:36:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [08:01:52] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 21% free memory [08:54:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [10:51:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-000003ef.pmtpa.wmflabs output: Critical: 5% free memory [11:27:12] PROBLEM Disk Space is now: WARNING on deployment-jobrunner06 i-0000031d.pmtpa.wmflabs output: DISK WARNING - free space: / 572 MB (5% inode=82%): [11:45:53] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [13:03:39] Hello unicorns! A question concerning the labs instance configuration interface: What exactly do I get when I check the appserver stuff? [13:05:57] !log wikidata-dev wikidata-dev-2: Modified apache conf for test-repo to rewrite URLs (#40389) [13:05:59] Logged the message, Master [13:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [13:12:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [13:18:58] Or - more generally - whom can I bother with labs - puppet questions here? [13:22:33] jeremyb: poke [13:28:15] Ryan_Lane, too early for you? [13:28:30] too early [13:28:53] PROBLEM Current Load is now: CRITICAL on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: Connection refused by host [13:28:54] Silke_WMDE: anyone or andrewbogott [13:29:31] ok [13:29:33] PROBLEM Current Users is now: CRITICAL on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: Connection refused by host [13:30:12] PROBLEM Disk Space is now: CRITICAL on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: Connection refused by host [13:30:52] PROBLEM Free ram is now: CRITICAL on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: Connection refused by host [13:30:52] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 22% free memory [13:32:22] PROBLEM Total processes is now: CRITICAL on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: Connection refused by host [13:32:46] hiya, anyone know why bastion.wmflabs.org closes my connection as soon as it lets me in? [13:33:12] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: Connection refused by host [13:34:14] you have a rubbish connection? [13:34:30] naw, and actually, i take some of that back, i think it isn't letting me in [13:34:32] RECOVERY Current Users is now: OK on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [13:34:46] i saw the 'if you are having access problems' message [13:34:50] and assumed that was the motd [13:34:54] but i don't think it is [13:35:07] i get [13:35:08] debug1: Offering RSA public key: /Users/otto/.ssh/id_rsa-wmf [13:35:08] debug1: Server accepts key: pkalg ssh-rsa blen 279 [13:35:08] Connection closed by 208.80.153.207 [13:35:12] RECOVERY Disk Space is now: OK on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: DISK OK [13:35:52] RECOVERY Free ram is now: OK on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: OK: 3139% free memory [13:36:32] Dunno without seeing the logs, might be the mount for keys being rubbish. It let me in though [13:36:56] the mount for keys being rubbish? like in my ~/.ssh dir? [13:37:22] RECOVERY Total processes is now: OK on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: PROCS OK: 106 processes [13:37:26] yeah its weird, it doesn't even deny my key, it just closes the connection [13:38:12] RECOVERY dpkg-check is now: OK on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: All packages OK [13:38:52] RECOVERY Current Load is now: OK on wikidata-dev-4 i-000004bb.pmtpa.wmflabs output: OK - load average: 0.10, 0.42, 0.41 [13:39:25] they don't live there [13:40:00] /public/keys//.ssh is actually where they live [13:43:24] can you see if my key is there? [13:43:27] can you just ls that dir? [13:46:18] I dont understand what happens on my instance when I check the appserver boxes. Can anyone (or you, andrewbogott, if back) explain this to me? How mich is already there to play with Solr and where? o_O [14:18:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [14:29:52] Silke_WMDE: What's the class you ticked? [14:30:03] ottomata: mount seems to be ok, not sure [14:30:11] Hmm today has strange feelings [14:30:28] yeah totally weird [14:30:59] There's a few reasons it might do that, non of which I can really poke nor replicate. [14:31:56] Damianz: applicationserver::api, bits, homeless, jobrunner -> what are they? [14:32:08] 1sec [14:33:52] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 22% free memory [14:34:35] they relate to the files under https://gerrit.wikimedia.org/r/gitweb?p=operations/puppet.git;a=tree;f=modules/applicationserver/manifests;h=3c9eff870b223d07975e40a14ca40711ba0a1388;hb=HEAD [14:35:31] Hmm no [14:35:34] so wtf [14:35:38] that class doesn't actually exist [14:35:42] applicationserver_old::api does [14:35:47] the new stuff is under role::applicationserver::appserver::api [14:35:54] which is related to that module from the look of it [14:37:00] ah. For some reason I didn't see the role directory [14:37:22] so these are a bit weird [14:37:48] because class appserver::api{ lives in class role::applicationserver { it's like saying class role::applicationserver::appserver::api { [14:38:10] But yeah, we're suppose to be moving to use modules with a role class in the role dir that basically configures the class [14:39:18] Can you explain me how to use a file I find there on my labs instance? e.g. lucene? [14:41:21] err [14:41:30] So you want to say install the 'search stack' on your instance? [14:41:40] yes [14:43:44] So (not that I really understand lucene) you'd want to add lucene::server as a puppet class under a group (that just lets you add it in the interface), then tick it and run puppetd -tv, that should then install it. [14:44:01] It doesn't have a role class but (unless you want to specify args) it should work, I think [14:44:15] * Damianz thinks we need to document the puppet stuff and work on documenting the puppet classes/pulling them into the interface [14:47:37] The question is do I want to write this in ruby or do I want to write a parser in python [14:48:38] Why does AST suck so much to work with [14:55:34] Damianz: I dont have to tick it in the interface - would be sufficient to understand how to use the puppet files. Where to put them? [14:55:50] Do I need a local puppetmaster for it? [14:56:15] Porbably not... [14:56:16] You don't need a local puppetmaster unless you're developing the manifests [14:56:52] For new stuff it should go under modules/something/{manifests,templates,files} for current stuff it goes under manifests/somethign [15:00:38] so I copy it there and then what? [15:04:57] paravoid: do we have any way to add more memory to a labs instance ? [15:05:08] not yet afaik [15:05:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [15:05:46] paravoid: even by using some hidden / root only commands in openstack ? [15:05:52] no [15:06:17] We tried it the other day on a test instance before doing bots - it requires ssh to places it can't ssh too and breaks the instance [15:06:40] * Damianz wonders what Silke_WMDE is actually trying to do [15:09:25] :) [15:10:06] * Silke_WMDE is trying to understand how to use puppet without access to the puppetmaster [15:11:14] Well /using/ it is done via labsconsole, which updates ldap, which puppetmaster uses to figure out what's suppose to be on a node. [15:11:43] To develop puppet stuff use puppetmaster::self then request it to be merged into the production branch of the puppet repo which will get pulled to the puppetmaster [15:15:04] hm, ok. [15:16:34] So what am I talking about here, using or developing? [15:18:32] Using I think... though my assumption of everyone knowing puppet hinders this. [15:18:48] The main difference for using in labs is the classes are stored in ldap rather than node definitions like production [15:19:21] ok [15:21:59] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 40983] beta autoupdater does not work anymore - https://bugzilla.wikimedia.org/show_bug.cgi?id=40983 [15:23:34] Damianz: Thanks for explaning! [15:24:06] * Damianz did a really rubbish job of explaining [15:25:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 97 processes [15:27:22] :) [15:37:43] PROBLEM Total processes is now: WARNING on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS WARNING: 154 processes [15:40:05] is there a web proxy for instances working now? [15:41:31] no [15:44:03] or any attempt done? [15:45:25] not really [15:57:42] RECOVERY Total processes is now: OK on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS OK: 146 processes [16:39:43] New patchset: jan; "Add $wfSharedDB and -Tables" [labs/centralauth] (shared-wiki-config) - https://gerrit.wikimedia.org/r/27735 [16:40:35] Change merged: jan; [labs/centralauth] (shared-wiki-config) - https://gerrit.wikimedia.org/r/27735 [17:03:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [17:39:28] * Damianz waves at Ryan [17:46:24] Damianz: howdy [17:50:03] Dang, my backscroll paged me right before colloquy crashed and threw away the messages. I hope whoever it was doesn't mind repeating themselves [17:52:24] andrewbogott_afk: From what I've seen it was mostly about puppet help, which I think I sorta cleared up and made slightly more confusing [17:52:29] * Damianz brb, f00d [17:58:52] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 22% free memory [18:15:40] hi Ryan_Lane [18:15:58] 10/12/2012 - 18:15:57 - Creating a project directory for project-proxy [18:15:58] 10/12/2012 - 18:15:57 - Created a home directory for laner in project(s): project-proxy [18:17:09] liangent: howdy [18:17:28] Ryan_Lane: seems you're doing that for me. [18:17:32] yeah [18:17:34] all done [18:17:36] ready to go [18:17:43] and there's another issue, 'git' seems unavailable from apt? [18:18:00] it's git-core [18:18:13] hell if I know why [18:18:15] <^demon|busy> I thought that was renamed. [18:18:29] <^demon|busy> There was another project called git originally in the apt repos. [18:18:32] in lucid is is for sure [18:18:39] maybe in precise it's git? [18:19:07] hmm my server is lucid [18:19:27] <^demon|busy> I think it was in between Lucid and Precise when it changed. [18:19:31] <^demon|busy> On a non-LTS release. [18:19:55] what will be the difference between installing from apt or puppet? [18:19:58] We really need to fix the git-core package as you can't install it anywhere without that class... either standardize on wrapping stuff in classes or just kill it with fire [18:20:05] liangent: Puppet is replicatable [18:20:57] Damianz: I mean when I want to install something for my own use? [18:20:59] 10/12/2012 - 18:20:59 - User laner may have been modified in LDAP or locally, updating key in project(s): project-proxy [18:21:00] <^demon|busy> Original git package: http://packages.ubuntu.com/hardy/git [18:21:05] <^demon|busy> "GNU Interactive Tools" [18:21:20] for example, I want to install apache, should I type apt-get install apache2 or go to labsconsole and select from webserver/ [18:21:28] select it from labsconsole [18:21:50] Damianz: any advantage then? [18:21:53] I kinda hate puppet lets you tick multiple classes on a side note - it encourages people to not wrap things in service classes which makes boxes hard to duplicate #firstworldproblems [18:22:04] liangent: If you need another box it's installed identically. [18:23:05] Damianz: so it'll follow my configuration changes done on my webserver installed with puppet? [18:23:20] and copy them to the new box when needed [18:23:27] Well nope, but the idea is you'd puppetize your configure changes and push them back into the repo. [18:23:31] Which for apache, got help you [18:23:32] liangent: if you use puppet, then if you need to rebuild (like for instance, the instance gets owned), it's way easier [18:23:35] <^demon|busy> Damianz: Really, I've wondered if git shouldn't be a part of the core install--I really can't think of a server that wouldn't want it intalled. [18:23:39] <^demon|busy> Ryan_Lane: Thoughts on that? [18:23:42] or if you want to move from lucid to precise [18:23:46] You might not want it on prod servers [18:23:47] ^demon|busy: it probably should [18:23:51] Damianz: why not? [18:23:59] it's going to be on all application servers soon [18:24:06] we're using git for deployment soonish [18:24:14] In the same sense you don't install ethertool on prod servers [18:24:22] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-000003ef.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:24:22] PROBLEM Total processes is now: CRITICAL on dumps-bot3 i-000003ef.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:24:42] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-000003ef.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:24:42] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-000003ef.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:24:47] Imo git should be, but it depends on requirements for keeping skinny [18:25:02] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-000003ef.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [18:26:58] so when I'm trying to do something new and before I fully tune it I don't really need to use puppet? [18:27:20] puppetmaster::self! [18:27:36] what? [18:29:12] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-000003ef.pmtpa.wmflabs output: Server answer: [18:30:34] lets you develop puppet manifests locally so you can edit configs within puppet then push the changes back [18:30:51] * Damianz really wishes we forced https on wikipedia [18:33:51] Damianz: any user guide about this? [18:34:01] !puppetmaster::self [18:34:01] https://labsconsole.wikimedia.org/wiki/Help:SelfHostedPuppet [18:34:23] !puppetmaster::self|liangent [18:34:24] liangent: https://labsconsole.wikimedia.org/wiki/Help:SelfHostedPuppet [18:35:56] Related, does anyone have an opinion on making puppetmaster::self do a git pull automatically? It wouldn't overwrite work as it would just conflict on the merge but would keep everything else up to date and avoid massive diverges which is basically what killed the test branch [18:37:20] which instance type should I choose this time? [18:37:46] maybe puppet would help me if I underestimate its size so I can create another without redoing everything manually [18:38:17] anything that's not tiny [18:38:56] We have a lot of stuff not in puppet, it makes upgrading and moving instances around painful and manual. [18:42:15] Ryan_Lane: where's a public ip? [18:42:41] ah [18:42:42] right [18:42:42] sec [18:43:41] o.0 so the mounts for my Panniers cost near enough the same as the Panniers! [18:43:51] liangent: you can allocate one now [18:44:24] ok. time to set up a webplatform project [18:44:29] they can't keep testing live [18:45:07] <^demon|busy> Testing live worked for us for years. [18:45:13] <^demon|busy> They're a new project, they can still test live ;-) [18:45:15] "worked" [18:45:33] I keep having to purge the entire cache so that they can test skin changes [18:45:40] that's not going to work [18:45:52] Change on 12mediawiki a page Wikimedia Labs/Terms of use was modified, changed by Andrew Bogott link https://www.mediawiki.org/w/index.php?diff=593087 edit summary: Loosen the content restrictions to include non-cc licenses (mostly to allow ODbL) [18:45:58] 10/12/2012 - 18:45:58 - Creating a project directory for webplatform [18:45:58] 10/12/2012 - 18:45:58 - Created a home directory for laner in project(s): webplatform [18:45:59] <^demon|busy> Teach them how to purge the cache ;-) [18:46:00] ^demon|busy: Wait, you don't test live anymore?! *looks at recent outages* :P [18:46:37] Ewww no, never teach them [18:46:40] Just give them a button [18:46:50] Developers are capable of clicking not of following instructions [18:47:44] <^demon|busy> I resent that. I can follow instructions. [18:47:48] <^demon|busy> When they're good instructions. [18:48:15] Well ops people aren't known for writing good instructions aimed at people who don't know what they're doing :P [18:48:24] Ryan_Lane: done. wondering whether you were assigning it to me or my project? [18:48:29] project [18:48:40] the assign ip form will let you select an ip and instance [18:48:46] then it magically goes and NATs it [18:48:53] PROBLEM Current Load is now: CRITICAL on instance-proxy i-000004bc.pmtpa.wmflabs output: Connection refused by host [18:49:06] Change on 12mediawiki a page Developer access was modified, changed by Paul.irish link https://www.mediawiki.org/w/index.php?diff=593088 edit summary: [18:49:33] PROBLEM Current Users is now: CRITICAL on instance-proxy i-000004bc.pmtpa.wmflabs output: Connection refused by host [18:49:48] Hmm I half want ice cream and I half want to destroy the project request queue with fire and replace the shell request one with a generic form but also don't feel like fighting SMW forms to apply logic [18:49:57] Damianz: but why do I still need to click "allocate" for that project then tie it to an instance? [18:49:58] ^demon|busy, Damianz: no. people shouldn't be purging the cache [18:50:07] liangent: it's tied to the project [18:50:12] PROBLEM Disk Space is now: CRITICAL on instance-proxy i-000004bc.pmtpa.wmflabs output: Connection refused by host [18:50:23] liangent: Because you might have multiple instances and it can't read your mind? [18:50:32] liangent: in "manage addresses" you need to click "Allocate IP" in your project [18:50:47] Ryan_Lane: Selective purging and for style it should only be like 1 css file assuming you're minimizing etc on deploy which you probably should be. [18:50:52] PROBLEM Free ram is now: CRITICAL on instance-proxy i-000004bc.pmtpa.wmflabs output: Connection refused by host [18:50:58] 10/12/2012 - 18:50:57 - User laner may have been modified in LDAP or locally, updating key in project(s): webplatform [18:51:42] Damianz: yeah, it's not terribly easy with resource loader [18:51:50] I don't get to play with CDNs at the moment as we use Akamai, 3months and we get to build our own as we spend far, far, far too much money with Akamai which will be fuuuuun :D [18:52:00] liangent: I didn't add an IP to your project [18:52:04] liangent: I changed your quota [18:52:10] from 0 to 1 [18:52:23] Ryan_Lane: yeah that looks more clear :) [18:52:42] sorry. I should be more clear when I explain that to people :D [18:52:56] meh, screw being clear [18:53:01] document it them tell people to rtfm [18:53:01] Change on 12mediawiki a page Developer access was modified, changed by Leaverou link https://www.mediawiki.org/w/index.php?diff=593090 edit summary: [18:53:15] repeating information is dull [18:53:52] RECOVERY Current Load is now: OK on instance-proxy i-000004bc.pmtpa.wmflabs output: OK - load average: 0.35, 0.67, 0.43 [18:54:22] so the first connection is always fail with "Permission denied (publickey)." ? [18:54:22] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK WARNING - free space: /export 596 MB (3% inode=59%): [18:54:29] and the second try is ok [18:54:32] RECOVERY Current Users is now: OK on instance-proxy i-000004bc.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [18:54:46] Change on 12mediawiki a page Developer access was modified, changed by Paul.irish link https://www.mediawiki.org/w/index.php?diff=593091 edit summary: /* User:Paul.irish */ [18:54:56] to which instance? [18:55:12] RECOVERY Disk Space is now: OK on instance-proxy i-000004bc.pmtpa.wmflabs output: DISK OK [18:55:18] bleh [18:55:19] Damianz: a newly created one [18:55:36] that's not really helpful, I was thinking name/instance id [18:55:42] Time to clear nfs again it seems [18:55:50] Damianz: instance-proxy [18:55:52] RECOVERY Free ram is now: OK on instance-proxy i-000004bc.pmtpa.wmflabs output: OK: 1452% free memory [18:56:14] * Damianz questions petan's maths [18:57:07] so how to have *.wmflabs.org resolved to my ip? [18:57:28] <28>Oct 12 18:56:28 instance-proxy nslcd[13294]: [cdf8f6] error writing to client: Broken pipe [18:57:31] diaf [18:57:37] also puppet seems broken on that instance. [18:57:51] Damianz: broken in what way? [18:58:00] Dependancy failures [18:58:05] Run puppetd -tv and see what it says [18:58:09] Might have fixed its self. [18:58:33] *.wmflabs.org isn't going to happen [18:58:54] Change on 12mediawiki a page Developer access was modified, changed by Ryan lane link https://www.mediawiki.org/w/index.php?diff=593092 edit summary: [18:58:55] you can have .wmflabs.org though [18:59:06] Also I hope you're not using apache to do instance proxying. [18:59:09] Kittens will die. [18:59:18] Damianz: of course "something" [18:59:33] huh what will happen? [18:59:47] nginx would be better :) [19:00:04] I wouldn't try using apache if it was the last webserver on earth [19:00:22] haproxy or nginx, prefrably nginx so ssl and lb support can be added [19:00:40] Kibana is rather helpful for seeing issues... shame we don't take auth logs though [19:00:50] Makes sense for security but meh for login issues [19:01:17] Now an awesome thing to do would be to have logstash trigger nagios alters for stuff it sees in the logs. [19:01:37] so the dns things? [19:02:40] https://labsconsole.wikimedia.org/wiki/Special:NovaAddress 'add host name' [19:03:13] Btw, that interface really sucks and we should turn it into a combined proxy/ip management page [19:03:39] Or even just proxy everything, only put public ips on lvs boxes and NAT everything outside of openstack [19:04:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 190 processes [19:04:49] Damianz: it doesn't accept a wildchar like *.instance-proxy [19:05:02] ask Ryan_Lane to add a domain [19:05:40] labs-home-wm: You know what, unless you're going to transpose that onto a graph so it's useful, shut up. Historic data in a human understandable form such as a graph for service state/value is useful, spam is not. [19:06:16] sec [19:06:51] liangent: ok, now you can add a * entry to instance-proxy domain [19:09:44] Ryan_Lane: done [19:10:42] Everytime I get the feeling of just wanting to trash the puppet repo and starting from scratch I know I need more ice cream [19:10:49] heh [19:11:37] Though yay to only having 1 change open [19:14:36] Change on 12mediawiki a page Developer access was modified, changed by Nad link https://www.mediawiki.org/w/index.php?diff=593098 edit summary: [19:16:36] heyaaa, can someone help me connect to bastion.wmflabs.org? [19:16:39] it used to work for me [19:16:41] and now no more [19:16:54] https://gist.github.com/d7b8c9c8d85adfb91cb9 [19:17:37] probably Ryan_Lane as I can't see the logs [19:19:32] ottomata: try now [19:19:46] same [19:19:46] Connection closed by 208.80.153.207 [19:19:50] oh [19:19:51] right [19:19:57] you're in the ops group now [19:20:04] ottomata: you need to use bastion-restricted.wmflabs.org [19:20:07] oh! [19:20:20] well howdy doody [19:20:22] that works [19:20:45] Thanks! [19:21:48] <^demon|busy> Ryan_Lane: Daniel put in a change for us :) https://gerrit.wikimedia.org/r/#/c/27753/ [19:22:42] some issues again, maybe on my side, but I don't know how to resolve it: [19:22:43] merged [19:22:47] I can dig instance-proxy.wmflabs.org but not ping instance-proxy.wmflabs.org [19:22:56] with ping: unknown host instance-proxy.wmflabs.org [19:23:02] liangent: it doesn't have a DNS record [19:23:18] eh? [19:23:22] I remember I added one [19:23:32] dig @labs-ns0.wikimedia.org en.instance-proxy.wmflabs.org [19:23:41] en.instance-proxy.wmflabs.org. 3600 IN A 208.80.153.147 [19:23:49] hm [19:23:52] it digs for me [19:23:57] dig @labs-ns0.wikimedia.org instance-proxy.wmflabs.org [19:24:02] instance-proxy.wmflabs.org. 3600 IN A 208.80.153.147 [19:26:57] Ryan digs it [19:26:58] brb phone [19:28:33] liangent: did you try to access it before the entry was added? [19:28:38] if so it has a negative cache [19:28:45] and will likely be about an hour [19:29:30] Ryan_Lane: it works now, without any manual change [19:29:35] ah [19:29:35] cool [19:37:52] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [19:52:52] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 41% free memory [20:04:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 287 processes [20:07:38] how should I try to protect those badly-configured instances? [20:07:53] aude: yah? [20:08:04] badly configured? [20:09:28] Damianz: if an instance is open to all other instances but not the world, and for example, runs a badly configured webserver, or a service that can misread an http request [20:09:39] so with my proxy it's exposed to the world [20:09:54] this is why I hate your idea [20:10:20] security groups stop the issue though [20:10:33] unless you open aceess its blocked [20:10:57] hmm [20:14:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 194 processes [20:14:58] so just done as a quick work at .instance-proxy.wmflabs.org or optionally ..instance-proxy.wmflabs.org [20:19:36] fuck you gerrit [20:21:53] <^demon|busy> Gerrit didn't do anything wrong :p [20:23:39] it was dumb [20:23:44] and ignored the change id [20:27:24] how long is git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git expected to take on labs ? [20:28:34] <^demon|busy> An initial clone of core to anywhere takes awhile. It's roughly ~150M repacked. [20:28:55] Oh also as chad is here [20:28:58] GITWEB SUCKS [20:29:12] <^demon|busy> I KNEW THAT BEFORE YOU DID [20:29:29]