[00:05:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.44, 5.81, 5.20 [00:06:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:07:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:16:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.70, 5.99, 5.46 [00:36:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:38:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:55:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.97, 4.22, 4.97 [00:56:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.45, 3.46, 4.44 [01:03:54] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.42, 5.43, 5.22 [01:06:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:07:53] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:08:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:12:52] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:37:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:38:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:07:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:08:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:38:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:38:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:08:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:09:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:39:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:39:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:41:05] Change on 12mediawiki a page Developer access was modified, changed by Thibaultmarin link https://www.mediawiki.org/w/index.php?diff=604111 edit summary: [04:07:02] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.60, 4.63, 4.91 [04:09:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:09:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:15:13] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 4.96, 5.09, 5.06 [04:39:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:39:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:10:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:10:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:20:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.45, 4.82, 4.99 [05:41:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:41:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:43:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 6.17, 5.46, 5.16 [05:53:53] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.84, 4.50, 4.89 [06:11:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:11:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:29:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:41:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:41:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:49:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [07:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:11:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:41:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:41:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:11:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:20:21] Hi! is anyone present and awake who knows something about the wmf-vagrant project on github? [08:42:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:05:39] Silke_WMDE: yup [09:05:47] what's up? [09:05:58] (for very small values of "awake") [09:12:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:12:10] ori-l: I forked the project for wikidata. I keep running into the problem that the mediawiki install script fails to create a mysql database. When I execute the same command inside the vagrant machine it works. But on the puppet side it fails. Do you have any idea why? [09:12:31] how long ago did you fork it? [09:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:14:08] zeljkof uncovered a bug a day or two ago: "apt-get update" would sometime run _after_ "apt-get install mysql", which would cause the machine to try and fetch an old version of mysql that is no longer on canonical's servers. [09:15:31] ori-l: the bug is still in the repo? or did your recent changes fix it? [09:15:39] ori-l: about a week ago [09:15:55] this would be consistent with what you're describing: 1) puppet runs `apt-get install mysql` and fails because it tries to retrieve a stale package, 2) puppet runs `apt-get update`, 3) you run `apt-get install mysql` which now magically works because the apt cache is up-to-date. [09:16:01] but I merged zeljkofs changes [09:16:03] zeljkof: no, it's fixed [09:16:13] hmmmm. [09:16:31] can you still scroll back and see the output for puppet? [09:16:56] yes, but it doesn't give any output concerning the db creation [09:18:19] Silke_WMDE: can you try cloning wmf repo? [09:19:02] zeljkof: which one? [09:19:14] http://github.com/wikimedia/wmf-vagrant [09:20:00] Silke_WMDE: ori-l was faster :) if you clone that repo it should work fine [09:20:51] ok, I'll try. [09:21:56] Silke_WMDE: let me know how it goes [09:22:04] well, let _us_ know [09:22:04] :) [09:22:08] ok [09:22:14] thanks for the hint [09:22:43] np, thanks zeljkof [09:42:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:43:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:42:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:43:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:03:33] PROBLEM Free ram is now: WARNING on swift-be4 i-000001ca.pmtpa.wmflabs output: Warning: 19% free memory [11:12:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:14:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:42:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:44:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:12:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:14:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:42:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:44:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs CRITICAL - Plugin timed out after 10 seconds [13:13:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:14:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:43:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:44:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:13:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:14:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:43:34] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:45:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:04:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 227 processes [15:13:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:15:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:20:44] I'd like to request an account on labs. [15:20:49] !ping [15:20:49] pong [15:24:22] Change on 12mediawiki a page Developer access was modified, changed by Tim1357 link https://www.mediawiki.org/w/index.php?diff=605089 edit summary: [15:26:23] Tim1357: Do you already have a gerrit account? [15:26:47] Nope. I had a toolserver account but that's apparently not a thing anymore. [15:27:01] ok! Stay tuned... [15:28:20] Tim1357: OK, I need your on-wiki name, your preferred shell name, and your email address. Feel free to pm me if you'd prefer not to broadcast these things on-channel. [15:29:12] I have a request here, if that's what you were talking about. http://www.mediawiki.org/wiki/Developer_access#User:Tim1357 [15:29:32] But if not: Tim1357 (enwikip), Tim1357, sears.tim@gmail.com [15:29:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 134 processes [15:29:46] oh, that works. Hm… I wonder why there's such a backlog there. [15:32:18] Tim1357: OK, while that is processing… have you already connected with admins of a specfic project, or are you hoping to start your own? [15:32:36] Well this is what I'm confused about. [15:32:53] Do tell [15:32:58] I'd like to migrate my things from toolserver to tools lab. [15:33:15] And I'm not sure if I can do that yet. [15:34:34] Change on 12mediawiki a page Developer access was modified, changed by Andrew Bogott link https://www.mediawiki.org/w/index.php?diff=605093 edit summary: /* User:Tim1357 */ [15:35:03] It depends a bit on what your things are. [15:35:28] We don't have a replication of the wikipedia data available yet, and in general our db-sharing isn't great. [15:35:51] There's already an active and working 'bots' project. [15:35:59] And we can provide many other resources... [15:36:02] depends on what you need. [15:36:52] PROBLEM Current Load is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: WARNING - load average: 10.02, 9.17, 6.82 [15:37:07] andrewbogott: Can you fix something? [15:37:40] Hydriz, what's up? [15:37:42] I am getting "Creating directory '/home/hydriz'. [15:37:42] Unable to create and initialize directory '/home/hydriz'." [15:37:51] when logging in to dumps-2 [15:38:20] Hm… I've seen this before but we weren't able to figure out why it happened… last time it was just a failed instance creation (a race condition or something.) [15:38:30] Hydriz, is this an instance you're attached to or is it brand new? [15:38:46] brand new [15:39:07] Do you mind scrapping it and trying a new one? Might just work on a second attempt. [15:39:10] and this is the second time, I recreated the instance a few hours before the second creation [15:39:20] Oh… same failure both times? [15:39:27] yep [15:39:36] dang. Ok, lemme see if I can reproduce. [15:39:40] the only weird thing on-wiki is that the instance page ain't created [15:40:48] Huh, what is the hostname to ssh into labs? wmflabs.org doesn't work... [15:41:03] Matthew_: bastion.wmflabs.org [15:41:06] !bastion [15:41:06] http://en.wikipedia.org/wiki/Bastion_host; lab's specific bastion host is: bastion.wmflabs.org which should resolve to 208.80.153.194; see !access [15:41:53] RECOVERY Free ram is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK: 86% free memory [15:42:01] Hmmm... permission denied. [15:42:49] do you have access at all? [15:42:51] !access [15:42:51] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [15:43:26] No, I don't >.< [15:43:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:43:59] lol, you need to request for shell access [15:44:40] Ya, I just realized I forgot to note "labs" on my request for developer access ... >.< [15:45:15] Matthew, what is your account name? [15:45:23] PROBLEM host: i-00000503.pmtpa.wmflabs is DOWN address: i-00000503.pmtpa.wmflabs CRITICAL - Host Unreachable (i-00000503.pmtpa.wmflabs) [15:45:31] andrewbogott: matthewrbowker [15:45:51] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:46:47] Matthew_: Did you set up your keypair already? If so you should be able to access in a minute or two. [15:47:01] andrewbogott: In gerrit? Yes. [15:47:06] Thanks :) [15:47:30] Matthew_: No, here: https://labsconsole.wikimedia.org/wiki/Special:NovaKey [15:47:43] (Pretty sure that's a different keyset from gerrit.) [15:48:13] andrewbogott: OK. Done. Thanks :) [15:48:35] (the one I added on gerrit isn't there, so I'd say it is a different keyset) [15:48:41] Matthew_: You're already coordinating with someone on an existing project? [15:48:53] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 0.39, 0.18, 0.06 [15:49:03] RECOVERY host: i-00000503.pmtpa.wmflabs is UP address: i-00000503.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 9.81 ms [15:49:14] finally, dumps-bot3 is back [15:49:25] andrewbogott: I'm considering porting my Toolserver tools over. I wanted to start doing some testing/debugging to see what I can do. [15:50:13] Matthew_: Ah, OK. So… labs is a cloud platform. So there isn't a single host, rather you will be creating and deleting hosts as needed. [15:50:21] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [15:50:21] So logging into bastion won't really teach you anything interesting, it's just a gatekeeper. [15:50:40] 11/13/2012 - 15:50:20 - Creating a home directory for matthewrbowker at /export/keys/matthewrbowker [15:50:49] (all of this goes for Tim1357 as well. I hope I'm not being too cryptic.) [15:51:23] RECOVERY Total processes is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: PROCS OK: 122 processes [15:51:43] RECOVERY dpkg-check is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: All packages OK [15:51:47] andrewbogott: OK. But (by reading the link Hydriz gave me) I understand that I need to ssh into bastion before I can create my own project. [15:51:53] RECOVERY Current Load is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK - load average: 0.17, 1.39, 4.10 [15:52:03] RECOVERY Free ram is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK: 1601% free memory [15:52:06] no, you can't create a project, you need someone else to do it for you [15:52:32] Hmmmm... OK. [15:52:41] Matthew_: I recommend you starting here: https://labsconsole.wikimedia.org/wiki/Help:Getting_Started#Projects [15:52:43] RECOVERY SSH is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [15:52:53] RECOVERY Current Load is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK - load average: 0.25, 1.36, 4.10 [15:53:19] Hydriz: Do those log messages mean that your new instances are starting to behave properly? I'm digging in the logs but don't see anything interesting yet. [15:53:33] which log messages? [15:53:51] Um… the nagios stuff. [15:53:57] And your recent comment re: dums-3 [15:54:10] *dumps-3 [15:54:15] andrewbogott: Ok, thanks. This is different than Toolserver :/ [15:54:24] andrewbogott: These instances nagios is reporting are those that are created a long time ago [15:54:38] I just restarted them, they already behaved properly from the beginning :P [15:54:39] Matthew_ and Tim1357: In labs a 'project' is a group of instances that are under the control of a particular team. If you each want to be a team of one then that's fine, it just means you want a private project. [15:55:17] Matthew_ and Tim1357: Then within a project you can create, destroy, and configure servers yourself as needed. Those servers are private to the project. [15:55:22] (So, yes, very different from Toolserver.) [15:55:24] 11/13/2012 - 15:55:24 - Updating keys for matthewrbowker at /export/keys/matthewrbowker [15:56:01] its just solely dumps-2 that is behaving badly and was only recently created. [15:56:13] Oh, OK. [15:56:24] I will continue to dig then. [15:56:26] cool, we are getting database servers for labs [15:56:43] Huh, OK. So I can create a "project" that's accessible via the web, for all of my web-based tools? Or will that not work. [15:57:25] (just trying to get my head wrapped around how things work here) [15:59:43] bots-1:/tmp is rofl :P [15:59:53] one day I might get that box cleaned [16:00:15] Might be a good idea XD [16:00:30] Matthew_: At the moment because labs is in beta, you need to ask one of the elect to create a project. (In the future that will be user-driven...) [16:00:51] The project may or may not be publicly accessible depending on how you configure your security groups. [16:01:16] And, instances by default have a private IP that's only visible from w/in labs. You can map a public address and DNS name to a server later on if/when you want others to visit. [16:02:09] petan: btw, I tried to half-assedly repair the logbot, has it been treating you OK lately? [16:02:15] Matthew_: Making sense? Sort of? [16:04:10] andrewbogott: It does, actually. So, if I wanted to put my Tools (I have two varieties, bots and web-based), I could request two projects. I keep my "bots" one private, and run my bots as I need. I can also create the "web-based" tools, map that to be public, and allow Wikimedians to access them. Did I get that right? [16:04:47] Yep, that's right. [16:04:59] Yay :D [16:05:14] As it happens, there's already an existing bots community with a project and instances. It's a bit wasteful to have a whole VM for each bot, so best if you catch up with the existing bots people for resources there. [16:05:50] andrewbogott: Yeah, especially since my bots will only be activated once an hour :) [16:06:01] e.g. I believe that petan can hook you up. [16:06:11] hi [16:06:33] !log bastion test [16:06:33] I can't open the cache file, which means I have pretty much no idea what projects exist. [16:06:47] goddamn! [16:06:48] andrewbogott I don't think it works [16:06:50] :D [16:06:52] petan: OK, I will revisit :( [16:06:56] thanks [16:07:03] But, hey, I wrote that error message! Better than crashing, right? [16:07:04] andrewbogott: Nevermind, I have deleted the instance, will retry next time :) [16:07:16] Hydriz: OK, thanks. Let me know if it keeps failing. [16:07:21] sure [16:07:31] I'm confused, the scheduler said dumps-2 was going on virt7 but virt7 denies all knowledge. [16:07:43] All right, one more (slightly rediculous) question. Can I share MySQL databases between projects? [16:07:53] Matthew_ somehow yes [16:07:58] depends on security settings [16:08:02] heh [16:08:05] you can access foreign mysql server [16:08:12] from another project [16:08:19] if both projects are configured to allow that [16:08:43] Hmmm... OK. But I bet the Bots project doesn't allow that... [16:08:49] Matthew_: /within/ a project, you can put the db on a volume that's shared on all instances, so that's pretty easy. [16:09:04] (But that's not what you asked, I realize :) ) [16:09:18] Matthew_ I don't really see a reason why we shouldn't allow that [16:09:27] accessing foreign sql from bots project is fine [16:09:36] allowing foreign to access bots mysql servers is worse [16:10:22] but AFAIK it's not possible right now, once you are sure about the design of your application and it's clear if u need it let me know [16:10:39] petan: My bot needs to read a database that I have a web-based PHP form inserting into. Maybe I can just run them all in one project? [16:10:54] sure u can [16:11:00] there is apache server in bots project [16:11:14] cluebot works the sam [16:11:15] * same [16:11:20] and it lives there [16:12:25] OK. Makes sense. I'm just trying to figure out what's easiest (and most stable). [16:13:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:14:49] this is most easy [16:14:53] use local apache :P [16:16:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:16:24] Heh, maybe. [16:21:30] !log feeling better? [16:21:30] feeling is not a valid project. [16:21:35] petan, try it now? [16:21:44] !log bastion test [16:21:46] Logged the message, Master. [16:21:49] yay [16:22:00] :D [16:22:15] Puppet is maybe messing with the permissions, we'll see if it breaks again in an hour. [16:43:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:46:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:14:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:16:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:35:32] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 152 processes [17:40:32] RECOVERY Total processes is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS OK: 146 processes [17:44:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:46:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:03:32] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 152 processes [18:14:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:16:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:22:38] * andrewbogott back soon [18:27:08] Change on 12mediawiki a page Developer access was modified, changed by PiRSquared17 link https://www.mediawiki.org/w/index.php?diff=605134 edit summary: [18:27:54] Ἀνερρίφθω κύβος [18:44:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:47:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:07:34] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 227 processes [19:14:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:17:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:44:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:45:07] !htmllogs [19:45:08] experimental: http://bots.wmflabs.org/~wm-bot/html/%23wikimedia-labs [19:48:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:06:35] Would someone mind taking a look at https://labsconsole.wikimedia.org/wiki/Shell_Request/Mattflaschen ? [20:14:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:14:36] ^ andrewbogott [20:14:48] * ori-l whistles innocently. [20:16:59] * ^demon whistles deviously [20:17:55] superm401: You'll need to set up a keypair here https://labsconsole.wikimedia.org/wiki/Special:NovaKey and then you should be all set. [20:18:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:19:47] ^demon did you get adminbot into git, or did that turn out to be hard and/or annoying? [20:20:01] <^demon> I just hadn't finished it yet. [20:20:11] <^demon> Roan pulled the live copy for me Friday. [20:20:49] ok [20:26:27] andrewbogott, thanks. [20:26:49] working? [20:27:53] I didn't test yet. I actually already had a key. [20:28:07] Is there a way to test without a project, or is the next step to get added? [20:29:44] Is it possible to create a labs project with multiple (at least 3) subdomains? [20:44:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:48:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:54:02] anyone know why mysql doesn't work on bots.wmflabs.org? [20:54:19] or better yet, how to fix it? [20:56:38] That might be a petan or Damianz question [21:05:26] 11/13/2012 - 21:05:25 - Creating a home directory for olenz at /export/keys/olenz [21:10:20] 11/13/2012 - 21:10:19 - Updating keys for olenz at /export/keys/olenz [21:14:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:19:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:27:13] petan, Damianz: ^ [21:44:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:49:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:14:14] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:19:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:21:59] andrewbogott: can i request another public ip for editor-engagement? (it has quite a few already, but it's also a "mega-project" that lumps several WMF teams, all working on front-endy things that require easy sharing of URLs, and we're just adding a new person.) [22:22:23] ori-l: Sure. Give me a few... [22:22:28] thanks! [22:27:13] ori-l: ok, done. [22:32:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 134 processes [22:44:37] andrewbogott: thanks! [22:45:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:50:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:16:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:20:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:33:32] PROBLEM Total processes is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: PROCS WARNING: 151 processes [23:46:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:50:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%