[00:23:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:26:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:53:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:56:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:04:53] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:09:52] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:23:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:26:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:56:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:24:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:27:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:54:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:57:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:24:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:27:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:54:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:57:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:57:02] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [04:24:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:27:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:55:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:57:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:25:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:28:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:56:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:58:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:26:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:32:22] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:42:22] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:56:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:58:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:26:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:28:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:57:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:59:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:28:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:29:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:58:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:01:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:28:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:31:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:58:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:00:39] !ping [10:00:39] pong [10:00:45] @infobot-detail ping [10:00:45] Info for ping: this key was created at N/A by N/A, this key was displayed 63 time(s), last time at 11/12/2012 10:00:39 AM (00:00:05.6897450 ago) [10:01:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:29:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:32:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:59:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:02:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:33:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:49:02] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 7.63, 7.48, 6.09 [11:59:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:03:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:08:52] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:09:02] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.82, 4.61, 4.96 [12:09:32] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:10:13] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:10:33] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.33, 5.81, 5.14 [12:10:53] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:11:23] PROBLEM Total processes is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:12:53] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CRITICAL - Socket timeout after 10 seconds [12:25:32] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.35, 4.68, 4.95 [12:30:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:33:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:01:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:03:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:05:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 227 processes [13:10:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 134 processes [13:31:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:34:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:01:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:04:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:12:53] PROBLEM Current Load is now: WARNING on dumps-bot2 i-000003f4.pmtpa.wmflabs output: WARNING - load average: 5.00, 5.01, 5.01 [14:27:32] RECOVERY Current Users is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [14:27:32] RECOVERY Total processes is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: PROCS OK: 224 processes [14:28:22] RECOVERY Current Load is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: OK - load average: 0.03, 0.29, 0.49 [14:28:52] RECOVERY Disk Space is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: DISK OK [14:28:52] RECOVERY SSH is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:29:42] PROBLEM Free ram is now: WARNING on aggregator2 i-000002c0.pmtpa.wmflabs output: Warning: 8% free memory [14:29:52] RECOVERY dpkg-check is now: OK on aggregator2 i-000002c0.pmtpa.wmflabs output: All packages OK [14:31:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:34:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:01:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:04:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:04:42] PROBLEM Free ram is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: NRPE: Unable to read output [15:05:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 227 processes [15:06:53] PROBLEM Disk Space is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:07:53] PROBLEM dpkg-check is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:10:32] PROBLEM Current Users is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:10:32] PROBLEM Total processes is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:11:22] PROBLEM Current Load is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [15:11:52] PROBLEM SSH is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: Server answer: [15:19:10] Today is a WMF holiday, so there won't be much staff around IRC. [15:19:33] I'll try to check in periodically -- y'all should feel free to page me but I may be less responsive than usual [15:19:39] (which is already not all that responsive :) ) [15:30:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 134 processes [15:31:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:36:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:40:22] 11/12/2012 - 15:40:22 - Updating keys for emw at /export/keys/emw [16:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:06:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:20:55] i'm returning to work on a labs project after a hiatus, and notice that it's giving an odd error message. http://pdbhandler.wmflabs.org says "Can't contact the database server: Unknown database 'my_wiki' (localhost)". the 'my_wiki' database on my instance (pdbhandler-1 / I-0000030e) seems to be gone, even though i haven't modified the code since it was working in july. [16:22:11] has this kind of issue popped up elsewhere? can i recover that database? [16:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:36:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:45:49] Emw: Are you using the single-node mediawiki puppet class? [16:46:14] A while ago we changed where it looks for its db by default, so maybe the db is still there but not where mediawiki is looking... [16:51:05] andrewbogott: i don't know -- how would i find out? for the "Puppet class" field i see "base, ldap::client::wmf-test-cluster, exim::simple-mail-sender, sudo::labs_project, role::mediawiki-install::labs" in the relevant instance (https://labsconsole.wikimedia.org/wiki/Nova_Resource:I-0000030e) [16:52:05] Emw: Yep, that's the one. [16:52:13] Can you look in /var/mysql and see if your db is there? [16:54:11] andrewbogott: No luck, /var/mysql doesn't exist on the instance [16:54:21] Ok, just a moment... [16:55:06] ok, my mistake, it would be /var/lib/mysql [16:55:59] It should be there or in /mnt/mysql. I predict that it's in the former place though. [16:56:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [16:57:50] Woo! 'mywiki' is in '/var/lib/mysql'. What should I do now? [16:58:02] my_wiki* [16:58:34] good news, we can probably fix this! [16:58:49] Are you mostly familiar with the puppet configuration web interface? Or mostly not? [16:59:46] mostly not [16:59:55] What you want is to define the mysql_datadir variable to point at where your db is. [17:00:56] You may or may not already have an existing field to set that value. Look at the 'configure' page for your instance... [17:01:02] do you see a field with that name? [17:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:02:07] It would be next to the 'role::mediawiki-install::labs' entry (that is, presumably, checked.) [17:03:12] Sorry, where would I find the 'configure' page for the instance? [17:03:33] Start here: https://labsconsole.wikimedia.org/wiki/Special:NovaInstance [17:03:51] There's a set of links on the far right, the same set for each instance [17:03:59] (which, for you that page maybe only shows the one instance :) ) [17:04:14] The 'configure' link takes you to the puppet config page. [17:04:34] But the set of available puppet config options varies per project, so I don't know quite what it will look like for you. [17:06:25] Alright, I don't see a "mysql_datadir" on the configure page, but I do see a "mysql::datadirs" field with a checkbox [17:06:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:07:32] Huh, I wonder what that is? Give me a moment... [17:07:56] OK, I declare that to be a meaningless distraction. [17:08:34] Anyway… lacking that field means there's an extra level of complexity but I think we'll survive. Go to this page: https://labsconsole.wikimedia.org/wiki/Special:NovaPuppetGroup [17:08:43] That's the place where you can manage /which/ fields are available for a given project. [17:08:57] Under your project name there should be an 'add variable' link. See it? [17:10:18] Hmm, nope. I see 'pdbhandler [Toggle, Add group]', but the string 'add ' only appears for 'add group' on that page [17:10:58] Oh, OK. Add a group called 'mwreview' [17:11:07] Then, within that group, add the variable 'mysql_datadir'. [17:11:13] (The group name is arbitrary, the var name is not.) [17:11:53] OK, done [17:12:59] Now rewind a few steps… go back to the instance config page and set that variable to point to your db. [17:13:10] So, I think that'll just be /var/lib/mysql [17:13:26] (I should say -- don't point it to the db, point it to the place where mysql keeps the db.) [17:14:37] Alright, that's now done [17:16:05] OK! Now... [17:16:17] Well, things might already be working. Or we might need to restart Apache, I'm not sure. [17:16:26] What happens when you load your base wiki page? Still an error message? [17:16:45] Yup -- http://pdbhandler.wmflabs.org/wiki/Main_Page [17:16:52] I'll restart Apache. [17:20:31] No luck. I notice in mysql that the database 'my_wiki' still doesn't appear, even though a few databases that weren't there before are there now. [17:21:23] Oh, y'know, I forgot a step -- we have to do a puppet refresh. [17:21:40] that's just 'puppetd -tv' as root. [17:25:57] I get an error message when running that as root: "err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class db::es:slave for i-0000030e.pmtpa.wmflabs on node i-0000030e.pmtpa.wmflabs; warning: Not using cache on failed catalog; err: Could not retrieve catalog; skipping run" [17:26:28] Looking in the 'Configure instance' page, though, I see 'db::es:slave' checked [17:26:46] Man, I have no idea what that is. [17:26:52] Did you set up this instance yourself, originally? [17:27:14] Mostly, I think [17:27:32] That was in June/early July [17:29:25] I'm digging, give me a minute [17:29:37] Thanks, I appreciate your help a ton [17:30:41] OK, I think I see what happened to that class. But unless you know why you need it I think we should try just unchecking it and see what happens. [17:31:29] btw -- depending on how attached you are to this particular system and its current database… we could also just trash it and start a fresh instance. Debugging is interesting but maybe unnecessary depending on your needs :) [17:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:32:46] 'es' stands for 'external storage.' But it otherwise sounds like you were just using a local db file… are you expecting the db contents to be super huge? [17:33:28] Nope, I'm not expecting the DB contents to be large [17:33:50] OK. So, try unchecking and refreshing puppet. [17:36:33] Fantastic, that worked. http://pdbhandler.wmflabs.org/wiki/Main_Page is back up and working :) [17:36:50] Thank you! [17:36:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:37:04] That's great! [17:37:34] So… I think you're good to go. [17:38:01] New instances keep their dbs in /srv because of filesystem issues. But as long as you're not worried about running out of space I think you're god. [17:38:02] good. [17:40:35] Alright, I'll (try to) keep that in mind [17:41:22] I don't actually remember what the difference is between /srv and /var [17:42:03] I thought they were different mount points but it's not obvious that that's right on your instance. [17:42:33] so maybe it's just an organizational thing. *shrug* [17:46:42] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: Critical: 5% free memory [17:56:43] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 84% free memory [18:02:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:06:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:33:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:36:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:04:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:05:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 227 processes [19:06:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:34:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:36:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:05:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:06:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:36:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:37:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:06:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:07:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:35:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [21:36:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:37:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:43:02] hello [21:43:05] spetrea@build1:~$ cat /etc/hostname [21:43:06] i-000002b3 [21:43:16] is it ok if I replace that to build1 ? [21:43:30] is someone depending on that hostname "i-000002b3" ? [21:45:52] average_drifter: Maybe, maybe not. But whatever the reason, don't do it. [21:46:24] As you can see from the prompt prefix, the information is available already. [21:47:03] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 8.80, 7.48, 5.88 [21:48:12] average_drifter: There is env var $INSTANCENAME (accompanied by $HOSTNAME), and local file /etc/wmflabs-instancename (accompanied by /etc/hostname) [21:50:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [21:53:53] Krinkle: oh cool ! I can use that then ! [21:54:40] average_drifter: That's also what the $ prefix uses in ("spetrea@build1:~$"), see your ~/.bashrc [21:54:43] you're welcome [22:06:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:07:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:27:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.07, 5.74, 5.19 [22:28:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.16, 5.94, 5.29 [22:36:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:37:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:42:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.24, 4.80, 4.93 [22:43:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.76, 4.11, 4.75 [22:45:33] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 134 processes [22:48:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 211 processes [22:50:59] anybody around? i am trying to assign a public ip to the vm 'master' in project 'hadoop' but I get the message 'Failed to allocate new public IP address." any suggestions? [22:52:53] andrewbogott, can you maybe help ^^ [22:53:19] Lemme look. [22:53:27] Do you know whether or not you're over quota? [22:53:33] no idea :) [22:53:43] Oh, well, then probably you're over quota :) [22:53:51] didn't know there was such a thing [22:54:04] Yeah, public IPs are precious. [22:54:17] Do you know, is that project already using one? [22:54:29] no, afaik, it is not using one yet [22:54:46] Will you be having lots of people besides you access the server? [22:54:52] * drdee is hearing 'precious…..my precious' in the background [22:55:03] uhhh no, two folks in total [22:55:24] So in that case maybe you can just use a proxy. There's a page with a howto, lemme find it. [22:55:36] http://cdn.memegenerator.net/instances/400x/28693067.jpg < totally andrewbogott [22:55:54] My hair is thinner [22:56:08] That's true, but you have a hat [22:56:37] drdee, this is a bit intimidating, but give this a try: https://labsconsole.wikimedia.org/wiki/Help:Access#Accessing_web_services_using_a_SOCKS_proxy [22:58:36] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 190 processes [22:58:44] so i am working with louisdang on this but he hasn't been successful yet [22:59:07] louisdang can you explain to andrewbogott what is causing issues when setting up the SOCKS proxy? [22:59:55] one sec [23:00:16] apply more magic [23:00:41] first, when I tried port forwarding I get messages like: channel 4: open failed: connect failed: Connection timed out in bastion [23:01:27] Are y'all on windows? [23:01:34] I'm on ubuntu [23:01:56] Oh, well, that should be easy then... [23:02:20] You get that message when you do 'ssh @bastion.wmflabs.org -D 8080' on the commandline? [23:03:27] when using ssh '@bastion.wmflabs.org -L 8080::80' [23:04:37] I don't know that that's necessarily wrong, but… why varying from the advice on the page above? [23:04:57] it's on that page in the section above [23:05:03] any reason you're not using socks over direct ports? [23:05:15] couldn't get the SOCKS to work with my proxy switcher [23:05:26] Ah, so I see. [23:05:46] let me try another browser (firefox) first [23:06:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:06:19] I'm using SOCKS and foxyproxy and it works fine. I'm on a mac, but it should be the same... [23:06:21] 'should' [23:07:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:09:26] andrewbogott, could you get the url pattern file at: https://labsconsole.wikimedia.org/wiki/MediaWiki:Foxy-proxy-labs?action=raw to work? [23:11:27] I just entered it by hand... [23:11:55] yeah me too [23:12:24] just wondering, since I couldn't get it to work first. [23:12:49] so on firefox it looks like it attempts to connect but fails [23:12:52] louisdang try disabling foxy-proxy and just enter the proxy in Firefox -> Preferences -> Advanced -> Network tab::Connection -> Settings [23:12:58] ok [23:13:19] I was actually trying to find that menu :) [23:15:05] I still get the same kind of errors using both methods [23:15:07] channel 4: open failed: connect failed: Connection timed out [23:15:28] But you can ssh to bastion otherwise? [23:16:05] yes [23:16:43] Hm. [23:16:45] No idea what that's about. [23:16:54] You two have suffered enough, I allocated you a public IP :) [23:17:16] Please be sure to tell me or Ryan to return it to the pool if/when you finish up. [23:17:25] thank you very much andrewbogott [23:17:48] thx! [23:18:00] drdee: Hopefully the web interface will be more polite to you now. [23:19:57] ty andrewbogott! [23:20:08] louisdang: let me know if you run into new problems [23:20:18] drdee, ok [23:36:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:37:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs)