[00:05:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:05:27] 11/23/2012 - 00:05:26 - Updating keys for odie5533 at /export/keys/odie5533 [00:28:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:35:13] Change on 12mediawiki a page Developer access was modified, changed by Dimension link https://www.mediawiki.org/w/index.php?diff=608623 edit summary: [00:35:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:54:04] Change on 12mediawiki a page Developer access was modified, changed by Dimension link https://www.mediawiki.org/w/index.php?diff=608625 edit summary: /* User:Dimension */ [00:59:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:05:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:07:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:12:33] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 98 processes [01:29:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:36:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:59:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:03:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 19% free memory [02:06:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:07:51] On beta.wmflabs, I seem to be logged in and out randomly [02:09:13] Can someone solve this? [02:29:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:36:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:38:43] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 21% free memory [02:38:53] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 20% free memory [02:46:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 19% free memory [02:58:06] odie5533: yes [02:59:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:01:42] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory [03:04:13] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=608639 edit summary: /* User:Dimension */ done [03:04:22] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=608640 edit summary: /* User:Orsagi */ done [03:05:09] Krinkle: for a while it wasn't letting you log in at all? [03:05:29] jeremyb: ? [03:05:35] jira [03:05:42] oh, gah, wrong channel [03:06:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:29:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:36:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:37:10] I am unable to properly connect to instances. [03:37:44] I can connect to bastion, and it appears to connect to instances, but then the connection instantly closely. Using ssh -v, it says Exit status 254 [03:37:56] *instantly closes [03:38:46] Perhaps someone could connect to the instance "webtools-odie" and check the sshd logs? [04:01:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:07:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:31:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:01:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:08:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:27:53] PROBLEM Free ram is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Critical: 5% free memory [05:31:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:39:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:01:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:09:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:30:32] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:30:32] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:31:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:38:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [06:39:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:40:32] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 148 processes [06:50:24] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:53:34] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [07:03:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:09:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:33:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:39:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:44:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 8.46, 7.75, 5.96 [07:59:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 3.03, 3.68, 4.74 [08:03:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:08:01] Change on 12mediawiki a page Developer access was modified, changed by Tomertuc link https://www.mediawiki.org/w/index.php?diff=608715 edit summary: [08:10:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:22:56] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=608718 edit summary: /* User:Tomertuc */ done [08:33:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:35:27] 11/23/2012 - 08:35:27 - Updating keys for kontokostas at /export/keys/kontokostas [08:41:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:03:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:11:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:33:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:41:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:02:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [10:03:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:12:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:17:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [10:34:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:47:52] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [11:04:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:13:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:34:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:43:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:53:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [11:58:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [12:04:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:14:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:32:44] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 4% free memory [12:34:14] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:39:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 10% free memory [12:44:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:04:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:14:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:34:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:44:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:59:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 38% free memory [14:04:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:14:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:18:15] If anyone has a moment, I am unable to connect to instances once connected to bastion. [14:22:32] hi [14:22:38] !access | odie5533 [14:22:39] odie5533: https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [14:22:52] Hi petan! [14:22:54] you need to set up aah key [14:22:56] ssh [14:23:00] I did [14:23:04] I can connect to Bastion [14:23:04] twice [14:23:11] oh? [14:23:14] one for bastion, second for instances [14:23:22] if you don't want to store private key on bastion [14:23:26] Is it two different keys? [14:23:31] should be [14:23:38] for your own security yes [14:25:11] petan: Do I just add the second key to Nova? [14:26:00] nevermind, reading [14:26:28] !sshkey is To create a ssh key, type ssh-keygen on a machine you want to connect from, then execute $cat .ssh/id_*.pub >> .ssh/authorized_keys then scp .ssh/authorized_keys :temp_keys ssh to target machine and cat temp_keys >> .ssh/authorized_keys since now you are able to connect from source to target :) [14:26:28] Key was added [14:26:35] odie5533 yes [14:26:39] you have to [14:27:09] petan: Well, I can't connect by forwarding just my private key. Isn't that bad? [14:27:44] at some point it is, but mostly users are not able to forward their keys correctly [14:27:58] so I guess that's where the problem is [14:28:01] petan: It lets me connect for one second [14:28:08] but then disconnects me with exit status 254 [14:28:12] hmm [14:28:26] then there is a problem with forwarding I guess, what's the box u ssh to [14:28:35] webtools-odie [14:28:38] but also bots-1 [14:28:46] can you try bots-4 [14:28:56] bots-1 is worst for testing of anything [14:29:04] it's written in docs I think [14:29:27] same problem [14:29:42] Can you check the sshd logs on bots-4? [14:32:08] sure [14:32:13] can you paste what you get in terminal [14:33:07] http://pastebin.com/cEZ82itP [14:33:20] I think this might be the problem: Unable to create and initialize directory '/home/odie5533'. [14:33:29] oh [14:33:31] funny [14:33:37] I see [14:33:49] mutante, paravoid the bot for home creation is dead [14:34:22] heh, why am I the only one that noticed? I guess no one else is logging in to instances for the first time [14:35:00] indeed [14:35:52] petan: Regarding the separate key for instances, where do I put the private key for that? on bastion? [14:36:38] @seenrx home [14:36:38] petan: Last time I saw Aaron|home they were quitting the network with reason: Ping timeout: 276 seconds at 11/23/2012 12:37:42 PM (01:58:56.5470950 ago) (multiple results were found: labs-home-wm, DQ|homework, Aaron|home_, labs-home-wn, labs-home-wm_ and 1 more results) [14:36:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:36:53] @seenrx labs.*home-wm [14:36:54] petan: Last time I saw labs-home-wm they were talking in the channel, they are still in the channel. It was in #wikimedia-labs at 11/23/2012 8:35:27 AM (06:01:26.4786190 ago) (multiple results were found: labs-home-wm_) [14:37:05] hmm the bot is here [14:37:26] did you try logging in to webtools-odie? [14:37:31] because that is a new instance [14:37:55] use -v [14:37:58] sec [14:38:02] @rss+ bz [14:38:03] There is no such item, if you want to define new item, please use 2 parameters [14:38:08] @rss+ bugzilla [14:38:08] This item was enabled now [14:38:20] ok bz ticket was logged I guess [14:38:28] oh [14:38:34] I am wondering how to create your home by hand [14:38:39] Do you know where the ticket is? [14:38:44] you get a same error in that other instance? [14:38:49] you probably just mkdir /home/odie5533 and chown it to me [14:38:50] https://bugzilla.wikimedia.org/show_bug.cgi?id=42387 [14:38:54] nah [14:39:03] home is automount dir [14:39:10] you can't create folders in there [14:39:16] of course my method would be too easy :) [14:39:28] I can of course temporarily change your home to something else on bots-4 [14:40:29] If you can, please do, then I can at least start working on testing things from the WMF Labs instances. [14:40:38] well, actually [14:40:46] I think I'll wait at this point [14:41:23] because I can test pretty much anything I need to from my home PC. So once the creation bot is working, I can start on the environment in webtools. And having an environment on bots won't help me with that [14:42:00] I am getting an error logging in to labs. It says I have cookies disabled. [14:43:19] odie5533 try logging to bots [14:43:30] I managed to create your home properly [14:43:37] yep, it's working [14:43:52] ok [14:44:20] there is a same problem on your instance but unfortunatelly I won't be able to ssh there either [14:44:35] ah, that makes sense [14:44:43] only way to fix it would be to mount your shared home dir from labs-nfs somewhere and insert home folder by hand [14:44:46] because it won't make the home dir for you either [14:44:59] or fix bot [14:45:09] fix bot is probably the best long term option [14:45:16] It was working before [14:45:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:45:29] that can be done by ryan [14:45:33] @seen Ryan_Lane [14:45:33] petan: Last time I saw Ryan_Lane they were quitting the network with reason: Read error: Connection reset by peer N/A at 11/21/2012 9:36:57 PM (1.17:08:36.1523390 ago) [14:45:51] almost 2 days... [14:45:56] he likely has vacation [14:46:05] He was flying somewhere. [14:46:09] hmm [14:46:16] I don't know if paravoid can fix that [14:46:41] Why is it a bot that does it instead of a puppet script or something? [14:47:21] I think it has something to do with ldap and puppet which doesn't work well together [14:47:29] there is some weird cache on puppet catalog or something [14:47:43] I know that for this reason we aren't able to use puppet for nagios [14:48:00] also, bot is faster than puppet [14:48:17] it run like once in hour [14:48:29] so you could have to wait hour for your home dir to be created [14:48:36] Instances take a long time to build here. EC2 is quite fast. Is the difference puppet? [14:48:55] I think so, the instances are created nearly immediately [14:48:59] !console [14:48:59] in case you want to see what is happening on terminal of your vm, check console output [14:49:06] if u check this you would see that [14:49:15] but then the puppet run takes a long time [14:49:22] ah [14:49:34] Eeven loading the console output took a long time while it was building [14:49:38] yup [14:49:48] that has something to do with nova extension for mediawiki [14:49:58] i guess [14:51:16] Thank you for helping me figure out what was wrong. I was going kinda crazy thinking I was the only one who somehow had a broken account that couldn't connect to anything [14:52:08] I don't know if we are still beta, but this happens a lot, random troubles :D [15:01:04] I think Gmail is still beta, so WMFLabs can be too ;) [15:02:52] petan: for the bot/webtools environment, should things be done with puppet or is that not as important for bots/webtools since they are never deployed elsewhere? [15:08:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:09:15] odie5533 IMHO it shouldn't be important, but the environment setup should be always done using puppet (libraries and stuff that needs to be set up with root account) [15:11:26] petan: Can you allocate volumes for projects? [15:11:40] eh [15:11:43] I don't think so [15:11:51] neither I know what you mean [15:12:07] https://labsconsole.wikimedia.org/wiki/Special:NovaVolume [15:13:11] Did you set up the bots and bots-apache? [15:15:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:28:11] yes [15:28:57] odie5533 I don't know how this volume thing works, I have access to it, but I don't know what it actually is for [15:29:12] there are no volumes in any project right now [15:29:23] at least not displayed in that table [15:38:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:45:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:57:44] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [16:08:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:16:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:17:52] PROBLEM Free ram is now: UNKNOWN on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Unknown [16:22:52] PROBLEM Free ram is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Critical: 3% free memory [16:38:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:46:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:02:42] PROBLEM Free ram is now: CRITICAL on bots-3 i-000000e5.pmtpa.wmflabs output: Critical: 1% free memory [17:07:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 92% free memory [17:08:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:16:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:38:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:46:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:08:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:16:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:38:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:46:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:08:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:16:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:39:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:46:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:09:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:17:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:38:07] Change on 12mediawiki a page Developer access was modified, changed by Williamjohndonahue link https://www.mediawiki.org/w/index.php?diff=608918 edit summary: [20:39:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:47:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:09:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:18:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:41:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:49:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:00:53] PROBLEM Current Users is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:00:53] PROBLEM Current Load is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:00:53] PROBLEM Disk Space is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:01:13] PROBLEM dpkg-check is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [22:01:33] PROBLEM SSH is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Server answer: [22:05:53] RECOVERY Current Users is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [22:05:53] RECOVERY Current Load is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK - load average: 0.31, 0.92, 0.59 [22:05:53] RECOVERY Disk Space is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: DISK OK [22:06:13] RECOVERY dpkg-check is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: All packages OK [22:06:33] RECOVERY SSH is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [22:13:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:19:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:43:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:49:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:13:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:19:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:41:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [23:43:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:46:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [23:49:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:56:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory