[00:01:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:04:16] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:16:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:31:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:34:18] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:47:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:01:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:04:14] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:08:54] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 172 processes [01:13:53] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 95 processes [01:17:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:31:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:34:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:48:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:01:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:04:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:10:22] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [02:18:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:31:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:34:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:39:42] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 22% free memory [02:48:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:50:32] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 18% free memory [02:52:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 17% free memory [02:58:22] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [03:01:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:04:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:18:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:31:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:33:56] PROBLEM Free ram is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Critical: 5% free memory [03:34:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:35:52] PROBLEM Free ram is now: CRITICAL on aggregator-test1 i-000002bf.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [03:38:42] PROBLEM Current Users is now: CRITICAL on aggregator-test1 i-000002bf.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [03:39:05] PROBLEM Current Load is now: CRITICAL on aggregator-test1 i-000002bf.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [03:39:42] PROBLEM SSH is now: CRITICAL on aggregator-test1 i-000002bf.pmtpa.wmflabs output: CRITICAL - Socket timeout after 10 seconds [03:41:02] PROBLEM Disk Space is now: CRITICAL on aggregator-test1 i-000002bf.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [03:42:02] PROBLEM dpkg-check is now: CRITICAL on aggregator-test1 i-000002bf.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [03:42:42] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [03:49:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:00:32] PROBLEM Free ram is now: CRITICAL on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Critical: 5% free memory [04:02:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:04:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:15:24] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 11% free memory [04:19:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:32:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:34:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:40:32] RECOVERY Free ram is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK: 28% free memory [04:49:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:02:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:04:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:11:44] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 17% free memory [05:19:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:32:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:34:34] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:50:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:02:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:04:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:08:23] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 147 processes [06:20:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:31:23] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:32:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:34:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:51:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:51:26] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 146 processes [07:02:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:04:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:21:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:32:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:34:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [07:43:34] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 13% free memory [07:51:16] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:03:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:04:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:21:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:24:27] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 9% free memory [08:30:21] 10/31/2012 - 08:30:21 - Updating keys for valhallasw at /export/keys/valhallasw [08:33:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:34:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [08:52:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:03:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:04:43] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:06:30] !beta running apt-get upgrade on -dbdump [09:06:31] !log deployment-prep running apt-get upgrade on -dbdump [09:06:38] Logged the message, Master [09:12:55] !ping [09:12:55] pong [09:22:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:33:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:37:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [09:41:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [09:52:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:03:55] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:07:03] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:14:42] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 19% free memory [10:23:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:25:22] 10/31/2012 - 10:25:22 - Updating keys for nasirkhan at /export/keys/nasirkhan [10:33:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:38:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [10:53:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:56:12] PROBLEM dpkg-check is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:56:55] PROBLEM Current Load is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:57:02] PROBLEM Disk Space is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:58:52] PROBLEM Total processes is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [10:59:42] PROBLEM Current Users is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:01:03] PROBLEM SSH is now: CRITICAL on dumps-bot2 i-000003f4.pmtpa.wmflabs output: Server answer: [11:03:54] RECOVERY Total processes is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: PROCS OK: 131 processes [11:03:54] RECOVERY Free ram is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK: 74% free memory [11:03:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:04:43] RECOVERY Current Users is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [11:06:02] RECOVERY SSH is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [11:06:12] RECOVERY dpkg-check is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: All packages OK [11:06:52] RECOVERY Current Load is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: OK - load average: 0.17, 0.81, 0.83 [11:07:02] RECOVERY Disk Space is now: OK on dumps-bot2 i-000003f4.pmtpa.wmflabs output: DISK OK [11:08:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:10:04] !log wikidata-dev wikidata-dev-2: Implemented new ChangesDatabase settings on test clients (en & he), but it doesn't work properly. We suppose this is due to old code (two weeks since last update) and will see if it works after today's demo time. [11:10:07] Logged the message, Master [11:23:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:24:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [11:32:43] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 15% free memory [11:33:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:38:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [11:45:57] !seen Ryan_Lane [11:51:53] Jan_Luca: probably still sleeping (he is in SF) and he is in holiday tonight [11:52:38] hashar: I only does not see him the last days so I wanted to check if he was there when I not [11:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:54:32] PROBLEM Current Users is now: CRITICAL on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: Connection refused by host [11:55:13] PROBLEM Disk Space is now: CRITICAL on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: Connection refused by host [11:55:34] @seen Ryan_Lane [11:55:34] petan: Ryan_Lane is in here, right now [11:55:42] Jan_Luca ^ [11:55:44] :P [11:55:53] PROBLEM Current Load is now: CRITICAL on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: Connection refused by host [11:56:06] PROBLEM Free ram is now: CRITICAL on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: Connection refused by host [11:56:21] petan: I used NickServ :P [11:56:56] or that :P [11:57:23] PROBLEM Total processes is now: CRITICAL on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: Connection refused by host [11:57:53] PROBLEM dpkg-check is now: CRITICAL on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: Connection refused by host [12:02:23] RECOVERY Total processes is now: OK on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: PROCS OK: 90 processes [12:02:53] RECOVERY dpkg-check is now: OK on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: All packages OK [12:03:56] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:04:33] RECOVERY Current Users is now: OK on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [12:05:12] RECOVERY Disk Space is now: OK on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: DISK OK [12:05:52] RECOVERY Current Load is now: OK on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: OK - load average: 0.21, 0.62, 0.56 [12:06:05] RECOVERY Free ram is now: OK on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: OK: 1871% free memory [12:08:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:08:41] hashar: Could you review this change: [12:08:44] !g 29975 [12:08:45] https://gerrit.wikimedia.org/r/#q,29975,n,z [12:09:11] Jan_Luca: you should talk about it with ops in #wikimedia-operations [12:09:18] Jan_Luca: Faidon commented about it : "I think all of these classes are an overkill. Can't we have a php::extension definition that installs the respective extension? Or have nothing at all, as I don't see how including php::foo is better than defining Package["php5-foo"] as it is." [12:09:25] Jan_Luca: he is known as paravoid on IRC :-] [12:11:32] hashar: I have upload a new version with less classes [12:11:58] because I have talked with Faidon about this change [12:12:06] (here in IRC) [12:12:54] hi [12:13:07] paravoid: we are talking about https://gerrit.wikimedia.org/r/#/c/29975/ [12:13:08] yeah I noticed [12:13:12] which brings some puppet classes like php:xxxx [12:13:21] I still wonder who's going to use that [12:13:40] you said at one point that it was a much needed abstraction :-] [12:13:41] paravoid: You are there (he seems to have a long day) [12:14:03] hashar: having a php module is a nice abstraction to have [12:14:07] ahhh [12:14:12] but not a class per extension ? ;-) [12:14:21] having all kinds of package collections with no clear intent is not imho [12:14:21] paravoid: I have reduced the number of classes [12:14:49] and created two big classes php::most_used and php::nearly_all [12:14:59] what's the target for these two classes? [12:15:05] where are they going to be referenced from [12:15:18] That should easy install a php enviroment on a instance [12:16:28] and the other classes that are in my change are for use in classes like webserver [12:16:39] do you have a set of classes that you're going to switch to include that? [12:16:43] or a number of VMs? [12:17:13] for example bot-instances for php-bots can use them [12:17:50] or instances with tools migrated from Toolserver that need many php features [12:18:29] and the php::most_used have some extensions MediaWiki uses (intl or gd) [12:19:37] so, I'd prefer having a base php class that installs a few things [12:19:43] and then having a php::module or php::extension or something [12:19:46] as a definition [12:20:08] so that you can say php::module { "intl": ensure => installed } [12:20:20] er, present even [12:21:43] and then have the bots class(es)/module(s) include the extensions they need [12:21:53] The problem is that you can select in labs only classes via the web interface [12:22:40] !g 27611 [12:22:41] https://gerrit.wikimedia.org/r/#q,27611,n,z [12:24:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:24:45] paravoid: You are right that many small classes are bad so I thought create two bigger ones would be nice [12:24:45] *nicer [12:25:54] PROBLEM dpkg-check is now: CRITICAL on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [12:26:11] paravoid: I'm away for ca. two hours now [12:26:36] When do you be there (in UTC please)? [12:27:33] I don't see what the labs web interface has to do with anything [12:28:02] we should create puppet classes for services and not have low-level classes be included from the labs web interface [12:28:17] I'll be around when you're going to be back [12:28:28] I live in Athens, which is EET/UTC+2 [12:29:39] paravoid: That's nice because I live in Germany :-) [12:29:45] :-) [12:29:57] I thought you live in USA [12:30:42] nope [12:31:04] although I frequently work at least part of the US working day so I can overlap with a few colleagues of mine [12:31:16] hashar: why "zuulwikimedia"? [12:32:42] paravoid: so zuul is from openstack which install the zuul software [12:32:50] zuulwikimedia provides the configuration [12:33:39] I don't see a zuul class [12:34:12] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:34:50] paravoid: I have split the change in two part. https://gerrit.wikimedia.org/r/#/c/25235/ provides a Zuul modules which is mostly copied from upstream OpenStack [12:35:13] ah! [12:35:23] paravoid: tweaked to use git::clone instead of the third party vcsrepo module [12:35:47] the module just install the zuul software and a very basic configuration [12:36:32] the second change https://gerrit.wikimedia.org/r/#/c/27611/ contains our own configuration / roles [12:37:08] I used zuulwikimedia for our zuul configuration [12:37:15] to differentiate it from the "zuul" module [12:37:41] hm [12:37:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 18% free memory [12:37:59] so you bootstrap a role using zuulwikimedia::instance [12:38:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [12:40:17] hm. [12:41:09] I'm wondering if all that belongs in the role class [12:42:13] paravoid: isn't configuration the point of role classes ? [12:43:15] that is more or less what is being done in the other role files [12:43:23] yeah... [12:43:32] aka you simply set the system_role, call a parameterized class with all the parameters for that role [12:43:41] it's just that this kind of doulbe abstraction annoys me :P [12:43:52] triple really [12:44:33] I also wanted to keep the zuul module as close as possible from the openstack one [12:44:54] and yeah, I have indeed abstracted out our specific configuration [12:44:55] yeah, I figured that [12:45:04] I think you're right [12:45:19] I find it easier to navigate around, probably easier to modify / get reviewed [12:45:50] 27 [12:45:51] 28 » » » # FIXME missing in Lucid :( [12:45:51] 29 » » » 'python-jenkins', [12:45:53] hm? [12:45:56] ohh [12:46:15] we no more care about that since gallium got upgraded to Precise :-] I can remove the comment [12:46:19] ok [12:46:31] do that and I'll merge it [12:46:33] \O/ [12:46:33] lgtm [12:46:52] oh and a last question [12:46:59] also the instance need a password to be filled in the private repo [12:47:01] how's integration/zuul-config going to get updated? [12:47:48] git pull on gallium? [12:48:14] heh [12:48:26] or will git::clone refresh it automatically ? [12:48:26] okay :) [12:49:09] it is a single file : https://gerrit.wikimedia.org/r/gitweb?p=integration/zuul-config.git;a=tree ;-] [12:49:10] it won't iirc [12:49:23] ah, we have ensure => latest [12:49:25] extracted it out of operations/puppet to avoid having to get ops review a dumb configuration file [12:49:25] for git::clone [12:49:37] good! [12:49:40] for all of us [12:52:35] paravoid: I removed the comments https://gerrit.wikimedia.org/r/#/c/25235/ [12:52:36] https://gerrit.wikimedia.org/r/#/c/25235/7/modules/zuul/manifests/init.pp,unified [12:53:23] merged [12:53:37] \O/ [12:53:45] then there is the configuration part which is https://gerrit.wikimedia.org/r/27611 [12:53:47] let's merge the other one now too [12:53:51] yeah [12:53:56] not excited with the zuulwikimedia class but oh well [12:53:57] also need a password to be filled in the password repository [12:54:05] we can give it a better name [12:54:10] like what? [12:54:19] wikimediazuul ? ;-]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]] [12:54:21] hahahahahaha [12:54:22] seriously, I don't know :/ [12:54:26] yeah nevermind [12:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:54:36] <^demon> wikizuulmedia? [12:54:52] zuulimedia! [12:55:00] merged [12:55:18] can you put the password up somewhere on fenari in a non-world-readable file? [12:55:28] or gallium [12:55:30] sure [12:55:33] let me find it out [12:55:35] the labs change is https://gerrit.wikimedia.org/r/#/c/24912/ [12:55:50] * hashar connects to Jenkins [12:56:10] how did you merge it yourself? [12:56:13] I'm confused [12:56:25] I don't mind, but I didn't know you could :) [12:56:31] I got push / merge right there since that is labs [12:56:33] (I think) [12:56:45] okay :) [12:56:52] sounds good [12:56:52] maybe that might let me get root access on lab instances [12:56:56] yes it can [12:57:13] nice [12:57:36] just put something that gives you root on passwords::root [12:57:38] or change the root password [12:57:39] :P [12:57:39] logging in gallium to put the pass [12:57:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 20% free memory [12:59:16] paravoid: the API key is in gallium.wikimedia.org:/home/hashar/zuul [13:02:17] !log integration trying out role::zuul::labs on integration-jenkins2 [13:02:19] Logged the message, Master [13:02:30] that must be working [13:03:28] does it? [13:04:25] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:04:25] forgot to add the jenkins class too [13:04:40] oops [13:04:54] it is required to get a "jenkins" system user [13:04:57] since zuul runs as jenkins [13:05:02] guess that should be a requirement in the module [13:05:14] yes [13:05:37] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [13:07:27] still installing [13:08:04] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:10:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [13:11:09] I am doomed [13:11:19] [git_clone_integration/zuul-config]/returns: change from notrun to 0 failed: git clone -b master https://gerrit.wikimedia.org/r/p/integration/zuul-config.git /etc/zuul/wikimedia returned 128 instead of one of [0] [13:11:48] will retry [13:22:44] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [13:24:48] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:28:34] GIT_SSH="" git fetch; echo $? [13:28:35] error: cannot run : No such file or directory [13:28:36] fatal: unable to fork [13:28:37] 128 [13:28:38] oh no [13:28:48] seems git::clone does not work whenever the ssh parameter is missing [13:28:55] seems to be set to the empty string [13:29:05] which might be true for puppet [13:34:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:39:13] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [13:49:07] Good morning hashar [13:49:32] anomie: hello :-) [13:50:00] debugging in puppet :-] [13:50:39] I took a look at puppet docs yesterday, after I finished putting the config change in Gerrit. [13:52:53] puppet is a nice tool [13:53:00] I guess most of us learned about it by hacking our files [13:53:44] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 19% free memory [13:54:38] I took a look in operations/puppet.git, but I haven't found the end of the thread yet to start unraveling the knot [13:55:31] <^demon> anomie: Well, the structure for production is (roughly): servers are defined in manifests/site.pp, which will include other classes based on what they're doing. [13:55:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:55:59] <^demon> The files/ directory is files that puppet is copying to hosts (like config, etc). templates/ is similar to files, but are erb templates you can perform logic in. [13:56:23] <^demon> The only major difference in labs is site.pp isn't used, and instead you use the configure UI in labs. [14:04:18] grbmbmbmbmb [14:04:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:04:25] I am lost with puppet [14:04:41] paravoid: I got a permission issue with git::clone. [14:05:02] paravoid: it seems the default value for $owner is 'root' which is in turn used as a user for the exec {} call [14:05:08] but logging show that $owner is set to 'nobody' [14:05:16] notice: /Stage[main]/Role::Zuul::Labs/Zuulwikimedia::Instance[zuul-labs]/Git::Clone[integration/zuul-config]/Notify[Running git clone with user nobody for /etc/zuul/wikimedia]/message: defined 'message' as 'Running git clone with user nobody for /etc/zuul/wikimedia' [14:05:16] ;( [14:07:47] ohhh [14:07:47] found the issue [14:07:47] took many git:clones :-] [14:09:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:11:45] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 17% free memory [14:17:13] jeremyb finally managed to remove all code from wmib core to separate plugins, so it should be possible to handle all stuff without rebooting bot, except for patching [14:17:32] I will update docs [14:21:58] gluster has horrible IO [14:22:25] I was wondering how these apaches started to be so slow [14:22:30] it's because of IO [14:22:32] anomie: sorry I am busy with some puppet stuff today ;-] [14:22:53] anomie: if you wanna play with puppet, you could create a labs instance and use the puppetmaster::self class https://labsconsole.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [14:23:07] hashar- No problem. What should we get started on now? [14:23:20] petan: cool [14:23:21] have you ever logged on a labs instance ? [14:23:28] hashar- I might do that later [14:23:35] sure ;-] [14:24:04] anomie: wanted to let you know that you can test/play with puppet on labs :-) [14:24:09] can cover that later on [14:24:11] hashar- I've SSHed into bastion and then into a few other hosts (like deployment-apache23), if that's what you're asking [14:24:20] \O/ [14:26:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:26:43] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 20% free memory [14:36:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:39:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [14:56:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:02:25] paravoid: I'm back, do you have time to discuss [15:04:33] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [15:06:32] PROBLEM Free ram is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: Warning: 18% free memory [15:06:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:09:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:21:32] RECOVERY Free ram is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK: 20% free memory [15:26:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:32:57] hashar- Still here? [15:33:21] yup [15:33:40] looks like anomie has some internet connectivity issues :/ [15:34:02] yeah :/ [15:35:04] * hashar nature's call [15:35:05] brb [15:36:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:39:23] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [15:46:17] Ok, let's see if I can stay online now after rebooting my cable modem... [15:46:23] :-/ [15:46:29] anomie: and I am about to leave to get my daughter [15:46:36] got 10 minutes ahead :/ [15:47:42] hashar- So anything you can point me at in 10 minutes to work on for a few hours here? [15:48:01] looking for a change [15:50:59] anomie: Dereckson wrote a test suite for InitialiseSettings.php [15:51:13] which would definitely help us ensuring changes made to the production/beta cluster are working as expected [15:51:14] anomie: https://gerrit.wikimedia.org/r/#/c/28627/ [15:51:21] though that is a work in progress [15:53:38] Ugh. Internet apparently not fixed. :( [15:53:52] anomie: definitely take sometime to call your cable operator [15:53:52] and get that fixed :-] [15:54:20] or maybe that is freenode that disconnect you ? [15:55:30] hashar: you might want to set up a different means of talking with Anomie that can deal better with disconnections.... [15:55:47] yeah maybe we should gtalk [15:55:50] or hangout [15:56:12] * anomie_ calls twcable [15:56:55] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:57:37] maybe the cable modem has some low session timeout [15:57:53] though IRC most usually has continuous traffic which should reset the timeout [15:58:00] but that might be something else entirely :( [15:59:59]