[00:05:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:09:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:11:27] hey preilly, can you explain something pretty basic to me about your instructions at https://www.mediawiki.org/wiki/Git/Tutorial#Set_Up_SSH_Keys_in_Gerrit [00:29:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:36:00] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:39:25] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [00:59:24] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:06:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:08:54] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 174 processes [01:09:24] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:10:27] BewareofDoug: hmm, why not just ask the question to everybody here? [01:10:57] suppose I could :-) [01:11:54] The instructions say to generate a key which is then in the file /home/preilly/.ssh/id_rsa.pub in the example [01:12:24] then it says to open the file in a text editor and then the next instruction is to run cat /home/preilly/.ssh/id_rsa.pub [01:12:45] that doesn't make sense to me [01:13:34] what am I doing? it looks like I'm adding the contents of id_rsa.pub to the file id_rsa.pub [01:13:52] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:15:38] don't I need to copy my key to my account on the server rather than copying it to my own machine? [01:19:11] hmm. yeah, it's weird. no idea why it recommends using cat there. [01:19:48] I'd drop that step. At least I dropped running cat when I added by SSH key a few days ago :) [01:19:54] s/by/my [01:21:08] OK, thanks! yeah there are several things there that could probably be fixed and maybe I will, later, after I'm up and running ;-) [01:31:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:36:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:39:32] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [01:40:51] 7 [01:51:57] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [02:01:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:06:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:09:35] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:15:07] andre__: or anybody, when I do 'git review -s' it returns [02:15:09] No '.gitreview' file found in this repository. [02:15:10] We don't know where your gerrit is. Please manually create [02:15:10] a remote named gerrit and try again. [02:18:12] Never mind [02:18:24] operator head space and timing error [02:21:52] hmm, but: Exception: Could not connect to gerrit at ssh://Doug@gerrit.wikimedia.org:29418/test/mediawiki/extensions/examples.git [02:31:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:37:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:39:33] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [02:41:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [02:46:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:00:21] 11/05/2012 - 03:00:21 - Creating a home directory for doug at /export/keys/doug [03:01:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:03:33] aha [03:03:42] the instructions are missing a whole bunch of steps [03:03:52] you have to go to labs and follow the instructions there [03:05:22] 11/05/2012 - 03:05:22 - Updating keys for doug at /export/keys/doug [03:08:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:09:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [03:31:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:38:26] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:39:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:02:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:09:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:09:42] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [04:32:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:39:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:39:45] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:02:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:09:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:11:22] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [05:32:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:39:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:42:02] PROBLEM host: i-000003ef.pmtpa.wmflabs is DOWN address: i-000003ef.pmtpa.wmflabs CRITICAL - Host Unreachable (i-000003ef.pmtpa.wmflabs) [06:00:52] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Connection refused by host [06:01:34] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Connection refused by host [06:02:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:02:12] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Connection refused by host [06:02:22] PROBLEM Total processes is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Connection refused by host [06:02:52] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Connection refused by host [06:03:04] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Connection refused by host [06:10:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:10:52] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 0.40, 0.81, 0.57 [06:11:35] RECOVERY Current Users is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [06:12:12] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [06:12:22] RECOVERY Total processes is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: PROCS OK: 103 processes [06:12:52] RECOVERY dpkg-check is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: All packages OK [06:13:02] RECOVERY Free ram is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK: 3021% free memory [06:31:23] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:32:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:41:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:51:25] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [07:02:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:11:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:32:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:41:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:02:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:12:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:32:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:42:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:03:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:13:25] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:33:46] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:38:38] Change on 12mediawiki a page Developer access was modified, changed by Olenz link https://www.mediawiki.org/w/index.php?diff=601229 edit summary: [09:43:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:48:55] PROBLEM Current Load is now: WARNING on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: WARNING - load average: 10.54, 8.27, 6.33 [09:50:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 7.53, 7.95, 6.21 [10:00:52] Change on 12mediawiki a page Developer access was modified, changed by Bhargavgolla link https://www.mediawiki.org/w/index.php?diff=601233 edit summary: [10:00:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 9.08, 8.58, 5.99 [10:03:46] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:03:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.53, 7.82, 6.23 [10:05:32] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 6.34, 7.59, 6.12 [10:05:52] PROBLEM Free ram is now: CRITICAL on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: Connection refused by host [10:05:52] PROBLEM Current Load is now: CRITICAL on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: Connection refused by host [10:06:32] PROBLEM Current Users is now: CRITICAL on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: Connection refused by host [10:07:12] PROBLEM Disk Space is now: CRITICAL on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: Connection refused by host [10:07:26] PROBLEM Total processes is now: CRITICAL on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: Connection refused by host [10:08:12] PROBLEM dpkg-check is now: CRITICAL on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: Connection refused by host [10:13:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:15:57] RECOVERY Current Load is now: OK on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: OK - load average: 1.42, 1.27, 0.78 [10:15:57] RECOVERY Free ram is now: OK on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: OK: 267% free memory [10:16:34] RECOVERY Current Users is now: OK on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [10:17:14] RECOVERY Disk Space is now: OK on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: DISK OK [10:17:24] RECOVERY Total processes is now: OK on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: PROCS OK: 84 processes [10:17:54] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 5.62, 6.11, 5.48 [10:18:08] RECOVERY dpkg-check is now: OK on stackfarm-sql2 i-00000504.pmtpa.wmflabs output: All packages OK [10:33:46] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:43:27] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:56:53] PROBLEM Free ram is now: UNKNOWN on dumps-bot1 i-000003ed.pmtpa.wmflabs output: NRPE: Call to fork() failed [11:00:32] PROBLEM Disk Space is now: UNKNOWN on dumps-bot1 i-000003ed.pmtpa.wmflabs output: NRPE: Call to fork() failed [11:01:42] PROBLEM dpkg-check is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:01:42] PROBLEM Current Users is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:01:52] PROBLEM Total processes is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:01:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:03:52] PROBLEM Current Load is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:04:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:05:32] PROBLEM Disk Space is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:05:56] PROBLEM SSH is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Server answer: [11:10:32] RECOVERY Disk Space is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: DISK OK [11:10:52] RECOVERY SSH is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [11:12:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 2.99, 3.45, 4.82 [11:13:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:16:42] RECOVERY dpkg-check is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: All packages OK [11:16:42] RECOVERY Current Users is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [11:16:55] RECOVERY Total processes is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: PROCS OK: 122 processes [11:16:55] RECOVERY Free ram is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK: 1590% free memory [11:18:53] RECOVERY Current Load is now: OK on dumps-bot1 i-000003ed.pmtpa.wmflabs output: OK - load average: 0.10, 0.18, 0.10 [11:23:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.00, 3.15, 4.50 [11:24:31] Change on 12mediawiki a page Developer access was modified, changed by Praneesh link https://www.mediawiki.org/w/index.php?diff=601244 edit summary: [11:30:33] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 2.58, 2.78, 4.34 [11:30:53] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 1.16, 2.52, 4.93 [11:34:15] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:40:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.11, 4.25, 5.00 [11:44:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:53:53] RECOVERY Current Load is now: OK on parsoid-roundtrip5-8core i-000004db.pmtpa.wmflabs output: OK - load average: 4.04, 4.13, 4.85 [12:04:15] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:14:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:34:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:45:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:04:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:05:34] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [13:06:04] PROBLEM Free ram is now: CRITICAL on spellcheckself-bot i-0000041e.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [13:10:34] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [13:10:52] RECOVERY Free ram is now: OK on spellcheckself-bot i-0000041e.pmtpa.wmflabs output: OK: 45% free memory [13:15:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:24:01] petan: since I know you're around, any chance you could look at the .htaccess file I put in /data/project/public_html/legoktm/cgi-bin? For some reason I'm getting 403 errors on my python scripts now [13:24:02] like http://bots.wmflabs.org/~legoktm/cgi-bin/test.py [13:35:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:44:10] legoktm hi [13:44:25] hi [13:44:32] aha [13:44:46] ok, problem is that we reinstalled web server last week [13:44:53] probably missing some stuff [13:45:08] ok [13:45:28] because i remember someone had set up .py to globally be cgi if in the cgi-bin, so i wasnt sure why it wasnt working [13:45:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:45:58] do you think its an easy fix? or will it require some time? [13:46:09] I will fix it within few minutes I hope [13:47:42] thanks :D [13:54:26] fixed [13:54:34] !log bots enabled +ExecCGI on apache for userdir's [13:54:36] Logged the message, Master [13:57:51] thanks! [14:00:50] petan: do i need to separately install packages on the webserver if i want to use them via cgi? [14:01:10] which packages? [14:01:21] python-pyexiv2 [14:01:33] I have it installed on bots-3 right now [14:02:34] and this python module that's not available as a package: http://pypi.python.org/pypi/requests [14:06:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:13:33] petan: would you be able to install those? [14:14:32] RECOVERY dpkg-check is now: OK on centralauth-mysql i-00000497.pmtpa.wmflabs output: All packages OK [14:14:40] sure [14:15:47] thanks [14:16:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:19:51] !log bots petrb: sudo pip install requests [14:19:53] Logged the message, Master [14:20:09] legoktm done [14:20:09] thanks petan :) [14:36:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:46:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:58:25] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 151 processes [15:04:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [15:06:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:08:27] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [15:16:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:24:32] RECOVERY Total processes is now: OK on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS OK: 107 processes [15:36:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:47:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:59:15] !ping [15:59:22] pong [16:06:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:17:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:25:51] !ping [16:25:52] pong [16:36:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:48:27] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:55:35] Change on 12mediawiki a page Developer access was modified, changed by Kaoskeya link https://www.mediawiki.org/w/index.php?diff=601314 edit summary: [17:06:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:19:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:36:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:48:57] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:00:20] Change on 12mediawiki a page Developer access was modified, changed by Vdanilchenko link https://www.mediawiki.org/w/index.php?diff=601330 edit summary: [18:01:12] Change on 12mediawiki a page Developer access was modified, changed by Vdanilchenko link https://www.mediawiki.org/w/index.php?diff=601331 edit summary: /* User:Vdanilchenko */ [18:07:05] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:19:06] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:25:33] hello [18:25:42] we're experiencing some problems with build1.pmtpa.wmflabs [18:25:52] could you please offer some assistance to this end ? [18:26:06] tl;dr => can't connect through ssh, used to work fine [18:27:37] this also applies to drdee and ottomata [18:27:45] public key is denied [18:37:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:41:02] maybe there's a nagios available somewhere which can help us look closer at build1 [18:49:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:02:41] build1 is now down, doesn't answer to ping [19:03:17] oh wait it's up [19:03:24] still can't ssh [19:04:30] sumanah: hello [19:07:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:07:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [19:08:11] hello paravoid [19:08:21] drdee: sumanah indicated that paravoid can help us with build1 [19:08:39] http://lists.wikimedia.org/pipermail/labs-l/2012-October/000492.html - Faidon is doing Labs work while Ryan is away :) [19:09:07] andrewbogott too :) [19:09:08] so, what's your problem? [19:09:13] sumanah: ok, but we need build1 for building packages and such [19:10:03] so, could you please tell when it's going to be available again or maybe we can use something different in the meantime ? [19:10:12] like some other machine [19:10:31] * sumanah defers to paravoid & andrewbogott_afk [19:11:21] I can try to fix your issue [19:11:24] if you tell me what that is :-) [19:11:41] paravoid: public key is rejected [19:11:49] for me, ottomata, average_drifter [19:11:53] machine used to work fine [19:12:41] ok, try again [19:14:40] thanks paravoid! it's working for me again [19:14:48] average_drifter ^^ [19:16:26] thanks paravoid [19:19:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:24:58] paravoid: What did you do just then? [19:27:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [19:32:39] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [19:37:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:46:04] andrewbogott: restarted nscd/nscld [19:49:04] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:08:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:16:04] !log deployment-prep rebooting deployment-dbump (stalled git processes) [20:16:11] Logged the message, Master [20:19:47] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:36:59] !log deployment-prep uninstalling beta updater from jobrunner06 , will be deployed on -bastion [20:37:01] Logged the message, Master [20:38:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:45:14] PROBLEM dpkg-check is now: CRITICAL on deployment-jobrunner06 i-0000031d.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:50:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:52:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [20:56:56] !log deployment-prep rebooting jobrunner06 to ensure that wmf-beta-autoupdater is gone [20:56:56] Logged the message, Master [21:00:45] !log deployment-prep changing ownership of all files in /home/wikipedia/common to mwdeploy:mwdeploy as per deployment-bastion GID/UID. Running as root in screen 27986.pts-1.i-00000390 . [21:00:47] Logged the message, Master [21:04:18] [bz] (8RESOLVED - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 41311] mwdeploy unix user should be in svn group - https://bugzilla.wikimedia.org/show_bug.cgi?id=41311 [21:04:19] [bz] (8NEW - created by: 2Antoine "hashar" Musso, priority: 4Unprioritized - 6normal) [Bug 40983] beta autoupdater does not work anymore - https://bugzilla.wikimedia.org/show_bug.cgi?id=40983 [21:04:33] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [21:08:46] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:12:33] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes [21:21:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:23:14] [bz] (8NEW - created by: 2spage, priority: 4Unprioritized - 6enhancement) [Bug 41797] role::mediawiki-install::labs can't run phpunit tests - https://bugzilla.wikimedia.org/show_bug.cgi?id=41797 [21:31:12] RECOVERY dpkg-check is now: OK on deployment-apache33 i-0000031b.pmtpa.wmflabs output: All packages OK [21:35:23] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [21:37:36] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [21:38:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:46:12] RECOVERY dpkg-check is now: OK on deployment-apache32 i-0000031a.pmtpa.wmflabs output: All packages OK [21:51:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:02:32] PROBLEM Total processes is now: CRITICAL on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS CRITICAL: 293 processes [22:05:27] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [22:08:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:21:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:27:33] PROBLEM dpkg-check is now: CRITICAL on deployment-video05 i-000004cc.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [22:35:42] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [22:39:04] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [22:42:32] PROBLEM Total processes is now: WARNING on wikistats-01 i-00000042.pmtpa.wmflabs output: PROCS WARNING: 200 processes [22:51:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:52:32] RECOVERY dpkg-check is now: OK on deployment-video05 i-000004cc.pmtpa.wmflabs output: All packages OK [23:05:42] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [23:09:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:11:03] !log deployment-prep applied the new role::beta::autoupdater class to -bastion. [23:11:05] Logged the message, Master [23:21:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:23:51] * Damianz tickles hashy [23:35:46] PROBLEM host: i-0000031d.pmtpa.wmflabs is DOWN address: i-0000031d.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000031d.pmtpa.wmflabs) [23:39:13] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [23:51:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%