[00:02:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:02:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:16:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:25:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 4.26, 4.91, 5.02 [00:28:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [00:32:25] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:33:35] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:33:55] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 6.75, 7.06, 6.23 [00:35:54] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 3.02, 4.35, 4.87 [00:46:45] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.48, 3.99, 4.95 [00:46:55] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:03:04] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:03:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:07:33] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:09:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.97, 5.46, 5.19 [01:12:32] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 97 processes [01:15:42] RECOVERY Free ram is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: 3367292 [01:16:22] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 149 processes [01:16:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [01:17:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:17:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.57, 3.76, 4.72 [01:19:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 2.28, 3.80, 4.59 [01:23:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.28, 3.61, 4.55 [01:24:22] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 161 processes [01:28:43] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3404508 [01:29:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [01:31:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [01:33:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:34:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:46:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [01:47:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:03:52] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:04:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:09:22] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 150 processes [02:18:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:33:53] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:34:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:38:42] RECOVERY Free ram is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: 3287660 [02:40:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:45:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.66, 6.42, 5.54 [02:49:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:55:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.72, 4.04, 4.82 [03:01:42] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3393048 [03:04:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:04:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:08:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory [03:19:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:33:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes [03:34:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:34:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:38:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [03:42:53] Change on 12mediawiki a page Developer access was modified, changed by Himeshi link https://www.mediawiki.org/w/index.php?diff=610286 edit summary: [03:43:25] Change on 12mediawiki a page Developer access was modified, changed by Himeshi link https://www.mediawiki.org/w/index.php?diff=610287 edit summary: /* User:Himeshi */ [03:47:23] PROBLEM Total processes is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [03:49:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:50:10] i've got an extension that i've got working on labs with MW 1.19, and i'd like to iron out any issues with the extension on MW 1.21. i've got an instance with MW 1.21 set up. could i get another public IP allocated to my project so i can develop against the MW 1.21 instance, while also having the MW 1.19 instance with the working extension publicly accessible? [03:50:42] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:50:52] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:52:22] PROBLEM Total processes is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:53:52] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Server answer: [03:54:22] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:54:32] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [04:04:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:04:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:09:32] RECOVERY Current Users is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [04:10:43] RECOVERY dpkg-check is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: All packages OK [04:10:53] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 0.39, 0.60, 0.72 [04:12:53] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [04:13:53] RECOVERY SSH is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [04:14:23] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [04:19:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:22:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [04:31:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [04:34:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:36:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:42:53] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [04:47:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 5.17, 4.75, 4.92 [04:49:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:56:43] RECOVERY Free ram is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: 3370648 [04:56:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [05:04:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:06:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:16:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [05:19:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:27:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 4% free memory [05:34:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:36:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:49:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:04:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:06:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:19:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:29:13] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:34:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:36:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:38:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:41:43] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3376472 [06:49:22] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 148 processes [06:49:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:51:43] RECOVERY Free ram is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: 3366432 [06:53:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 148 processes [07:06:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:06:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:19:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:36:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:37:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:50:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:52:13] PROBLEM Disk Space is now: WARNING on kubo i-000003dd.pmtpa.wmflabs output: DISK WARNING - free space: / 315 MB (3% inode=66%): [07:57:22] PROBLEM Disk Space is now: CRITICAL on kubo i-000003dd.pmtpa.wmflabs output: DISK CRITICAL - free space: / 285 MB (2% inode=66%): [08:07:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:08:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:21:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:38:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:38:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:51:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:07:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [09:07:42] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3375952 [09:08:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:08:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:10:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 6.69, 6.69, 5.38 [09:12:42] RECOVERY Free ram is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: 3371712 [09:21:54] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:26:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.13, 6.67, 5.30 [09:27:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 7.73, 7.13, 5.57 [09:33:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.96, 5.78, 5.19 [09:35:42] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3376168 [09:37:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [09:38:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:38:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:51:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [09:52:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:01:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 4.32, 4.34, 4.97 [10:03:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 2.50, 3.82, 4.75 [10:08:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:08:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:22:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:29:53] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.76, 5.86, 5.24 [10:33:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 2.38, 4.64, 5.01 [10:36:44] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 6.86, 6.23, 5.41 [10:38:14] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:38:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:43:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 2.65, 4.05, 4.67 [10:53:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:08:53] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:09:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:23:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:26:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.30, 4.15, 4.71 [11:37:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 5.44, 5.18, 5.31 [11:39:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:39:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:42:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.61, 4.53, 4.83 [11:50:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.22, 5.27, 5.17 [11:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:00:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.22, 4.49, 4.86 [12:07:23] PROBLEM Total processes is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [12:07:53] PROBLEM Free ram is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [12:09:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:09:23] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:13:42] PROBLEM dpkg-check is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [12:13:52] PROBLEM Current Load is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [12:14:42] PROBLEM Free ram is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: 3377808 [12:17:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to popen() failed [12:18:53] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:21:54] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Server answer: [12:22:23] PROBLEM Total processes is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:23:43] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:24:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:24:33] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:26:22] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [12:39:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:39:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:09:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:09:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:24:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:39:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:39:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:55:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:10:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:11:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:26:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:41:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:41:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:57:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:11:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:11:44] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:27:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:32:14] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 159 processes [15:41:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:41:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:58:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:02:22] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 150 processes [16:10:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.75, 4.61, 4.92 [16:12:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:13:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:28:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:33:26] Emw: are they both on the same box? did you get your IP? [16:42:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:43:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:59:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:05:52] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=610563 edit summary: /* User:Babitha */ done [17:05:56] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=610564 edit summary: /* User:Abhishek Das */ done [17:06:01] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=610565 edit summary: /* User:Himeshi */ done [17:12:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:13:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:29:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:39:04] i know there are dumps somewhere on labs, but where are they specifically? for bots project... [17:42:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:43:26] globally on /public/datasets/public/ [17:43:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:45:07] giftpflanze: thanks just found it :) [17:48:11] ryan_lane: It looks to me like labsconsole is sick again. Do you want to do a post-mortem before I restart memcached? [17:48:42] nope. seems memcache segfaulted [17:49:32] you should go ahead and restart it if you're logged in already :) [17:50:41] ok [17:54:49] it seems the OOM killer killed a whole bunch of crap at some time during this box's uptime [17:54:56] may be worth rebooting it at some point [17:55:01] so that it's up in a clean way [17:55:32] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 155 processes [17:55:35] virt0 was OOM? I thought that was just bastion... [17:55:41] maybe those are two different tales [17:55:48] different, yeah [17:55:59] OOM can occur without taking the box out [17:56:09] if it can free up enough memory to continue to run... [17:58:52] PROBLEM Current Load is now: CRITICAL on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: Connection refused by host [17:58:52] PROBLEM Current Load is now: WARNING on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: WARNING - load average: 6.43, 6.64, 5.41 [17:59:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:59:32] PROBLEM Current Users is now: CRITICAL on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: Connection refused by host [18:00:13] PROBLEM Disk Space is now: CRITICAL on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: Connection refused by host [18:01:04] PROBLEM Free ram is now: CRITICAL on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: Connection refused by host [18:02:23] PROBLEM Total processes is now: CRITICAL on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: Connection refused by host [18:02:53] PROBLEM dpkg-check is now: CRITICAL on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: Connection refused by host [18:07:23] RECOVERY Total processes is now: OK on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: PROCS OK: 83 processes [18:07:53] RECOVERY dpkg-check is now: OK on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: All packages OK [18:08:53] RECOVERY Current Load is now: OK on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: OK - load average: 0.27, 0.92, 0.69 [18:09:33] RECOVERY Current Users is now: OK on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:10:12] RECOVERY Disk Space is now: OK on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: DISK OK [18:11:02] RECOVERY Free ram is now: OK on mwreview-andrewdev i-0000051f.pmtpa.wmflabs output: OK: 1195% free memory [18:12:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:13:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:25:52] PROBLEM Current Load is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: WARNING - load average: 7.67, 7.40, 5.78 [18:28:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 5.54, 5.75, 5.25 [18:29:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:29:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 7.73, 7.45, 6.11 [18:38:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 3.99, 4.40, 4.85 [18:42:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:43:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:44:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.20, 4.21, 4.91 [18:59:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:12:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:13:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:17:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 5.59, 5.73, 5.25 [19:26:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 7.47, 6.45, 5.59 [19:29:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:31:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 5.48, 5.72, 5.21 [19:36:52] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 2.45, 3.84, 4.57 [19:42:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:43:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:59:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:03:43] PROBLEM Free ram is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Critical: 4% free memory [20:08:42] RECOVERY Free ram is now: OK on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: OK: 26% free memory [20:13:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:14:42] PROBLEM Free ram is now: UNKNOWN on aggregator2 i-000002c0.pmtpa.wmflabs output: Invalid host name i-000002c0.pmtpa.wmflabs [20:19:43] PROBLEM Free ram is now: CRITICAL on aggregator2 i-000002c0.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:20:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [20:21:43] PROBLEM Free ram is now: WARNING on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: Warning: 10% free memory [20:24:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.88, 5.45, 5.88 [20:30:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:43:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:43:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:00:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:00:53] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 3.81, 3.91, 4.68 [21:11:51] !log gerrit adding saper to project [21:11:53] Logged the message, Master [21:13:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:13:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [21:17:24] aye aye, sir [21:17:25] \o/ [21:17:29] !instances [21:17:29] need help? -> https://labsconsole.wikimedia.org/wiki/Help:Instances want to manage? -> https://labsconsole.wikimedia.org/wiki/Special:NovaInstance want resources? use !resource [21:17:33] !security [21:17:34] https://labsconsole.wikimedia.org/wiki/Help:Security_Groups [21:17:52] !access [21:17:52] https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [21:18:09] I highly recommend reading those specific docs [21:18:16] @search getting [21:18:16] No results were found, remember, the bot is searching through content of keys and their names [21:18:25] @search stared [21:18:25] No results were found, remember, the bot is searching through content of keys and their names [21:18:29] @search started [21:18:29] Results (Found 1): start, [21:18:33] !start [21:18:33] Welcome to Wikimedia Labs! Get yourself started at https://labsconsole.wikimedia.org/wiki/Help:Getting_Started [21:19:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.65, 4.71, 4.96 [21:20:24] !logging [21:20:24] To log a message, use the following format: !log [21:20:44] you should log what you are doing, when you do things. it makes it easier for people to collaborate [21:20:49] * saper logs scrollback [21:20:51] the logs show up on the project pages [21:20:56] !resource Gerrit [21:20:56] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Gerrit [21:21:24] <^demon> Yeah, so it's pretty much puppetized. The role::gerrit::labs stuff is kind of hardcoded still. [21:22:08] saper: also, if you have any problems or get stuck, I'm here to help [21:22:13] <^demon> For setting up a second instance, hashar would know too. He setup one recently to play with zuul. [21:22:17] oh, jenkins is there too! great [21:22:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 3.66, 4.29, 4.83 [21:22:52] Ryan_Lane: thanks, need to study a bit first. But there is no study like in the middle of fight [21:23:09] :D [21:23:21] hey:) [21:23:25] <^demon> Oh, and for mysql, I've got a totally unpuppetized instance called gerrit-db. [21:23:40] <^demon> Just feel free to create extra databases on it. Login info is in /root [21:23:59] saper I have installed gerrit/jenkins/zuul from scratch in labs to prepare production [21:24:13] ends up that putting zuul in production took 2 times half an hour :-] [21:24:23] saving a loooot of precious ops time [21:24:46] it probably took us less time to deploy it than to schedule the deployment. [21:24:54] * Ryan_Lane loves to hear this [21:25:04] <^demon> Did you end up using gerrit::instance or something? [21:25:13] <^demon> Or did you use the labs role, which we should clean up more. [21:25:58] Ryan_Lane: I still have to improve my puppet skills though [21:26:07] Ryan_Lane: but overall labs + puppetmaster:self have been a life changer [21:26:14] great :) [21:26:23] I am actually integrating the changes I want in production [21:26:30] hashar: I wanted to change your little script - it just got merged - it should be a three liner using JENKINS_HOME but wanted to make sure it's available [21:26:33] and ops just have to care about having doc :-] [21:26:46] ^demon: I got a labs role iirc [21:26:59] ^demon: relying on gerrit:instance , but I had to setup most of Gerrit manually [21:27:09] ^demon: I just followed your doc on wikitech [21:27:35] saper: which little script? i have a ton of scripts :-] [21:27:40] s/have/wrote/ [21:27:52] (cause we all know what we produce is OSS and belongs to the worldwide community) [21:28:16] Ryan_Lane: forgot to write you a report about zuul progress. Basically it is in production, pending a few minor tweaks [21:28:26] sweet [21:28:31] Ryan_Lane: and making sure eng folks have read my lengthy workflow mail to engineering list [21:28:47] with self::registration WMF is going to kick a** [21:28:50] hashar: wmfgrunt [21:28:57] saper: oh that one, sure! [21:29:04] agreed. I'm looking forward to self registration [21:29:06] saper: I should have added you as a reviewer since you are a bash guru :-] [21:29:26] Ryan_Lane: we all are. And that has been my focus since you raised it (again) 2 months ago [21:29:40] heh [21:29:40] Ryan_Lane: it took a long time but I got all the technical debt resorbed [21:29:46] yeah [21:29:49] and will be able to scale CI to more people [21:29:54] that's great [21:30:02] such a pity I could not have done that last spring :/ [21:30:16] <^demon> saper: gerrit-dev.wmflabs.org is now running master + my patch [21:30:19] hashar: I was added but ENOTIME [21:30:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:30:42] saper: at least you know about the script :-] [21:31:50] hashar: yes yes posted a "too late" review a moment ago [21:32:15] Ryan_Lane: btw, zuul has a bug that prevents it from triggering job on repository that do not have "master" as a default branch :-] [21:32:23] -_- [21:32:24] Ryan_Lane: so can't use it yet on operations/puppet.git :) [21:32:28] damn it [21:32:31] heh [21:32:49] that's what I get for naming master something practical rather than standard [21:32:57] ^demon: can't enter /root as "saper" [21:33:11] saper: use: sudo su - [21:33:16] or sudo -s [21:33:17] Ryan_Lane: I talked to openstack folks about it, I got a patch somewhere I need to test. [21:33:18] <^demon> Ryan_Lane, hashar: Add it to the big list of reasons I can be mad at Ryan for naming it production :) [21:33:29] it's your labsconsole password [21:33:35] ^demon: :) [21:33:41] well now that we no more have a "test" branch, we could probably rename production to master [21:33:47] Ryan_Lane: both ask for password [21:33:48] but that can cause some nasty side effects :/ [21:33:57] or make master to points to production [21:34:01] <^demon> Eh, easy enough to rename. [21:34:01] ah [21:34:05] <^demon> Would just have to adjust HEAD. [21:34:36] could rename, but we reference it in a bunch of places [21:34:52] including every checkout [21:34:59] and puppetmaster::self [21:35:11] hashar: is zuul code review as slow as git-review's? :) [21:35:16] I'll bring it up at the next ops meeting [21:35:36] saper: it depends, I usually get my code reviewed during my evenings [21:35:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 4.67, 4.98, 5.03 [21:35:50] saper: I send amend when I am connected during the evening, and that got merged the day after [21:35:52] PROBLEM Free ram is now: CRITICAL on gerrit-dev-zap i-00000520.pmtpa.wmflabs output: Connection refused by host [21:35:52] PROBLEM Current Load is now: CRITICAL on gerrit-dev-zap i-00000520.pmtpa.wmflabs output: Connection refused by host [21:35:52] PROBLEM Current Load is now: CRITICAL on mwreview-andrewdev2 i-00000521.pmtpa.wmflabs output: Connection refused by host [21:35:57] saper: I would say 2 days on average. [21:36:02] PROBLEM Free ram is now: CRITICAL on mwreview-andrewdev2 i-00000521.pmtpa.wmflabs output: Connection refused by host [21:36:17] hashar: lucky [21:36:27] certainly, git-review has priority -1 [21:36:32] PROBLEM Current Users is now: CRITICAL on gerrit-dev-zap i-00000520.pmtpa.wmflabs output: Connection refused by host [21:36:33] PROBLEM Current Users is now: CRITICAL on mwreview-andrewdev2 i-00000521.pmtpa.wmflabs output: Connection refused by host [21:36:45] saper: I am mostly sending doc updates and tiny changes to fit our needs. So they are probably easier to review. [21:36:56] I just created gerrit-dev-zap [21:37:12] PROBLEM Disk Space is now: CRITICAL on gerrit-dev-zap i-00000520.pmtpa.wmflabs output: Connection refused by host [21:37:12] PROBLEM Disk Space is now: CRITICAL on mwreview-andrewdev2 i-00000521.pmtpa.wmflabs output: Connection refused by host [21:37:22] PROBLEM Total processes is now: CRITICAL on gerrit-dev-zap i-00000520.pmtpa.wmflabs output: Connection refused by host [21:37:23] PROBLEM Total processes is now: CRITICAL on mwreview-andrewdev2 i-00000521.pmtpa.wmflabs output: Connection refused by host [21:37:24] hashar: I am desperate already. I made a 2nd attempt; splitting changes into little small chunks... didn't help [21:37:42] you should talk to them in #openstack-infra [21:37:48] hashar: did already [21:38:01] saper: I have a got a bit too much to do myself but whenever I got time I will test / review your changes [21:38:12] PROBLEM dpkg-check is now: CRITICAL on gerrit-dev-zap i-00000520.pmtpa.wmflabs output: Connection refused by host [21:38:12] PROBLEM dpkg-check is now: CRITICAL on mwreview-andrewdev2 i-00000521.pmtpa.wmflabs output: Connection refused by host [21:38:30] hashar: they now decided to do "proper automated testing" to make sure "release will not be botched". And I should wait for review until existing stuff gets released and the testing runs. [21:38:40] ah that also [21:38:57] that is definitely frustrating for you [21:39:06] I cannot figure out how you can possibly automate testing for a tool like this, really [21:39:16] but having tests will guarantee there will be no regressions when releasing a new version [21:39:22] hashar: I think I will just fork it [21:39:29] <^demon> I say fork it. [21:39:35] <^demon> Then we can make it as wmf-specific as we want. [21:39:44] <^demon> (Heck, it's easier to get something merged to gerrit itself) [21:39:59] they are kind of insane about requiring tests [21:40:20] come on, to simulate environment for git-review you need (1) gerrit (2) local git repo at (3) various states of decomposition and bugs. Plus botched .ssh/config, maybe [21:40:40]