[00:01:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:04:23] RECOVERY Free ram is now: OK on bots-cb i-0000009e.pmtpa.wmflabs output: OK: 333% free memory [00:15:27] 11/27/2012 - 00:15:27 - Updating keys for sborsody at /export/keys/sborsody [00:19:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:21:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [00:22:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:24:40] hi TParis! [00:24:45] Hey [00:24:50] Damianz: do you have a moment? [00:25:33] TParis: btw it would be great to update https://labsconsole.wikimedia.org/wiki/Help:Move_your_bot_to_Labs [00:26:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [00:27:25] TParis: I saw that one question of yours was about access to db replicas especially of en.wp [00:28:36] Yup [00:29:09] I know https://www.mediawiki.org/wiki/Wikimedia_Labs is pretty long, lemme see if an answer is in there [00:29:19] Enable database replication - Ryan hopes to get this done by the end of November or December 2012 [00:30:22] I understand. I'm just getting less and less reliability on Toolserver since the announcement it would close, but I cant migrate without the replication [00:30:41] Very few of X's tools do not use it [00:31:03] Understandable [00:31:21] I've now added it to https://labsconsole.wikimedia.org/wiki/Help:Move_your_bot_to_Labs#Set_up_your_bot so other people will also be able to see that [00:31:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs CRITICAL - Plugin timed out after 10 seconds [00:32:04] TParis: had you run into the "move your bot to Labs" page before? [00:32:10] if not, please feel free to link to it liberally [00:32:24] that's where we're trying to keep the knowledge & step-by-step guide you want [00:33:17] Sumanah: I need more than a step by step guide. I need to know why a lot of the extra stuff is important. It won't make sense to me unless I can associate the logic behind it. [00:33:50] TParis: can you send that as a request to labs-l so it's easier to follow up? [00:34:06] @lists.wikimedia.org? [00:34:18] yeah, https://lists.wikimedia.org/mailman/listinfo/labs-l [00:34:39] Yeah, I can do that, let me read more of it so I can recall the frustrations I had when I tried to do this months ago [00:34:57] TParis: thank you! and also this page didn't exist then so it might help [00:35:58] I do need to go and eat dinner [00:49:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [00:52:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:09:33] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 176 processes [01:14:33] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 98 processes [01:16:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [01:21:42] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c.pmtpa.wmflabs output: Warning: 18% free memory [01:21:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:23:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:31:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:51:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [01:53:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:01:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:06:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [02:21:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:23:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:32:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:39:42] RECOVERY Free ram is now: OK on bots-sql2 i-000000af.pmtpa.wmflabs output: OK: 20% free memory [02:40:30] 11/27/2012 - 02:40:29 - Updating keys for chicocvenancio at /export/keys/chicocvenancio [02:41:43] RECOVERY Free ram is now: OK on bots-2 i-0000009c.pmtpa.wmflabs output: OK: 22% free memory [02:42:33] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 149 processes [02:51:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [02:53:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:57:43] PROBLEM Free ram is now: WARNING on bots-sql2 i-000000af.pmtpa.wmflabs output: Warning: 14% free memory [03:01:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [03:02:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:04:42] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c.pmtpa.wmflabs output: Warning: 18% free memory [03:19:42] RECOVERY Free ram is now: OK on bots-2 i-0000009c.pmtpa.wmflabs output: OK: 21% free memory [03:21:52] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:23:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs CRITICAL - Plugin timed out after 10 seconds [03:32:32] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:33:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:34:42] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:34:52] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:34:52] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Server answer: [03:35:22] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [03:44:42] RECOVERY dpkg-check is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: All packages OK [03:44:52] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 1.66, 1.16, 0.72 [03:44:52] RECOVERY SSH is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [03:45:22] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [03:47:32] RECOVERY Current Users is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [03:47:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [03:52:52] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [03:53:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [03:53:52] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:57:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [04:04:03] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:06:42] PROBLEM Total processes is now: WARNING on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS WARNING: 168 processes [04:16:43] PROBLEM Total processes is now: CRITICAL on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [04:23:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:23:52] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:28:33] PROBLEM Free ram is now: WARNING on bastion1 i-000000ba.pmtpa.wmflabs output: Warning: 10% free memory [04:33:33] PROBLEM Free ram is now: CRITICAL on bastion1 i-000000ba.pmtpa.wmflabs output: Critical: 2% free memory [04:34:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:53:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [04:53:52] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:04:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:08:52] PROBLEM Total processes is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS WARNING: 151 processes [05:23:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:23:52] RECOVERY Total processes is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS OK: 144 processes [05:23:52] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:34:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:47:52] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [05:53:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [05:53:53] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:04:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:23:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:23:53] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:28:35] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:31:33] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:32:33] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 153 processes [06:34:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:37:44] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c.pmtpa.wmflabs output: Warning: 15% free memory [06:38:34] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:45:28] 11/27/2012 - 06:45:27 - Updating keys for ant at /export/keys/ant [06:46:33] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:52:32] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 149 processes [06:53:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [06:54:32] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:04:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:12:53] PROBLEM Total processes is now: WARNING on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS WARNING: 151 processes [07:17:53] RECOVERY Total processes is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: PROCS OK: 150 processes [07:23:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:24:32] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:31:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [07:34:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:42:42] RECOVERY Free ram is now: OK on bots-2 i-0000009c.pmtpa.wmflabs output: OK: 20% free memory [07:53:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [07:54:33] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:55:44] * jeremyb waves Ryan_Lane [07:55:54] howdy [07:55:57] Ryan_Lane: how hard/trivial is it to rename an LDAP user? [07:56:14] hard, because you need to rename it in mediawiki and gerrit [07:56:17] mediawiki is easy [07:56:17] has svn already, wants to have gerrit too [07:56:19] gerrit is not [07:56:37] gerrit doesn't exist yet? or does it? [07:56:42] let's see what the web UI thinks [07:56:47] I have no clue :) [07:56:53] 26 16:34:50 < jeremyb> https://www.mediawiki.org/w/index.php?title=Developer_access&oldid=609321#User:SHL [07:57:34] he doesn't want samuellampa? [07:57:50] he listed choices in order [07:57:54] svn could cause problems [07:58:01] would need to modify the access file for it [07:58:14] it should be doable, though, if the user doesn't have gerrit [07:58:17] i was thinking we could just leave svn as is and make a brand new (disconnected) account with the new name [07:58:30] could [07:58:40] and he'd use old name for svn and new name for all else [07:58:43] bleh [07:58:44] and just have 2 LDAPs [07:59:30] that's an option, for sure [07:59:41] ok, i think i'll just do that ;) [08:00:15] heh. ok [08:04:15] Ryan_Lane: btw, had an unresponsive guest a day or two ago. (ping/ssh). couldn't even get the console log? sometimes labsconsole gave a white screen of death, (but I failed to check if it was a 500) and sometimes a completely empty log. i gave up and hit reboot and then it was like 15-20 mins more before the machine was up [08:04:24] bots-3 [08:04:37] was it OOM'd? [08:04:46] that can happen on an OOM [08:04:53] no clue why it can't read the console log in that case, though [08:05:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:05:36] had oom killed some perl procs in recent days but not recent hours before it disappeared [08:05:51] unless it never made it into the log [08:06:01] it's possible [08:06:06] it may have just faulted and died [08:06:14] that's almost always the case of that error [08:07:02] but then we should have a way to find out if it's dead ;) [08:07:16] yep... [08:23:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:24:33] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:36:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:54:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [08:56:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:06:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:24:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:26:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:36:24] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:40:36] Change on 12mediawiki a page Developer access was modified, changed by Jeremyb link https://www.mediawiki.org/w/index.php?diff=610014 edit summary: /* User:SHL */ done! [09:54:03] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [09:56:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:06:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:24:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:26:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:26:27] Hi all! Is there anything wrong with bastion? It keeps throwing me out after a few seconds or 1-2 minutes. [10:30:34] (while our internet connection is okay) [10:37:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:48:26] Hi hashar! Whom can I ping about this? ^^ [10:54:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [10:56:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:08:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:24:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:26:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:31:52] Silke_WMDE: I have no real idea. Usually ssh connection drop because of a firewall in an office [11:32:12] the firewall might think that the ssh session has been inactive for too long and simply drop the connection [11:33:03] Silke_WMDE: maybe ServerAliveInterval could fix it [11:33:12] Silke_WMDE_: in your ~/.ssh/config try out adding: [11:33:15] Host * [11:33:21] ServerAliveInterval 60 [11:33:44] that will ping the server every 60 seconds [11:37:52] PROBLEM Free ram is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [11:38:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:42:42] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:42:52] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:42:52] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:42:53] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Server answer: [11:43:22] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [11:47:42] RECOVERY dpkg-check is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: All packages OK [11:47:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [11:47:52] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 0.54, 1.34, 1.22 [11:47:52] RECOVERY SSH is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [11:48:22] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [11:54:22] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [11:57:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:08:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:11:29] hi all, is BzAPI still activated? announced here but doesn't seem to work: http://blog.wikimedia.org/2010/02/26/wikimedia-bugzilla-upgraded-to-version-3-4-5-with-rest-apis/ [12:24:53] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:27:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:38:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:42:44] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Critical: 5% free memory [12:54:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [12:57:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:09:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:16:54] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [13:26:44] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:26:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [13:27:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:39:14] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:41:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [13:48:43] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 7% free memory [13:58:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:58:32] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [13:58:42] RECOVERY Free ram is now: OK on bots-3 i-000000e5.pmtpa.wmflabs output: OK: 27% free memory [14:09:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:28:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:28:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [14:31:42] PROBLEM Free ram is now: WARNING on bots-3 i-000000e5.pmtpa.wmflabs output: Warning: 19% free memory [14:39:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:58:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:58:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:08:34] hi hashar who can +2 the memcache change for beta? https://gerrit.wikimedia.org/r/#/c/35318/ [15:09:44] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:11:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [15:12:26] chrismcmahon: anyone from my team I guess [15:13:05] chrismcmahon: the real trouble is actually applying it on beta which is a bit behind the latest master [15:13:18] chrismcmahon: so running git pull will fetch like 10 - 12 days of changes which might break the site [15:14:14] hashar: beta isn't going to get any more broken than it is right now [15:28:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:28:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:40:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:44:10] chrismcmahon: now I am updating it :-) [15:44:38] chrismcmahon: I have been very busy with the Jenkins jobs recently, sorry for the lack of answer other the last few weeks [15:45:35] hashar: thanks! we have a couple of browser tests running against beta now. But we didn't pay much attention over the Thanksgiving break. [15:49:30] chrismcmahon: using selenium / browser stack ? [15:50:18] hashar: yes. running as builds in Jenkins, too. [15:52:43] pff [15:52:58] I can't connect to labs :( [15:53:50] keep getting disconnected :^D [15:56:27] * 6cc1ea7 - (HEAD, beta) LBFactory_Multi setup for labs (7 seconds ago) [15:56:28] * 03ade82 - (origin/master, origin/HEAD) beta: use IP for memcached server (11 minutes ago) [15:56:31] so that might be working [15:57:19] paravoid: regarding labs, I keep getting disconnected by the bastion [15:57:30] paravoid: someone from the WMDE had the same issue earlier [15:58:16] chrismcmahon: 10.4.0.81:11000 set: 100 incr: 100 get: 100 time: 0.20376801490784 [15:58:21] chrismcmahon: memcached is working again :-] [15:58:32] thanks very much hashar [15:58:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [15:58:45] there are most probably lot of other issues [15:58:52] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:01:06] I am off, dad duties [16:06:42] RECOVERY Total processes is now: OK on bastion1 i-000000ba.pmtpa.wmflabs output: PROCS OK: 92 processes [16:07:04] !log bastion disabled hazard-sj's cron, as it was accidentally DoSing bastion [16:07:05] Logged the message, Master [16:08:32] RECOVERY Free ram is now: OK on bastion1 i-000000ba.pmtpa.wmflabs output: OK: 863% free memory [16:10:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:23:50] mardetanha, are you sorted out now or still having problems? [16:28:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:28:52] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:40:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:57:16] Ryan_Lane, do you mind if I do a reset/rebase in OSM on nova-precise1? [16:57:32] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 152 processes [16:58:42] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [16:58:52] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:11:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:16:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [17:27:33] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 149 processes [17:28:43] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:28:53] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:41:17] andrewbogott: go for it [17:41:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:41:28] the live-hack I added is merged [17:42:13] <^demon> Ryan_Lane: https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/puppet+owner:Demon+CodeReview%252B1,n,z <3 [17:42:37] heh [17:42:46] let me get to these when I get into the office [17:43:02] I'm working on the deployment system today and tomorrow [17:43:47] <^demon> k :) [17:58:32] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 208 processes [17:59:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [17:59:32] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:03:32] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [18:11:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:16:52] PROBLEM dpkg-check is now: CRITICAL on gerrit-dev i-000004c9.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [18:21:52] RECOVERY dpkg-check is now: OK on gerrit-dev i-000004c9.pmtpa.wmflabs output: All packages OK [18:29:02] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:29:32] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:31:12] Ryan_Lane: do we have a policy on what's appropriate to run on bastion? [18:31:33] besides the incident at hand, I also saw what looked like an IRC bot running there [18:31:35] I don't think we have a written policy. but in general nothing is [18:31:39] bleh [18:31:44] annoying [18:31:45] I don't think that's appropriate for a bastion host [18:31:49] agreed [18:32:03] we should use cron.allow/at.allow [18:32:12] and cron.deny/at.allow [18:32:15] err [18:32:18] .deny [18:32:25] to deny everyone from running cron [18:32:26] but I thought that if we never said so to users, I shouldn't kill it right away [18:32:48] let me write an email to labs-l [18:33:00] feel free to kill it [18:33:52] PROBLEM Current Load is now: CRITICAL on gerrit-dev i-0000051d.pmtpa.wmflabs output: Connection refused by host [18:34:32] PROBLEM Current Users is now: CRITICAL on gerrit-dev i-0000051d.pmtpa.wmflabs output: Connection refused by host [18:35:13] PROBLEM Disk Space is now: CRITICAL on gerrit-dev i-0000051d.pmtpa.wmflabs output: Connection refused by host [18:35:16] <^demon> I'm rebuilding gerrit-dev. [18:35:18] <^demon> Shut up nagios. [18:36:03] PROBLEM Free ram is now: CRITICAL on gerrit-dev i-0000051d.pmtpa.wmflabs output: Connection refused by host [18:37:23] PROBLEM Total processes is now: CRITICAL on gerrit-dev i-0000051d.pmtpa.wmflabs output: Connection refused by host [18:37:53] PROBLEM dpkg-check is now: CRITICAL on gerrit-dev i-0000051d.pmtpa.wmflabs output: Connection refused by host [18:39:33] RECOVERY Current Users is now: OK on gerrit-dev i-0000051d.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [18:40:13] RECOVERY Disk Space is now: OK on gerrit-dev i-0000051d.pmtpa.wmflabs output: DISK OK [18:41:04] RECOVERY Free ram is now: OK on gerrit-dev i-0000051d.pmtpa.wmflabs output: OK: 3182% free memory [18:41:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:42:23] RECOVERY Total processes is now: OK on gerrit-dev i-0000051d.pmtpa.wmflabs output: PROCS OK: 100 processes [18:42:53] RECOVERY dpkg-check is now: OK on gerrit-dev i-0000051d.pmtpa.wmflabs output: All packages OK [18:43:54] RECOVERY Current Load is now: OK on gerrit-dev i-0000051d.pmtpa.wmflabs output: OK - load average: 0.08, 0.48, 0.45 [18:59:14] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [18:59:34] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:03:32] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes [19:11:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:12:23] PROBLEM Total processes is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [19:12:53] PROBLEM Free ram is now: UNKNOWN on dumps-bot3 i-00000503.pmtpa.wmflabs output: NRPE: Call to fork() failed [19:16:23] PROBLEM Disk Space is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:16:46] Morning labs people. Is there anybody around I can bug for bastion and editor-engagement access? [19:17:04] jeremyb or ryan_lane maybe? [19:17:23] PROBLEM Total processes is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:17:53] PROBLEM Free ram is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:18:37] lwelling: sure [19:18:45] lwelling: link to your request? [19:20:52] PROBLEM SSH is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: Server answer: [19:21:32] PROBLEM Current Users is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:21:42] PROBLEM dpkg-check is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:21:52] PROBLEM Current Load is now: CRITICAL on dumps-bot3 i-00000503.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:28:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [19:29:33] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [19:31:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:37:52] PROBLEM Free ram is now: WARNING on dumps-bot3 i-00000503.pmtpa.wmflabs output: Warning: 6% free memory [19:40:35] Change on 12mediawiki a page Developer access was modified, changed by Babitha link https://www.mediawiki.org/w/index.php?diff=610183 edit summary: [19:40:52] RECOVERY SSH is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [19:41:22] RECOVERY Disk Space is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: DISK OK [19:41:32] RECOVERY Current Users is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [19:41:42] RECOVERY dpkg-check is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: All packages OK [19:41:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:41:52] RECOVERY Current Load is now: OK on dumps-bot3 i-00000503.pmtpa.wmflabs output: OK - load average: 0.14, 0.37, 0.40 [19:52:52] PROBLEM dpkg-check is now: CRITICAL on gerrit-dev i-0000051d.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [19:59:54] PROBLEM host: i-0000039b.pmtpa.wmflabs is DOWN address: i-0000039b.pmtpa.wmflabs CRITICAL - Host Unreachable (i-0000039b.pmtpa.wmflabs) [20:01:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:11:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:18:34] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 213 processes [20:21:57] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 10% free memory [20:22:07] PROBLEM dpkg-check is now: CRITICAL on vumi i-000001e5.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:22:37] PROBLEM dpkg-check is now: CRITICAL on deployment-sql i-000000d0.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:22:37] PROBLEM dpkg-check is now: CRITICAL on nginx-dev1 i-000000f0.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:23:17] PROBLEM dpkg-check is now: CRITICAL on bastion-restricted1 i-0000019b.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:23:27] PROBLEM dpkg-check is now: CRITICAL on wep3 i-000003af.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:23:37] PROBLEM dpkg-check is now: CRITICAL on deployment-squid i-000000dc.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:23:37] PROBLEM dpkg-check is now: CRITICAL on scribunto i-0000022c.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:23:57] PROBLEM dpkg-check is now: CRITICAL on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:07] PROBLEM dpkg-check is now: CRITICAL on robh2 i-000001a2.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:07] PROBLEM dpkg-check is now: CRITICAL on swift-be2 i-000001c8.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:07] PROBLEM dpkg-check is now: CRITICAL on swift-be4 i-000001ca.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:07] PROBLEM dpkg-check is now: CRITICAL on swift-fe1 i-000001d2.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:07] PROBLEM dpkg-check is now: CRITICAL on resourceloader2-apache i-000001d7.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:07] PROBLEM dpkg-check is now: CRITICAL on udp-filter i-000001df.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:08] PROBLEM dpkg-check is now: CRITICAL on hugglewa-1 i-000001e0.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:08] PROBLEM dpkg-check is now: CRITICAL on opengrok-web i-000001e1.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:09] PROBLEM dpkg-check is now: CRITICAL on pediapress-packager i-000001e4.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:09] PROBLEM dpkg-check is now: CRITICAL on shop-analytics-main i-000001e6.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:10] PROBLEM dpkg-check is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:10] PROBLEM dpkg-check is now: CRITICAL on en-wiki-db-lucid i-0000023b.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:18] PROBLEM dpkg-check is now: CRITICAL on su-be1 i-000002e7.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:18] PROBLEM dpkg-check is now: CRITICAL on su-be2 i-000002e8.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:18] PROBLEM dpkg-check is now: CRITICAL on su-be3 i-000002e9.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:18] PROBLEM dpkg-check is now: CRITICAL on nagios-main i-0000030d.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:18] PROBLEM dpkg-check is now: CRITICAL on testing-arky i-0000033b.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:18] PROBLEM dpkg-check is now: CRITICAL on nova-ldap2 i-00000238.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:27] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-3 i-00000225.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:37] PROBLEM dpkg-check is now: CRITICAL on deployment-sql02 i-000004cb.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:37] PROBLEM dpkg-check is now: CRITICAL on swift-be1 i-000001c7.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:37] PROBLEM dpkg-check is now: CRITICAL on search-test i-000000cb.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:37] PROBLEM dpkg-check is now: CRITICAL on venus i-000000ea.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:37] PROBLEM dpkg-check is now: CRITICAL on aggregator1 i-0000010c.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:37] PROBLEM dpkg-check is now: CRITICAL on wikisource-web i-000000fe.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:37] PROBLEM dpkg-check is now: CRITICAL on bob i-0000012d.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:38] PROBLEM dpkg-check is now: CRITICAL on webserver-lcarr i-00000134.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:39] PROBLEM dpkg-check is now: CRITICAL on firstinstance i-0000013e.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:40] PROBLEM dpkg-check is now: CRITICAL on test2 i-0000013c.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:40] PROBLEM dpkg-check is now: CRITICAL on simplewikt i-00000149.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:24:41] PROBLEM dpkg-check is now: CRITICAL on dev-solr i-00000152.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:25:12] PROBLEM dpkg-check is now: CRITICAL on robh-spl i-00000369.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:25:12] PROBLEM dpkg-check is now: CRITICAL on solr-ci i-00000391.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:25:12] PROBLEM dpkg-check is now: CRITICAL on php-packaging i-000003ae.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:25:13] PROBLEM dpkg-check is now: CRITICAL on publicdata-administration i-0000019e.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:25:13] PROBLEM dpkg-check is now: CRITICAL on swift-be3 i-000001c9.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:25:13] PROBLEM dpkg-check is now: CRITICAL on deployment-mc i-0000021b.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:25:13] PROBLEM dpkg-check is now: CRITICAL on sultestdb i-0000032f.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:25:32] PROBLEM dpkg-check is now: CRITICAL on embed-sandbox i-000000d1.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [20:25:32]