[00:25:55] PROBLEM Current Load is now: CRITICAL on deployment-web5 i-00000176 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:25:55] PROBLEM Total Processes is now: CRITICAL on deployment-web5 i-00000176 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:26:06] PROBLEM Current Users is now: CRITICAL on deployment-web5 i-00000176 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:26:06] PROBLEM dpkg-check is now: CRITICAL on deployment-web5 i-00000176 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:26:15] PROBLEM Total Processes is now: CRITICAL on deployment-web3 i-00000162 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:26:20] PROBLEM Free ram is now: CRITICAL on deployment-web3 i-00000162 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:27:05] PROBLEM dpkg-check is now: CRITICAL on deployment-web3 i-00000162 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:27:45] PROBLEM Free ram is now: CRITICAL on deployment-web5 i-00000176 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:30:35] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000162 output: CRITICAL - Socket timeout after 10 seconds [00:31:05] PROBLEM SSH is now: CRITICAL on deployment-web3 i-00000162 output: CRITICAL - Socket timeout after 10 seconds [00:31:05] PROBLEM Current Load is now: CRITICAL on deployment-web3 i-00000162 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:31:05] PROBLEM Disk Space is now: CRITICAL on deployment-web3 i-00000162 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:31:05] PROBLEM Free ram is now: CRITICAL on deployment-web6 i-000001d9 output: Critical: 2% free memory [00:31:15] PROBLEM Current Users is now: CRITICAL on deployment-web3 i-00000162 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:32:15] PROBLEM Disk Space is now: CRITICAL on deployment-web5 i-00000176 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:32:45] PROBLEM SSH is now: CRITICAL on deployment-web5 i-00000176 output: CRITICAL - Socket timeout after 10 seconds [00:35:55] PROBLEM Free ram is now: WARNING on deployment-web2 i-00000125 output: Warning: 6% free memory [00:35:55] PROBLEM Current Load is now: WARNING on deployment-web6 i-000001d9 output: WARNING - load average: 15.70, 12.87, 6.57 [00:36:15] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000176 output: CRITICAL - Socket timeout after 10 seconds [00:37:05] PROBLEM Free ram is now: CRITICAL on deployment-web4 i-00000163 output: Critical: 5% free memory [00:37:35] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [00:40:55] PROBLEM Current Load is now: CRITICAL on deployment-web6 i-000001d9 output: CRITICAL - load average: 33.84, 23.54, 12.60 [00:41:05] PROBLEM Free ram is now: CRITICAL on deployment-web2 i-00000125 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:41:15] PROBLEM Current Load is now: CRITICAL on deployment-web2 i-00000125 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:42:35] PROBLEM Current Load is now: CRITICAL on deployment-web i-000000cf output: CHECK_NRPE: Socket timeout after 10 seconds. [00:42:45] PROBLEM Current Users is now: CRITICAL on deployment-web2 i-00000125 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:42:45] PROBLEM Total Processes is now: CRITICAL on deployment-web2 i-00000125 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:43:15] PROBLEM Disk Space is now: CRITICAL on deployment-web2 i-00000125 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:45:25] PROBLEM SSH is now: CRITICAL on deployment-web2 i-00000125 output: CRITICAL - Socket timeout after 10 seconds [00:45:48] PROBLEM Free ram is now: CRITICAL on deployment-web i-000000cf output: CHECK_NRPE: Socket timeout after 10 seconds. [00:45:55] PROBLEM Current Load is now: WARNING on deployment-web4 i-00000163 output: WARNING - load average: 1.96, 7.45, 6.18 [00:46:05] PROBLEM dpkg-check is now: CRITICAL on deployment-web2 i-00000125 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:46:05] PROBLEM HTTP is now: CRITICAL on deployment-web2 i-00000125 output: CRITICAL - Socket timeout after 10 seconds [00:46:15] PROBLEM SSH is now: CRITICAL on deployment-web i-000000cf output: CRITICAL - Socket timeout after 10 seconds [00:46:15] PROBLEM HTTP is now: CRITICAL on deployment-web i-000000cf output: CRITICAL - Socket timeout after 10 seconds [00:47:05] PROBLEM Free ram is now: WARNING on deployment-web4 i-00000163 output: Warning: 6% free memory [00:47:15] PROBLEM dpkg-check is now: CRITICAL on deployment-web i-000000cf output: CHECK_NRPE: Socket timeout after 10 seconds. [00:47:35] PROBLEM Disk Space is now: CRITICAL on deployment-web i-000000cf output: CHECK_NRPE: Socket timeout after 10 seconds. [00:47:35] PROBLEM Current Users is now: CRITICAL on deployment-web i-000000cf output: CHECK_NRPE: Socket timeout after 10 seconds. [00:47:35] PROBLEM Total Processes is now: CRITICAL on deployment-web i-000000cf output: CHECK_NRPE: Socket timeout after 10 seconds. [00:47:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [00:49:25] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [00:51:05] PROBLEM Current Load is now: CRITICAL on deployment-web4 i-00000163 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:52:15] PROBLEM Free ram is now: CRITICAL on deployment-web4 i-00000163 output: CHECK_NRPE: Socket timeout after 10 seconds. [00:57:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [00:58:05] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [01:08:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [01:08:35] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [01:18:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [01:20:25] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [01:27:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [01:28:05] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [01:39:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [01:39:35] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [01:49:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [01:50:25] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [01:57:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [01:58:05] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [02:10:25] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [02:10:36] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [02:20:25] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [02:20:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [02:27:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [02:28:05] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [02:40:25] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [02:40:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [02:50:25] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [02:50:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [02:57:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [02:58:05] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [03:10:25] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [03:10:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [03:20:25] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [03:20:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [03:27:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [03:28:05] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [03:39:05] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 13% free memory [03:39:05] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 14% free memory [03:40:25] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [03:40:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [03:44:05] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 17% free memory [03:50:26] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [03:50:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [03:53:55] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 14% free memory [03:54:08] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [03:54:08] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 5% free memory [03:57:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [03:58:05] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [03:59:05] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 96% free memory [04:04:05] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:04:05] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 4% free memory [04:09:05] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:10:25] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [04:10:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [04:13:55] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:18:55] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 93% free memory [04:20:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [04:20:55] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [04:27:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [04:28:15] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [04:40:25] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [04:40:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [04:50:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [04:50:55] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [04:53:55] PROBLEM Free ram is now: CRITICAL on test3 i-00000093 output: Critical: 3% free memory [04:57:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [04:58:15] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [04:58:55] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [05:10:25] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [05:10:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [05:20:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [05:20:55] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [05:27:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [05:28:15] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [05:40:25] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [05:40:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [05:50:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [05:50:55] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [05:57:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [05:58:15] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [06:10:25] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [06:10:35] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [06:20:55] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [06:20:55] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [06:27:05] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [06:28:15] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [06:40:54] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [06:41:19] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [06:51:04] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [06:51:14] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [06:58:34] PROBLEM host: deployment-web4 is DOWN address: i-00000163 CRITICAL - Host Unreachable (i-00000163) [06:59:44] PROBLEM host: deployment-web2 is DOWN address: i-00000125 CRITICAL - Host Unreachable (i-00000125) [07:10:54] PROBLEM host: deployment-web3 is DOWN address: i-00000162 CRITICAL - Host Unreachable (i-00000162) [07:12:04] PROBLEM host: deployment-web is DOWN address: i-000000cf CRITICAL - Host Unreachable (i-000000cf) [07:21:04] PROBLEM host: deployment-web5 is DOWN address: i-00000176 CRITICAL - Host Unreachable (i-00000176) [07:21:14] PROBLEM host: deployment-web6 is DOWN address: i-000001d9 CRITICAL - Host Unreachable (i-000001d9) [07:23:44] RECOVERY host: deployment-web4 is UP address: i-00000163 PING OK - Packet loss = 0%, RTA = 0.49 ms [07:25:14] RECOVERY SSH is now: OK on deployment-web2 i-00000125 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [07:25:24] RECOVERY host: deployment-web2 is UP address: i-00000125 PING OK - Packet loss = 0%, RTA = 0.77 ms [07:25:24] RECOVERY HTTP is now: OK on deployment-web3 i-00000162 output: HTTP OK: HTTP/1.1 302 Found - 549 bytes in 0.022 second response time [07:25:34] RECOVERY host: deployment-web3 is UP address: i-00000162 PING OK - Packet loss = 0%, RTA = 4.11 ms [07:25:34] RECOVERY host: deployment-web6 is UP address: i-000001d9 PING OK - Packet loss = 0%, RTA = 0.95 ms [07:25:34] RECOVERY Free ram is now: OK on deployment-web i-000000cf output: OK: 86% free memory [07:25:44] RECOVERY host: deployment-web is UP address: i-000000cf PING OK - Packet loss = 0%, RTA = 0.38 ms [07:25:44] RECOVERY host: deployment-web5 is UP address: i-00000176 PING OK - Packet loss = 0%, RTA = 4.32 ms [07:25:54] RECOVERY Current Load is now: OK on deployment-web5 i-00000176 output: OK - load average: 0.14, 0.15, 0.06 [07:25:54] RECOVERY Total Processes is now: OK on deployment-web5 i-00000176 output: PROCS OK: 111 processes [07:25:59] RECOVERY SSH is now: OK on deployment-web3 i-00000162 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [07:25:59] RECOVERY HTTP is now: OK on deployment-web2 i-00000125 output: HTTP OK: HTTP/1.1 302 Found - 549 bytes in 0.017 second response time [07:25:59] RECOVERY Current Load is now: OK on deployment-web3 i-00000162 output: OK - load average: 0.22, 0.12, 0.04 [07:25:59] RECOVERY Disk Space is now: OK on deployment-web3 i-00000162 output: DISK OK [07:25:59] RECOVERY Current Users is now: OK on deployment-web5 i-00000176 output: USERS OK - 0 users currently logged in [07:26:00] RECOVERY Current Load is now: OK on deployment-web4 i-00000163 output: OK - load average: 0.12, 0.11, 0.04 [07:26:00] RECOVERY Current Load is now: OK on deployment-web6 i-000001d9 output: OK - load average: 0.09, 0.09, 0.03 [07:26:01] RECOVERY Free ram is now: OK on deployment-web2 i-00000125 output: OK: 77% free memory [07:26:01] RECOVERY dpkg-check is now: OK on deployment-web2 i-00000125 output: All packages OK [07:26:02] RECOVERY dpkg-check is now: OK on deployment-web5 i-00000176 output: All packages OK [07:26:04] RECOVERY HTTP is now: OK on deployment-web i-000000cf output: HTTP OK: HTTP/1.1 302 Found - 549 bytes in 0.015 second response time [07:26:04] RECOVERY HTTP is now: OK on deployment-web5 i-00000176 output: HTTP OK: HTTP/1.1 302 Found - 549 bytes in 0.009 second response time [07:26:04] RECOVERY SSH is now: OK on deployment-web i-000000cf output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [07:26:04] RECOVERY Current Load is now: OK on deployment-web2 i-00000125 output: OK - load average: 0.27, 0.16, 0.06 [07:26:04] RECOVERY Free ram is now: OK on deployment-web3 i-00000162 output: OK: 85% free memory [07:27:04] RECOVERY dpkg-check is now: OK on deployment-web3 i-00000162 output: All packages OK [07:27:04] RECOVERY Free ram is now: OK on deployment-web4 i-00000163 output: OK: 61% free memory [07:27:04] RECOVERY dpkg-check is now: OK on deployment-web i-000000cf output: All packages OK [07:27:04] RECOVERY Disk Space is now: OK on deployment-web5 i-00000176 output: DISK OK [07:27:34] RECOVERY Disk Space is now: OK on deployment-web i-000000cf output: DISK OK [07:27:34] RECOVERY Free ram is now: OK on deployment-web5 i-00000176 output: OK: 79% free memory [07:27:34] RECOVERY Current Load is now: OK on deployment-web i-000000cf output: OK - load average: 0.09, 0.13, 0.07 [07:27:34] RECOVERY Total Processes is now: OK on deployment-web i-000000cf output: PROCS OK: 99 processes [07:27:39] RECOVERY Current Users is now: OK on deployment-web i-000000cf output: USERS OK - 0 users currently logged in [07:27:39] RECOVERY Current Users is now: OK on deployment-web2 i-00000125 output: USERS OK - 0 users currently logged in [07:27:39] RECOVERY Total Processes is now: OK on deployment-web2 i-00000125 output: PROCS OK: 109 processes [07:27:44] RECOVERY SSH is now: OK on deployment-web5 i-00000176 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [07:28:19] RECOVERY Disk Space is now: OK on deployment-web2 i-00000125 output: DISK OK [09:33:54] PROBLEM Free ram is now: WARNING on deployment-web2 i-00000125 output: Warning: 10% free memory [09:34:04] PROBLEM Free ram is now: WARNING on deployment-web6 i-000001d9 output: Warning: 14% free memory [09:35:04] PROBLEM Free ram is now: WARNING on deployment-web4 i-00000163 output: Warning: 13% free memory [09:41:04] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours [09:53:54] RECOVERY Free ram is now: OK on deployment-web2 i-00000125 output: OK: 20% free memory [09:54:04] RECOVERY Free ram is now: OK on deployment-web6 i-000001d9 output: OK: 21% free memory [09:55:04] RECOVERY Free ram is now: OK on deployment-web4 i-00000163 output: OK: 20% free memory [13:30:59] Deleting instances is broken? [13:53:26] vvv: Sometimes if an instance fails to come up it becomes difficult to delete... is that what you're seeing? [14:04:14] !log created account for cneubauer [14:04:19] Hm, that didn't work. [14:08:32] you have to put a project don't you andrewbogott? [14:09:08] what would that look like? [14:09:24] so !log bastion created account for cneubauer [14:09:44] I think that's right [14:09:56] In this case I didn't actually create a bastion account, just a labsconsole account. So I'm maybe logging in the wrong channel altogether. [14:11:25] hmm, then I don't know [14:11:27] https://labsconsole.wikimedia.org/wiki/Nova_Resource:Bastion/SAL [14:11:54] that suggests from what petan and mutante have done that that SAL might be used? [14:17:45] yeah, looks like. [15:06:04] andrewbogott: i was thinking the same thing, recently created git users (as in linking existing svn users) but not bastion accounts, so when you logged that it made me think "oh, should i have logged as well" [15:06:39] When Ryan shows up we can talk about what the best logging policy is. [17:00:25] Anyone that is able too check out log in credentials on labs console? [17:01:25] It seems like I have an account there, but I get no reply on any known email account even if password reset says an email has been sent [17:01:47] What would you think your username might be? [17:02:01] It should be John Erling Blad [17:03:11] https://labsconsole.wikimedia.org/wiki/User:John_Erling_Blad [17:03:17] Its an account there [17:05:01] You're in ldap with some keys so yeah you're there [17:05:08] are you just failing to login to the wiki? [17:05:36] the reset password email should be going to joxx.bxx@wikimedia.de where xx and xx look something like your nick [17:05:45] if not annoy rayn who isn't here [17:06:00] or ssmollett or someone else I can't remember [17:12:17] Strange.. it seems right [17:12:48] Ryan did push an update so he might have broken something [17:13:29] ;) [17:13:42] I'll sit around for a while [17:15:02] Daminaz: One thing, is the lastname (family name) really three letters? [17:15:21] It should be 4+4 letters [17:18:56] It's 4 letters [17:19:04] my x'ing was just not accurate [17:19:17] then I have no idea.. [17:19:23] Thanks for the help! [17:22:06] jeblad: I just created a bunch of new accounts -- was yours one of them? [17:22:51] Probably [17:23:02] wmde-accounts [17:23:33] * Damianz blames andrewbogott [17:24:10] One solution is to set the email address for the account "John Erling Blad" to jebladgmail.com [17:24:29] And then I change it later [17:27:03] hm... on second glance I don't think it was me. Unless you have an elaborate system of pseudonyms. [17:28:04] I am "John Erling Blad" aka "John Erling Blad (WMDE)" aka "jeblad" [17:28:19] http://meta.wikimedia.org/wiki/User:John_Erling_Blad_(WMDE) [17:28:44] http://meta.wikimedia.org/wiki/User:Jeblad [17:29:54] * andrewbogott nods [17:29:59] Wasn't me. [17:30:21] All of the WMF staff is about to vanish into a long, long meeting. There should be more helpful folks around in a couple of hours. [18:10:25] <^demon> Ryan_Lane: Is patchset-created run on all repos? [18:10:33] yes [18:10:36] why wouldn't it be? [18:10:45] <^demon> I'm grasping at straws. [18:10:47] heh [18:11:51] <^demon> http://p.defau.lt/?xXiXyBGTglIMtUmNjuC2Zg - I'm kinda confused why I don't see any log entries for it running on other repos. [18:22:00] petan: are you around? db error attempting to login to http://en.wikipedia.beta.wmflabs.org Unknown column 'user_options' in 'field list' (deployment-sql) [18:28:19] <^demon> Ryan_Lane: Mind poking https://gerrit.wikimedia.org/r/#change,4347 ? [18:29:36] merged through [18:29:47] <^demon> Mmk thanks. I'll be undoing that tomorrow. [18:31:30] <^demon> And if you could set something slightly more permissive on manganese:/var/www/mediawiki-extensions.txt so I can write to it, I'll get out of your hair I promise. [18:36:24] hm [18:36:37] temporarily, right/ [18:36:43] otherwise it's kind of a pain :) [18:37:34] ^demon: ok, you can write to it now [18:37:42] <^demon> Thank you. [18:37:49] yw [19:01:57] * jeremyb stabs teh console in the eye [19:01:59] > No Nova credentials found for your account. [19:41:14] PROBLEM Puppet freshness is now: CRITICAL on puppet-lucid i-00000080 output: Puppet has not run in last 20 hours [19:43:44] PROBLEM Current Load is now: CRITICAL on pediapress-packager i-000001e4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:44:24] PROBLEM Current Users is now: CRITICAL on pediapress-packager i-000001e4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:45:04] PROBLEM Disk Space is now: CRITICAL on pediapress-packager i-000001e4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:45:44] PROBLEM Free ram is now: CRITICAL on pediapress-packager i-000001e4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:46:54] PROBLEM Total Processes is now: CRITICAL on pediapress-packager i-000001e4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [19:47:34] PROBLEM dpkg-check is now: CRITICAL on pediapress-packager i-000001e4 output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:06:23] RECOVERY Disk Space is now: OK on pediapress-packager i-000001e4 output: DISK OK [20:06:53] RECOVERY Total Processes is now: OK on pediapress-packager i-000001e4 output: PROCS OK: 97 processes [20:07:33] RECOVERY dpkg-check is now: OK on pediapress-packager i-000001e4 output: All packages OK [20:08:43] RECOVERY Current Load is now: OK on pediapress-packager i-000001e4 output: OK - load average: 0.04, 0.03, 0.00 [20:09:23] RECOVERY Current Users is now: OK on pediapress-packager i-000001e4 output: USERS OK - 1 users currently logged in [20:10:43] RECOVERY Free ram is now: OK on pediapress-packager i-000001e4 output: OK: 91% free memory [20:29:01] andrewbogott: heh. Jeff_Green had also entered a bug for that :) [20:29:16] Two bugs means it'll get done twice as fast [20:29:17] ah, yeah, the project page has the same issue [20:29:23] it's already done [20:29:54] jeblad! Still around? Ryan_Lane may be able to sort out your login issue. [20:30:47] Ryan_Lane: earlier mutante and I were wondering if/where we should log new account creations. [20:31:07] it should be logged in the recent changes [20:31:17] it would be nice to have some kind of global projedt [20:31:19] *project [20:31:25] or well, a global log [20:31:53] How do I log to 'recent changes'? Is it just !log recent changes? [20:32:17] mediawiki does it automaticlaly [20:32:28] https://labsconsole.wikimedia.org/wiki/Special:RecentChanges [20:33:22] That's only if they're created via the webpage, right? I made some accounts with add-labs-user this morning. [20:33:30] (Folks who already had svn accounts) [20:33:45] when they log in, it should be logged [20:34:07] Ah, ok. I like it when the answer is 'you should do nothing'. [20:34:24] https://labsconsole.wikimedia.org/w/index.php?title=Special:ListUsers&dir=prev&creationSort=1 [20:34:56] yep, there they are. [20:37:05] do they show up here? https://labsconsole.wikimedia.org/w/index.php?title=Special%3ALog&type=newusers&user=&page=&year=&month=-1&hide_patrol_log=1 [20:37:11] nope [20:37:48] Just the one I created via the web interface [20:40:51] yeah [20:40:58] they show up in the user list, though [20:41:16] stupid mediawiki *still* has a bug with LDAP auth regarding the new user creation log [20:41:20] it's been fucking years [20:41:35] I reported a bug about this so, so long ago [21:02:40] hm. I wonder if I sort the project list returned by LDAP if it'll sort it everywhere automatically [21:02:44] * Ryan_Lane tries that [21:06:11] that does indeed work [21:06:24] that's a much better way to handle this [21:09:40] yay [21:25:36] PROBLEM dpkg-check is now: CRITICAL on pediapress-packager i-000001e4 output: DPKG CRITICAL dpkg reports broken packages [21:30:36] RECOVERY dpkg-check is now: OK on pediapress-packager i-000001e4 output: All packages OK [21:59:53] andrewbogott: better sorting? :) [22:00:37] I need to make the checkboxes in the filter list multi-column, rather than just a single one [22:00:37] much! thanks [22:00:52] 3-4 columns would make that much more compact [22:55:34] * Damianz runs around