[02:47:02] PROBLEM Current Load is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:48:58] PROBLEM Current Users is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:48:58] PROBLEM Disk Space is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:48:58] PROBLEM Free ram is now: CRITICAL on worker1 i-00000208 output: CHECK_NRPE: Socket timeout after 10 seconds. [02:49:44] PROBLEM Total Processes is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [02:49:49] PROBLEM Free ram is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [02:50:07] PROBLEM Current Users is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [02:50:45] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 6.46, 6.22, 3.92 [02:51:48] RECOVERY Current Load is now: OK on worker1 i-00000208 output: OK - load average: 2.15, 4.47, 3.08 [02:52:20] 05/19/2012 - 02:52:20 - Updating keys for laner at /export/home/deployment-prep/laner [02:52:54] RECOVERY Current Users is now: OK on worker1 i-00000208 output: USERS OK - 0 users currently logged in [02:52:54] RECOVERY Disk Space is now: OK on worker1 i-00000208 output: DISK OK [02:52:54] RECOVERY Free ram is now: OK on worker1 i-00000208 output: OK: 94% free memory [02:54:27] 05/19/2012 - 02:54:27 - Updating keys for laner at /export/home/deployment-prep/laner [02:54:27] RECOVERY Total Processes is now: OK on upload-wizard i-0000021c output: PROCS OK: 82 processes [02:54:32] RECOVERY Free ram is now: OK on upload-wizard i-0000021c output: OK: 94% free memory [02:54:32] RECOVERY Current Users is now: OK on upload-wizard i-0000021c output: USERS OK - 0 users currently logged in [02:55:27] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 1.78, 3.57, 3.41 [02:58:19] 05/19/2012 - 02:58:19 - Updating keys for laner at /export/home/deployment-prep/laner [03:00:30] RECOVERY Current Load is now: OK on nagios 127.0.0.1 output: OK - load average: 0.56, 1.60, 2.58 [03:03:20] 05/19/2012 - 03:03:19 - Updating keys for laner at /export/home/deployment-prep/laner [03:05:20] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [03:05:21] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000219 output: CRITICAL - Socket timeout after 10 seconds [03:05:21] PROBLEM HTTP is now: CRITICAL on deployment-web4 i-00000214 output: CRITICAL - Socket timeout after 10 seconds [03:05:21] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: CRITICAL - Socket timeout after 10 seconds [03:10:00] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.015 second response time [03:10:00] PROBLEM HTTP is now: WARNING on deployment-web3 i-00000219 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.013 second response time [03:10:00] PROBLEM HTTP is now: WARNING on deployment-web4 i-00000214 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.013 second response time [03:10:00] PROBLEM HTTP is now: WARNING on deployment-web5 i-00000213 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.008 second response time [03:13:20] 05/19/2012 - 03:13:20 - Updating keys for laner at /export/home/deployment-prep/laner [03:38:02] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 16% free memory [03:43:03] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 14% free memory [03:50:18] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 16% free memory [03:58:20] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 4% free memory [04:02:28] PROBLEM Free ram is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [04:03:07] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:06:49] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 15% free memory [04:07:09] RECOVERY Free ram is now: OK on rds i-00000207 output: OK: 94% free memory [04:08:09] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 5% free memory [04:11:21] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 3% free memory [04:11:21] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [04:11:21] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000219 output: CRITICAL - Socket timeout after 10 seconds [04:11:21] PROBLEM HTTP is now: CRITICAL on deployment-web4 i-00000214 output: CRITICAL - Socket timeout after 10 seconds [04:11:35] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: CRITICAL - Socket timeout after 10 seconds [04:16:04] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:18:19] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:26:53] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 5% free memory [04:37:03] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 95% free memory [05:34:08] PROBLEM Puppet freshness is now: CRITICAL on localpuppet1 i-0000020b output: Puppet has not run in last 20 hours [05:50:58] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.022 second response time [05:50:58] PROBLEM HTTP is now: WARNING on deployment-web4 i-00000214 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.012 second response time [05:50:58] PROBLEM HTTP is now: WARNING on deployment-web5 i-00000213 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.016 second response time [05:50:58] PROBLEM HTTP is now: WARNING on deployment-web3 i-00000219 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.021 second response time [06:01:11] PROBLEM HTTP is now: CRITICAL on deployment-apache20 i-0000026c output: CRITICAL - Socket timeout after 10 seconds [06:06:00] PROBLEM HTTP is now: WARNING on deployment-apache20 i-0000026c output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.012 second response time [06:33:58] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 2.59, 4.38, 3.00 [06:40:25] PROBLEM Current Load is now: CRITICAL on nagios 127.0.0.1 output: CRITICAL - load average: 3.90, 5.86, 4.20 [06:46:36] PROBLEM Free ram is now: CRITICAL on ganglia-test2 i-00000250 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:08] PROBLEM Free ram is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:08] PROBLEM Current Load is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:47:08] PROBLEM Total Processes is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:00] PROBLEM Current Users is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:00] PROBLEM Disk Space is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:00] PROBLEM dpkg-check is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:00] PROBLEM Free ram is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [06:55:05] PROBLEM SSH is now: CRITICAL on ganglia-test2 i-00000250 output: CRITICAL - Socket timeout after 10 seconds [07:00:31] PROBLEM Current Load is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:31] PROBLEM Current Users is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:31] PROBLEM Disk Space is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:31] PROBLEM Total Processes is now: CRITICAL on migration1 i-00000261 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:41] PROBLEM dpkg-check is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:00:41] PROBLEM Free ram is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:01:49] PROBLEM Current Load is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:01:49] PROBLEM Total Processes is now: CRITICAL on precise-test i-00000231 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:03:13] PROBLEM HTTP is now: CRITICAL on deployment-web4 i-00000214 output: CRITICAL - Socket timeout after 10 seconds [07:03:32] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: CRITICAL - Socket timeout after 10 seconds [07:03:32] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [07:03:32] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000219 output: CRITICAL - Socket timeout after 10 seconds [07:04:42] PROBLEM Current Load is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:04:42] PROBLEM dpkg-check is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:04:42] PROBLEM Total Processes is now: CRITICAL on bots-sql2 i-000000af output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:04] PROBLEM Current Load is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:04] PROBLEM Total Processes is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:09] PROBLEM dpkg-check is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:09] PROBLEM Current Users is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:09] PROBLEM Disk Space is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:09] PROBLEM Current Users is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:09] PROBLEM Disk Space is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:09] PROBLEM Free ram is now: CRITICAL on incubator-bot1 i-00000251 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:05:23] PROBLEM Current Load is now: WARNING on upload-wizard i-0000021c output: WARNING - load average: 6.70, 7.96, 6.36 [07:05:23] RECOVERY Total Processes is now: OK on upload-wizard i-0000021c output: PROCS OK: 87 processes [07:05:28] RECOVERY Free ram is now: OK on upload-wizard i-0000021c output: OK: 93% free memory [07:06:27] PROBLEM Current Load is now: WARNING on migration1 i-00000261 output: WARNING - load average: 3.34, 5.52, 5.07 [07:06:27] RECOVERY Current Users is now: OK on migration1 i-00000261 output: USERS OK - 0 users currently logged in [07:06:27] RECOVERY Disk Space is now: OK on migration1 i-00000261 output: DISK OK [07:06:27] RECOVERY Total Processes is now: OK on migration1 i-00000261 output: PROCS OK: 86 processes [07:06:32] RECOVERY Current Load is now: OK on precise-test i-00000231 output: OK - load average: 0.98, 4.14, 5.00 [07:06:32] RECOVERY dpkg-check is now: OK on precise-test i-00000231 output: All packages OK [07:06:34] RECOVERY Current Users is now: OK on upload-wizard i-0000021c output: USERS OK - 0 users currently logged in [07:06:34] RECOVERY Free ram is now: OK on precise-test i-00000231 output: OK: 86% free memory [07:06:34] RECOVERY Total Processes is now: OK on precise-test i-00000231 output: PROCS OK: 81 processes [07:06:37] RECOVERY Disk Space is now: OK on upload-wizard i-0000021c output: DISK OK [07:06:37] RECOVERY dpkg-check is now: OK on migration1 i-00000261 output: All packages OK [07:06:37] RECOVERY Free ram is now: OK on migration1 i-00000261 output: OK: 84% free memory [07:08:33] PROBLEM Current Load is now: WARNING on bots-sql2 i-000000af output: WARNING - load average: 9.89, 9.03, 7.92 [07:08:59] PROBLEM Current Load is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:59] PROBLEM Free ram is now: CRITICAL on rds i-00000207 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:08:59] PROBLEM Disk Space is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:10:46] PROBLEM Current Load is now: WARNING on bots-cb i-0000009e output: WARNING - load average: 9.37, 9.42, 7.51 [07:10:46] PROBLEM HTTP is now: WARNING on deployment-web5 i-00000213 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.009 second response time [07:10:46] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.015 second response time [07:10:46] PROBLEM HTTP is now: WARNING on deployment-web3 i-00000219 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.030 second response time [07:10:46] PROBLEM HTTP is now: WARNING on deployment-web4 i-00000214 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.008 second response time [07:12:16] RECOVERY Total Processes is now: OK on rds i-00000207 output: PROCS OK: 79 processes [07:12:21] RECOVERY Current Users is now: OK on rds i-00000207 output: USERS OK - 0 users currently logged in [07:12:21] RECOVERY Disk Space is now: OK on rds i-00000207 output: DISK OK [07:12:24] RECOVERY Total Processes is now: OK on bots-sql2 i-000000af output: PROCS OK: 95 processes [07:12:26] RECOVERY dpkg-check is now: OK on bots-sql2 i-000000af output: All packages OK [07:12:31] PROBLEM Current Load is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:35] PROBLEM Disk Space is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:37] PROBLEM dpkg-check is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:37] PROBLEM Free ram is now: CRITICAL on reportcard2 i-000001ea output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:40] PROBLEM dpkg-check is now: CRITICAL on incubator-bot2 i-00000252 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:40] RECOVERY Current Load is now: OK on migration1 i-00000261 output: OK - load average: 0.43, 1.65, 3.36 [07:13:35] PROBLEM Current Load is now: CRITICAL on upload-wizard i-0000021c output: CHECK_NRPE: Socket timeout after 10 seconds. [07:13:45] PROBLEM Current Load is now: WARNING on rds i-00000207 output: WARNING - load average: 4.06, 5.81, 6.43 [07:13:45] RECOVERY Free ram is now: OK on rds i-00000207 output: OK: 94% free memory [07:13:45] PROBLEM Current Load is now: WARNING on labs-nfs1 i-0000005d output: WARNING - load average: 1.39, 4.00, 5.13 [07:14:07] PROBLEM Current Load is now: WARNING on swift-be4 i-000001ca output: WARNING - load average: 5.41, 7.29, 6.28 [07:14:41] PROBLEM Current Load is now: WARNING on ganglia-test2 i-00000250 output: WARNING - load average: 5.06, 6.74, 12.85 [07:14:41] PROBLEM Free ram is now: WARNING on ganglia-test2 i-00000250 output: Warning: 13% free memory [07:14:47] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: CRITICAL - Socket timeout after 10 seconds [07:14:47] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [07:14:47] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000219 output: CRITICAL - Socket timeout after 10 seconds [07:14:47] PROBLEM HTTP is now: CRITICAL on deployment-web4 i-00000214 output: CRITICAL - Socket timeout after 10 seconds [07:16:23] PROBLEM Current Load is now: WARNING on incubator-bot1 i-00000251 output: WARNING - load average: 2.15, 5.85, 7.90 [07:16:23] RECOVERY Disk Space is now: OK on incubator-bot1 i-00000251 output: DISK OK [07:16:23] RECOVERY Current Users is now: OK on incubator-bot1 i-00000251 output: USERS OK - 0 users currently logged in [07:16:23] RECOVERY Free ram is now: OK on incubator-bot1 i-00000251 output: OK: 46% free memory [07:16:23] RECOVERY dpkg-check is now: OK on incubator-bot1 i-00000251 output: All packages OK [07:16:23] PROBLEM Current Load is now: WARNING on incubator-bot2 i-00000252 output: WARNING - load average: 1.67, 5.94, 7.12 [07:16:24] RECOVERY dpkg-check is now: OK on incubator-bot2 i-00000252 output: All packages OK [07:16:24] RECOVERY Disk Space is now: OK on reportcard2 i-000001ea output: DISK OK [07:16:24] RECOVERY Current Load is now: OK on reportcard2 i-000001ea output: OK - load average: 3.74, 3.83, 3.41 [07:16:25] RECOVERY Free ram is now: OK on reportcard2 i-000001ea output: OK: 85% free memory [07:16:26] RECOVERY dpkg-check is now: OK on reportcard2 i-000001ea output: All packages OK [07:17:33] PROBLEM Current Load is now: WARNING on deployment-nfs-memc i-000000d7 output: WARNING - load average: 8.23, 8.83, 6.73 [07:17:40] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: CHECK_NRPE: Socket timeout after 10 seconds. [07:18:34] RECOVERY Current Load is now: OK on labs-nfs1 i-0000005d output: OK - load average: 0.99, 2.02, 3.92 [07:18:34] RECOVERY Disk Space is now: OK on incubator-bot2 i-00000252 output: DISK OK [07:19:07] RECOVERY Current Load is now: OK on swift-be4 i-000001ca output: OK - load average: 0.32, 2.81, 4.59 [07:22:24] PROBLEM Current Load is now: WARNING on mobile-testing i-00000271 output: WARNING - load average: 2.19, 8.35, 9.48 [07:23:34] RECOVERY Current Load is now: OK on rds i-00000207 output: OK - load average: 0.02, 1.09, 3.66 [07:24:12] RECOVERY Current Load is now: OK on bots-cb i-0000009e output: OK - load average: 0.48, 1.53, 4.12 [07:26:38] RECOVERY Current Load is now: OK on incubator-bot1 i-00000251 output: OK - load average: 0.65, 1.30, 4.31 [07:26:38] RECOVERY Current Load is now: OK on incubator-bot2 i-00000252 output: OK - load average: 0.43, 0.96, 3.71 [07:34:41] RECOVERY Current Load is now: OK on ganglia-test2 i-00000250 output: OK - load average: 0.57, 0.84, 4.53 [07:37:30] RECOVERY Current Load is now: OK on deployment-nfs-memc i-000000d7 output: OK - load average: 0.88, 1.74, 4.10 [07:37:30] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 0.78, 1.19, 4.17 [07:38:20] RECOVERY Current Load is now: OK on bots-sql2 i-000000af output: OK - load average: 4.20, 4.21, 4.91 [07:41:30] PROBLEM Current Load is now: WARNING on nagios 127.0.0.1 output: WARNING - load average: 0.64, 0.84, 3.86 [09:05:19] PROBLEM HTTP is now: WARNING on deployment-web5 i-00000213 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.015 second response time [09:05:19] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.015 second response time [09:05:19] PROBLEM HTTP is now: WARNING on deployment-web4 i-00000214 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.016 second response time [09:05:19] PROBLEM HTTP is now: WARNING on deployment-web3 i-00000219 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.026 second response time [09:33:06] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [10:45:45] meh [10:46:18] mutante: we really want to get it to stable server [10:46:31] !log thingie [10:47:03] I guess only Ryan has the code of bot [11:46:08] !log deployment-prep petrb: creating aa wiki [11:46:10] Logged the message, Master [12:01:30] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: CRITICAL - Socket timeout after 10 seconds [12:01:30] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [12:01:30] PROBLEM HTTP is now: CRITICAL on deployment-web4 i-00000214 output: CRITICAL - Socket timeout after 10 seconds [12:01:30] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000219 output: CRITICAL - Socket timeout after 10 seconds [12:01:40] PROBLEM Free ram is now: CRITICAL on bots-2 i-0000009c output: CHECK_NRPE: Socket timeout after 10 seconds. [12:05:59] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.010 second response time [12:05:59] PROBLEM HTTP is now: WARNING on deployment-web5 i-00000213 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.014 second response time [12:05:59] PROBLEM HTTP is now: WARNING on deployment-web4 i-00000214 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.014 second response time [12:05:59] PROBLEM HTTP is now: WARNING on deployment-web3 i-00000219 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.013 second response time [12:06:19] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 13% free memory [12:24:35] petan|wk: http://simple.wikipedia.beta.wmflabs.org/wiki/Special:Contributions/Savh http://simple.wikipedia.beta.wmflabs.org/w/index.php?title=User:Mwchao&action=history what's the cause for this? someone mentioned to me before import tables ? [12:25:56] Thehelpfulone, what's the problem? [12:26:05] look at those dates Platonides [12:26:17] was it imported ? [12:26:43] hmm.. maybe the server had the wrong date [12:26:46] I'm not sure - I can't see anything in logs [12:27:01] wtf [12:27:10] compare http://simple.wikipedia.org/w/index.php?title=User:Mwchao&action=history and http://simple.wikipedia.org/w/index.php?title=User:Mwchao&action=history [12:27:17] and http://simple.wikipedia.beta.wmflabs.org/w/index.php?title=User:Mwchao&action=history [12:27:25] I copied the same url twice [12:27:38] it broke the names [12:27:46] yeah exactly that's what I don't get [12:28:02] oooh could it be to do with user IDs? [12:28:26] so whatever Savh's user ID is, that was what it was for Mwchao ? [12:28:58] looks like the user table was not imported from the original wikis [12:29:04] but we are using our own one [12:29:28] which leads to broken data when joining [12:29:39] but why would action=history perform a join with user table? [12:29:50] the data is there in a field! [12:38:56] r100286 :( [15:01:56] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: CRITICAL - Socket timeout after 10 seconds [15:01:57] PROBLEM HTTP is now: CRITICAL on deployment-web4 i-00000214 output: CRITICAL - Socket timeout after 10 seconds [15:01:57] PROBLEM HTTP is now: CRITICAL on deployment-web3 i-00000219 output: CRITICAL - Socket timeout after 10 seconds [15:01:57] PROBLEM HTTP is now: CRITICAL on deployment-web i-00000217 output: CRITICAL - Socket timeout after 10 seconds [15:06:45] PROBLEM HTTP is now: WARNING on deployment-web5 i-00000213 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.012 second response time [15:06:45] PROBLEM HTTP is now: WARNING on deployment-web4 i-00000214 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.007 second response time [15:06:45] PROBLEM HTTP is now: WARNING on deployment-web i-00000217 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.007 second response time [15:06:45] PROBLEM HTTP is now: WARNING on deployment-web3 i-00000219 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.058 second response time [15:34:50] PROBLEM Puppet freshness is now: CRITICAL on localpuppet1 i-0000020b output: Puppet has not run in last 20 hours [19:32:51] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [22:03:26] deployment doesn't send mails... [22:32:50] !log deployment-prep http://ee-prototype.wikipedia.beta.wmflabs.org/ fails with No localisation cache found for English. Please run maintenance/rebuildLocalisationCache.php. [22:33:07] !log deployment-prep Password changed for Platonides [22:33:07] Logged the message, Master [22:33:09] Logged the message, Master