[00:37:28] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [00:52:48] RECOVERY Free ram is now: OK on bots-2 i-0000009c output: OK: 20% free memory [01:00:48] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 19% free memory [01:05:52] RECOVERY Free ram is now: OK on bots-2 i-0000009c output: OK: 20% free memory [02:20:28] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/dzahn 3342 MB (19% inode=80%): [02:48:17] 05/14/2012 - 02:48:17 - Updating keys for laner at /export/home/deployment-prep/laner [02:59:19] 05/14/2012 - 02:59:18 - Updating keys for laner at /export/home/deployment-prep/laner [03:00:19] 05/14/2012 - 03:00:19 - Updating keys for laner at /export/home/deployment-prep/laner [03:05:19] 05/14/2012 - 03:05:18 - Updating keys for laner at /export/home/deployment-prep/laner [03:37:36] PROBLEM Free ram is now: WARNING on test-oneiric i-00000187 output: Warning: 14% free memory [03:40:46] PROBLEM Free ram is now: WARNING on orgcharts-dev i-0000018f output: Warning: 15% free memory [03:47:36] PROBLEM Free ram is now: WARNING on nova-daas-1 i-000000e7 output: Warning: 12% free memory [03:52:37] PROBLEM Free ram is now: CRITICAL on test-oneiric i-00000187 output: Critical: 5% free memory [03:54:46] PROBLEM Free ram is now: WARNING on utils-abogott i-00000131 output: Warning: 17% free memory [03:56:05] PROBLEM SSH is now: CRITICAL on dumps-2 i-00000257 output: No route to host [03:56:05] PROBLEM dpkg-check is now: CRITICAL on dumps-2 i-00000257 output: Connection refused or timed out [03:57:36] RECOVERY Free ram is now: OK on test-oneiric i-00000187 output: OK: 97% free memory [04:01:07] PROBLEM Free ram is now: CRITICAL on orgcharts-dev i-0000018f output: Critical: 4% free memory [04:01:27] PROBLEM host: dumps-2 is DOWN address: i-00000257 CRITICAL - Host Unreachable (i-00000257) [04:05:50] RECOVERY Free ram is now: OK on orgcharts-dev i-0000018f output: OK: 94% free memory [04:06:10] RECOVERY SSH is now: OK on dumps-2 i-00000257 output: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [04:06:20] RECOVERY dpkg-check is now: OK on dumps-2 i-00000257 output: All packages OK [04:06:20] RECOVERY host: dumps-2 is UP address: i-00000257 PING OK - Packet loss = 0%, RTA = 9.94 ms [04:07:40] PROBLEM Free ram is now: CRITICAL on nova-daas-1 i-000000e7 output: Critical: 4% free memory [04:12:40] RECOVERY Free ram is now: OK on nova-daas-1 i-000000e7 output: OK: 94% free memory [04:14:40] PROBLEM Free ram is now: CRITICAL on utils-abogott i-00000131 output: Critical: 4% free memory [04:19:40] RECOVERY Free ram is now: OK on utils-abogott i-00000131 output: OK: 97% free memory [04:22:53] RECOVERY Free ram is now: OK on incubator-bot1 i-00000251 output: OK: 24% free memory [05:06:07] PROBLEM Free ram is now: WARNING on test3 i-00000093 output: Warning: 12% free memory [05:11:07] RECOVERY Free ram is now: OK on test3 i-00000093 output: OK: 96% free memory [05:25:27] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [05:45:27] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/dzahn 2781 MB (16% inode=80%): [06:15:27] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [06:44:13] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [07:59:47] PROBLEM Disk Space is now: WARNING on nagios 127.0.0.1 output: DISK WARNING - free space: /home/dzahn 2757 MB (16% inode=80%): [09:26:18] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [10:08:15] PROBLEM HTTP is now: CRITICAL on deployment-web4 i-00000214 output: CRITICAL - Socket timeout after 10 seconds [10:08:15] PROBLEM HTTP is now: CRITICAL on deployment-web5 i-00000213 output: CRITICAL - Socket timeout after 10 seconds [10:13:00] PROBLEM HTTP is now: WARNING on deployment-web4 i-00000214 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.054 second response time [10:13:00] PROBLEM HTTP is now: WARNING on deployment-web5 i-00000213 output: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.008 second response time [10:31:04] PROBLEM Disk Space is now: CRITICAL on nagios 127.0.0.1 output: (Service Check Timed Out) [10:31:36] PROBLEM Current Load is now: WARNING on labs-nfs1 i-0000005d output: WARNING - load average: 18.96, 15.49, 9.29 [10:31:43] PROBLEM Free ram is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [10:31:43] PROBLEM Disk Space is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [10:31:43] PROBLEM dpkg-check is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [10:32:18] PROBLEM Total Processes is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [10:32:43] PROBLEM Current Load is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [10:32:43] PROBLEM Current Users is now: CRITICAL on ve-nodejs i-00000245 output: CHECK_NRPE: Socket timeout after 10 seconds. [10:36:23] RECOVERY Free ram is now: OK on ve-nodejs i-00000245 output: OK: 87% free memory [10:36:23] RECOVERY Disk Space is now: OK on ve-nodejs i-00000245 output: DISK OK [10:36:23] RECOVERY dpkg-check is now: OK on ve-nodejs i-00000245 output: All packages OK [10:36:56] RECOVERY Total Processes is now: OK on ve-nodejs i-00000245 output: PROCS OK: 80 processes [10:37:43] RECOVERY Current Users is now: OK on ve-nodejs i-00000245 output: USERS OK - 0 users currently logged in [10:37:43] RECOVERY Current Load is now: OK on ve-nodejs i-00000245 output: OK - load average: 0.18, 2.23, 1.79 [10:49:19] PROBLEM Free ram is now: WARNING on incubator-bot2 i-00000252 output: Warning: 19% free memory [10:51:29] RECOVERY Current Load is now: OK on labs-nfs1 i-0000005d output: OK - load average: 0.89, 1.91, 4.94 [11:21:10] PROBLEM Free ram is now: WARNING on incubator-bot1 i-00000251 output: Warning: 19% free memory [14:34:43] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d output: DISK WARNING - free space: /export 1014 MB (5% inode=80%): /home/SAVE 1014 MB (5% inode=80%): [14:52:33] Is faidon ever in here? [15:00:04] hexmode: is nick is paravoid [15:00:34] hexmode: both of us are back from SF so jet lagged / not really there I guess [15:00:46] hi hashar [15:01:00] I woke up at 4pm, aka 1 hour ago :-( [15:01:24] hashar: thanks.... sent paravoid an email :) [15:02:01] hashar: just out of curiousity (and to see if I can suck you into my project) how much do you know about Debian packaging? [15:02:15] wrote a bunch of them years ago ;-D [15:03:12] hexmode: you probably want to ask ops people [15:03:37] hashar: when I did, was told to talk to paravoid :) [15:04:08] he is part from the Debian community, so he must be the new dpkg expert [15:04:39] hashar: fwiw, I already have a good start on a bugzilla package, but I need a co-maintainer and someone to check over my work [15:05:00] don't ubuntu ship one already? [15:06:08] apparently none :-D [15:06:27] anyway, going to hunt for some food, clean house, grab daughter etc.. [15:06:32] will be back later [15:28:49] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 19% free memory [16:45:20] PROBLEM Puppet freshness is now: CRITICAL on deployment-syslog i-00000269 output: Puppet has not run in last 20 hours [17:28:24] Change on 12mediawiki a page OAuth was modified, changed by RobLa-WMF link https://www.mediawiki.org/w/index.php?diff=537588 edit summary: Adding activity page template [17:30:20] Change on 12mediawiki a page OAuth was modified, changed by RobLa-WMF link https://www.mediawiki.org/w/index.php?diff=537590 edit summary: trailing noinclude [17:32:05] Change on 12mediawiki a page OAuth was modified, changed by RobLa-WMF link https://www.mediawiki.org/w/index.php?diff=537591 edit summary: removing errant noinclude [17:40:28] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [17:44:19] RECOVERY Disk Space is now: OK on deployment-feed i-00000118 output: DISK OK [17:52:19] PROBLEM Disk Space is now: WARNING on deployment-feed i-00000118 output: DISK WARNING - free space: / 74 MB (5% inode=40%): [18:02:26] PROBLEM Disk Space is now: CRITICAL on nagios 127.0.0.1 output: DISK CRITICAL - free space: /home/dzahn 730 MB (4% inode=80%): [18:09:33] RECOVERY Free ram is now: OK on incubator-bot2 i-00000252 output: OK: 27% free memory [18:20:25] PROBLEM Current Load is now: WARNING on mobile-enwp i-000000ce output: WARNING - load average: 5.37, 9.99, 5.13 [18:25:25] RECOVERY Current Load is now: OK on mobile-enwp i-000000ce output: OK - load average: 0.35, 3.95, 3.84 [18:41:49] I reallly should update mediawiki hmm [18:42:20] mediawiki class on labs that as "ensure => latest"? [18:51:59] heh [18:52:02] that sounds dangerous :) [18:52:08] unless puppet is going to run update.php [18:52:55] ensure => latest is just generally a bad idea [18:55:20] ok, well, seperate as mediawiki-latest vs. the stable one. convinced:) [18:56:21] \o/ [18:56:29] OATHAuth is enabled on labconsole :) [18:56:35] for those who want to test latest... [18:56:44] cool Ryan! [18:56:56] Can we uave oauth too :D [18:57:00] I just enabled it for myself [18:57:09] Damianz: I'd love to have it [18:57:25] * Damianz finds a mw developer to harrass [19:17:54] RECOVERY Disk Space is now: OK on deployment-feed i-00000118 output: DISK OK [19:23:16] New patchset: Ryan Lane; "Change alias directory to 1.20wmf2" [operations/puppet] (test) - https://gerrit.wikimedia.org/r/7615 [19:23:31] New review: gerrit2; "Lint check passed." [operations/puppet] (test); V: 1 - https://gerrit.wikimedia.org/r/7615 [19:23:53] New review: Ryan Lane; "(no comment)" [operations/puppet] (test); V: 0 C: 2; - https://gerrit.wikimedia.org/r/7615 [19:23:55] Change merged: Ryan Lane; [operations/puppet] (test) - https://gerrit.wikimedia.org/r/7615 [19:27:14] PROBLEM Puppet freshness is now: CRITICAL on nova-ldap1 i-000000df output: Puppet has not run in last 20 hours [19:30:15] !log deployment-prep Fixed IP :-D [19:30:22] Logged the message, Master [19:30:27] !log deployment-prep hashar: fo [19:30:28] Logged the message, Master [19:30:43] !log deployment-prep hashar: Fixed X-Forwarded-For IP not being recognized [19:30:45] Logged the message, Master [20:11:29] PROBLEM dpkg-check is now: CRITICAL on mobile-enwp i-000000ce output: DPKG CRITICAL dpkg reports broken packages [20:12:57] Change on 12mediawiki a page OAuth was modified, changed by RobLa-WMF link https://www.mediawiki.org/w/index.php?diff=537646 edit summary: Some reorganization [20:15:14] PROBLEM Disk Space is now: CRITICAL on labs-nfs1 i-0000005d output: DISK CRITICAL - free space: /export 507 MB (2% inode=80%): /home/SAVE 507 MB (2% inode=80%): [20:16:24] RECOVERY dpkg-check is now: OK on mobile-enwp i-000000ce output: All packages OK [20:19:34] RECOVERY Free ram is now: OK on bots-2 i-0000009c output: OK: 20% free memory [20:27:32] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 19% free memory [20:30:14] Change on 12mediawiki a page OAuth was modified, changed by Jorm (WMF) link https://www.mediawiki.org/w/index.php?diff=537651 edit summary: spurious apostrophes. [20:32:32] RECOVERY Free ram is now: OK on bots-2 i-0000009c output: OK: 20% free memory [20:37:42] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [20:46:00] PROBLEM Disk Space is now: CRITICAL on nagios 127.0.0.1 output: DISK CRITICAL - free space: /home/dzahn 474 MB (2% inode=80%): [20:56:00] RECOVERY Disk Space is now: OK on nagios 127.0.0.1 output: DISK OK [21:00:40] PROBLEM Disk Space is now: WARNING on deployment-feed i-00000118 output: DISK WARNING - free space: / 74 MB (5% inode=40%): [21:43:16] PROBLEM Free ram is now: WARNING on bots-2 i-0000009c output: Warning: 17% free memory [22:06:38] Ryan_Lane: Created instance i-00000271 with image ami-00000026 and hostname i-00000271.pmtpa.wmflabs. [22:14:08] PROBLEM Current Load is now: CRITICAL on mobile-testing i-00000271 output: Connection refused by host [22:14:20] PROBLEM Disk Space is now: CRITICAL on nagios 127.0.0.1 output: DISK CRITICAL - free space: /home/dzahn 314 MB (1% inode=80%): [22:14:48] PROBLEM Current Users is now: CRITICAL on mobile-testing i-00000271 output: Connection refused by host [22:15:28] PROBLEM Disk Space is now: CRITICAL on mobile-testing i-00000271 output: Connection refused by host [22:16:08] PROBLEM Free ram is now: CRITICAL on mobile-testing i-00000271 output: Connection refused by host [22:17:18] PROBLEM Total Processes is now: CRITICAL on mobile-testing i-00000271 output: Connection refused by host [22:18:23] PROBLEM dpkg-check is now: CRITICAL on mobile-testing i-00000271 output: Connection refused by host [22:34:33] PROBLEM host: mobile-testing is DOWN address: i-00000271 CRITICAL - Host Unreachable (i-00000271) [23:04:55] PROBLEM host: mobile-testing is DOWN address: i-00000271 CRITICAL - Host Unreachable (i-00000271) [23:10:45] RECOVERY host: mobile-testing is UP address: i-00000271 PING OK - Packet loss = 0%, RTA = 546.72 ms [23:17:46] RECOVERY Current Users is now: OK on mobile-testing i-00000271 output: USERS OK - 0 users currently logged in [23:18:46] RECOVERY Disk Space is now: OK on mobile-testing i-00000271 output: DISK OK [23:19:16] RECOVERY Free ram is now: OK on mobile-testing i-00000271 output: OK: 97% free memory [23:20:26] RECOVERY Total Processes is now: OK on mobile-testing i-00000271 output: PROCS OK: 128 processes [23:21:16] RECOVERY dpkg-check is now: OK on mobile-testing i-00000271 output: All packages OK [23:22:26] RECOVERY Current Load is now: OK on mobile-testing i-00000271 output: OK - load average: 0.46, 0.56, 0.32 [23:50:13] 05/14/2012 - 23:50:13 - Updating keys for laner at /export/home/mediahandler/laner [23:50:14] 05/14/2012 - 23:50:14 - Updating keys for laner at /export/home/sugarcrm/laner [23:50:15] 05/14/2012 - 23:50:15 - Updating keys for laner at /export/home/ganglia/laner [23:50:15] 05/14/2012 - 23:50:15 - Updating keys for laner at /export/home/gerrit/laner [23:50:17] 05/14/2012 - 23:50:16 - Updating keys for laner at /export/home/deployment-prep/laner [23:50:18] 05/14/2012 - 23:50:17 - Updating keys for laner at /export/home/mobile-stats/laner [23:50:19] 05/14/2012 - 23:50:18 - Updating keys for laner at /export/home/nginx/laner [23:50:21] 05/14/2012 - 23:50:19 - Updating keys for laner at /export/home/patchtest/laner [23:50:21] 05/14/2012 - 23:50:19 - Updating keys for laner at /export/home/selenium/laner [23:50:21] 05/14/2012 - 23:50:20 - Updating keys for laner at /export/home/deployment-prepbackup/laner [23:50:21] 05/14/2012 - 23:50:20 - Updating keys for laner at /export/home/feeds/laner [23:50:21] 05/14/2012 - 23:50:21 - Updating keys for laner at /export/home/wikisource-dev/laner [23:50:21] 05/14/2012 - 23:50:21 - Updating keys for laner at /export/home/jenkins/laner [23:50:22] 05/14/2012 - 23:50:22 - Updating keys for laner at /export/home/phabricator/laner [23:50:22] 05/14/2012 - 23:50:22 - Updating keys for laner at /export/home/mail/laner [23:50:23] 05/14/2012 - 23:50:23 - Updating keys for laner at /export/home/outreach/laner [23:50:23] 05/14/2012 - 23:50:23 - Updating keys for laner at /export/home/translation-memory/laner [23:50:25] 05/14/2012 - 23:50:24 - Updating keys for laner at /export/home/puppet-cleanup/laner [23:50:25] 05/14/2012 - 23:50:25 - Updating keys for laner at /export/home/lvs-labs/laner [23:50:26] 05/14/2012 - 23:50:25 - Updating keys for laner at /export/home/editor-engagement/laner [23:50:26] 05/14/2012 - 23:50:26 - Updating keys for laner at /export/home/testing/laner [23:50:27] 05/14/2012 - 23:50:27 - Updating keys for laner at /export/home/simplewiki/laner [23:50:28] 05/14/2012 - 23:50:27 - Updating keys for laner at /export/home/otrs/laner [23:50:29] 05/14/2012 - 23:50:28 - Updating keys for laner at /export/home/wikistream/laner [23:50:30] 05/14/2012 - 23:50:30 - Updating keys for laner at /export/home/maps/laner [23:50:31] 05/14/2012 - 23:50:31 - Updating keys for laner at /export/home/publicdata/laner [23:50:33] 05/14/2012 - 23:50:32 - Updating keys for laner at /export/home/shop-analytics/laner [23:50:33] 05/14/2012 - 23:50:33 - Updating keys for laner at /export/home/category-sorting/laner [23:50:34] 05/14/2012 - 23:50:34 - Updating keys for laner at /export/home/wikidata-dev/laner [23:50:37] 05/14/2012 - 23:50:36 - Updating keys for laner at /export/home/opengrok/laner [23:50:37] 05/14/2012 - 23:50:37 - Updating keys for laner at /export/home/gareth/laner [23:50:38] 05/14/2012 - 23:50:37 - Updating keys for laner at /export/home/analytics/laner [23:50:38] 05/14/2012 - 23:50:38 - Updating keys for laner at /export/home/syslog-collection/laner [23:50:39] 05/14/2012 - 23:50:39 - Updating keys for laner at /export/home/mailman/laner [23:50:40] 05/14/2012 - 23:50:40 - Updating keys for laner at /export/home/demo/laner [23:50:41] 05/14/2012 - 23:50:41 - Updating keys for laner at /export/home/wiki-migration/laner [23:51:02] 05/14/2012 - 23:51:02 - Updating keys for laner at /export/home/testlabs/laner [23:51:03] 05/14/2012 - 23:51:02 - Updating keys for laner at /export/home/juju/laner [23:51:04] 05/14/2012 - 23:51:03 - Updating keys for laner at /export/home/etherpad/laner [23:51:05] 05/14/2012 - 23:51:05 - Updating keys for laner at /export/home/gluster/laner [23:51:06] 05/14/2012 - 23:51:05 - Updating keys for laner at /export/home/mobile-sms/laner [23:51:06] 05/14/2012 - 23:51:06 - Updating keys for laner at /export/home/pediapress/laner [23:51:07] 05/14/2012 - 23:51:07 - Updating keys for laner at /export/home/openstack/laner [23:51:07] 05/14/2012 - 23:51:07 - Updating keys for laner at /export/home/puppet/laner [23:51:08] 05/14/2012 - 23:51:08 - Updating keys for laner at /export/home/membase/laner [23:51:09] 05/14/2012 - 23:51:08 - Updating keys for laner at /export/home/turnkey-mediawiki/laner [23:51:09] 05/14/2012 - 23:51:09 - Updating keys for laner at /export/home/testswarm/laner [23:51:10] 05/14/2012 - 23:51:10 - Updating keys for laner at /export/home/bots/laner [23:51:11] 05/14/2012 - 23:51:11 - Updating keys for laner at /export/home/nagios/laner [23:51:11] 05/14/2012 - 23:51:11 - Updating keys for laner at /export/home/reportcard/laner [23:51:12] 05/14/2012 - 23:51:11 - Updating keys for laner at /export/home/mobile/laner [23:51:13] 05/14/2012 - 23:51:12 - Updating keys for laner at /export/home/storage/laner [23:51:14] 05/14/2012 - 23:51:14 - Updating keys for laner at /export/home/bastion/laner