[00:06:02] Russell Blau * Re: [Toolserver-l] Anoter SGE question [00:12:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 394726 MB (7% inode=39%): [00:19:14] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43077 MB (10% inode=99%): [00:23:54] /sql on cassia is WARNING: DISK WARNING - free space: /sql 83215 MB (7% inode=99%): [00:26:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:27:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:34:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:38:14] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 64102 MB (15% inode=99%): [00:38:33] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:40:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:41:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:43:24] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:08:23] Sun Grid Engine execd on willow is WARNING: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.668457/2.3, alarm hl:np_load_long=1.255859/2.5, alarm hl:cpu=99.600000/98, alarm hl:mem_free=428.000000M/200M, alarm hl:tmp_free=43247M/100M, alarm hl:available=1/0 [01:12:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 394762 MB (7% inode=39%): [01:26:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:27:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:29:26] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [01:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:40:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:41:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:00:53] /sql on thyme is WARNING: DISK WARNING - free space: /sql 189885 MB (19% inode=99%): [02:12:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 395658 MB (7% inode=39%): [02:26:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.266113/1.95, alarm hl:tmp_free=43021M/100M, alarm hl:np_load_avg=1.265137/2.0, alarm hl:mem_free=294.000000M/350M, alarm hl:available=1/0 [02:26:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:27:24] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:28:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:38:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.985840/1.95, alarm hl:tmp_free=42987M/100M, alarm hl:np_load_avg=1.116699/2.0, alarm hl:mem_free=323.000000M/350M, alarm hl:available=1/0 [02:40:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:42:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:46:24] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [03:13:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 394694 MB (7% inode=39%): [03:23:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:28:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:28:34] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:35:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:40:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:42:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:43:33] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:46:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.230957/1.95, alarm hl:tmp_free=42794M/100M, alarm hl:np_load_avg=1.164062/2.0, alarm hl:mem_free=267.000000M/350M, alarm hl:available=1/0 [03:52:24] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [03:57:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.132812/1.95, alarm hl:tmp_free=42765M/100M, alarm hl:np_load_avg=1.069336/2.0, alarm hl:mem_free=305.000000M/350M, alarm hl:available=1/0 [04:08:25] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2047 MB (17% inode=99%): [04:13:25] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:13:54] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 396275 MB (7% inode=39%): [04:14:04] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:27:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:27:23] /tmp on ortelius is OK: DISK OK - free space: /tmp 3501 MB (27% inode=99%): [04:28:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:30:10] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7402&oldid=7384&rcid=10000 * 91.198.174.202 * (+1) (updated page) [04:36:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:40:23] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2432 MB (20% inode=99%): [04:40:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:42:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.134277/1.95, alarm hl:tmp_free=42642M/100M, alarm hl:np_load_avg=1.219726/2.0, alarm hl:mem_free=219.000000M/350M, alarm hl:available=1/0 [04:42:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:47:34] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:50:24] /tmp on ortelius is OK: DISK OK - free space: /tmp 2668 MB (22% inode=99%): [05:02:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.333496/1.95, alarm hl:tmp_free=42589M/100M, alarm hl:np_load_avg=1.232910/2.0, alarm hl:mem_free=296.000000M/350M, alarm hl:available=1/0 [05:13:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 396218 MB (7% inode=39%): [05:14:54] Load avg. on willow is WARNING: WARNING - load average: 16.32, 12.81, 10.68 [05:15:54] Load avg. on willow is OK: OK - load average: 13.39, 12.63, 10.75 [05:27:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:28:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:36:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:41:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:42:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:00:42] 3(created) [DBQ-187] Get all namespaces of all Wikipedias; Database Queries; Trivial Task <10https://jira.toolserver.org/browse/DBQ-187> (Avicennasis ) [06:02:32] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:07:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.252441/1.95, alarm hl:tmp_free=42415M/100M, alarm hl:np_load_avg=1.384277/2.0, alarm hl:mem_free=265.000000M/350M, alarm hl:available=1/0 [06:14:14] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 396044 MB (7% inode=39%): [06:19:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:27:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:28:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:36:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:41:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:42:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:57:14] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1890.000000 [07:07:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.841797/1.95, alarm hl:tmp_free=42251M/100M, alarm hl:np_load_avg=2.099121/2.0, alarm hl:mem_free=342.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.841797/2.3, alarm hl:np_load_long=1.608399/2.5, alarm hl:cpu=99.100000/98, alarm hl:mem_free=342.000000M/200M, al [07:08:12] Load avg. on willow is WARNING: WARNING - load average: 17.56, 16.14, 12.78 [07:13:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [07:14:12] Load avg. on willow is OK: OK - load average: 9.41, 15.00, 13.88 [07:15:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 395968 MB (7% inode=39%): [07:27:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:28:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:31:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.579590/1.95, alarm hl:tmp_free=42189M/100M, alarm hl:np_load_avg=1.995117/2.0, alarm hl:mem_free=386.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.579590/2.3, alarm hl:np_load_long=1.725586/2.5, alarm hl:cpu=97.700000/98, alarm hl:mem_free=386.000000M/200M, al [07:32:14] Load avg. on willow is WARNING: WARNING - load average: 15.67, 15.23, 13.63 [07:32:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [07:35:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3643.000000 [07:37:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:38:12] Load avg. on willow is OK: OK - load average: 12.14, 14.86, 14.10 [07:41:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:42:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:52:42] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.800781/1.00, alarm hl:np_load_long=0.361816/1.50, alarm hl:mem_free=526.000000M/600M, alarm hl:tmp_free=14307M/100M, alarm hl:available=1/0 [07:53:41] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [07:53:41] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:56:41] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.707519/1.00, alarm hl:np_load_long=0.422851/1.50, alarm hl:mem_free=545.000000M/600M, alarm hl:tmp_free=14300M/100M, alarm hl:available=1/0 [07:58:13] Load avg. on willow is WARNING: WARNING - load average: 23.06, 15.37, 13.38 [07:59:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.514648/1.95, alarm hl:tmp_free=42123M/100M, alarm hl:np_load_avg=2.055664/2.0, alarm hl:mem_free=446.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.514648/2.3, alarm hl:np_load_long=1.746094/2.5, alarm hl:cpu=99.700000/98, alarm hl:mem_free=446.000000M/200M, al [08:01:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [08:02:13] Load avg. on willow is OK: OK - load average: 11.02, 14.68, 13.71 [08:03:41] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:10:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.063965/1.95, alarm hl:tmp_free=42099M/100M, alarm hl:np_load_avg=1.206055/2.0, alarm hl:mem_free=254.000000M/350M, alarm hl:available=1/0 [08:15:13] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 395853 MB (7% inode=39%): [08:28:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:31:51] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.330566/1.10, alarm hl:np_load_long=0.271973/1.55, alarm hl:mem_free=337.000000M/500M, alarm hl:tmp_free=14269M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.330566/1.00, alarm hl:np_load_long=0.271973/1.50, alarm hl:mem_free=337.000000M/600M, alarm hl:tmp_free= [08:32:50] [[Special:Log/newusers]] create 10 * Cactus26 * (New user account) [08:33:51] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [08:35:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6724.000000 [08:37:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:37:44] 3(created) [ACCAPP-532] Account Approval Cactus26; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-532> (Cactus26) [08:39:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.133789/1.95, alarm hl:tmp_free=42028M/100M, alarm hl:np_load_avg=1.189453/2.0, alarm hl:mem_free=321.000000M/350M, alarm hl:available=1/0 [08:41:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:41:51] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.268555/1.00, alarm hl:np_load_long=0.273438/1.50, alarm hl:mem_free=535.000000M/600M, alarm hl:tmp_free=14257M/100M, alarm hl:available=1/0 [08:42:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:49:51] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [08:52:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.982910/1.95, alarm hl:tmp_free=41996M/100M, alarm hl:np_load_avg=1.166992/2.0, alarm hl:mem_free=345.000000M/350M, alarm hl:available=1/0 [09:15:15] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 394097 MB (7% inode=39%): [09:28:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:28:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:35:23] s4 replag on cassia is CRITICAL: (Service Check Timed Out) [09:36:32] s5 replag on cassia is CRITICAL: (Service Check Timed Out) [09:36:43] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 229.000000 [09:37:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:40:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.938965/1.95, alarm hl:tmp_free=41880M/100M, alarm hl:np_load_avg=0.899902/2.0, alarm hl:mem_free=185.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.938965/2.3, alarm hl:np_load_long=0.983887/2.5, alarm hl:cpu=80.200000/98, alarm hl:mem_free=185.000000M/200M, al [09:41:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:42:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:50:03] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:03] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:14] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:14] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:14] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:15] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:15] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:22] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:22] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:23] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:23] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:23] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:23] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:23] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:43] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:50:44] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [09:50:44] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 2039 MB (99% inode=99%): [09:50:44] Load avg. on z-dat-s7-a is OK: OK - load average: 1.48, 1.46, 1.71 [09:50:45] Load avg. on z-dat-s3-a is OK: OK - load average: 1.55, 1.48, 1.71 [09:50:45] SMF on z-dat-s7-a is OK: OK - all services online [09:50:45] SMF on z-dat-s4-a is OK: OK - all services online [09:50:54] SMF on z-dat-s6-a is OK: OK - all services online [09:50:54] SMF on z-dat-s3-a is OK: OK - all services online [09:51:14] NTP on hyacinth is OK: NTP OK: Offset -0.003002 secs [09:51:14] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:51:15] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:51:15] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:51:15] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:58:53] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [10:09:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.919434/1.95, alarm hl:tmp_free=41813M/100M, alarm hl:np_load_avg=1.019043/2.0, alarm hl:mem_free=312.000000M/350M, alarm hl:available=1/0 [10:11:54] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [10:16:15] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 394664 MB (7% inode=39%): [10:28:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:29:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:35:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11695.000000 [10:37:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:42:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:43:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:17:15] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 392640 MB (7% inode=39%): [11:28:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:29:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:34:13] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 42.16, 21.43, 11.15 [11:35:14] Load avg. on ortelius is WARNING: WARNING - load average: 23.72, 20.32, 11.44 [11:36:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13992.000000 [11:38:14] Load avg. on ortelius is OK: OK - load average: 6.21, 13.65, 10.37 [11:38:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:41:22] Load avg. on adenia is WARNING: WARNING - load average: 18.68, 12.32, 6.96 [11:42:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:43:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:44:22] Load avg. on adenia is OK: OK - load average: 14.40, 13.57, 8.45 [12:05:47] 3(created) [ACCAPP-533] Accountrequest for no.wiki category watch tool; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-533> [12:13:04] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 80 MB (4% inode=99%): [12:16:03] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1116 MB (40% inode=99%): [12:17:13] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 392504 MB (7% inode=39%): [12:24:14] /sql on cassia is WARNING: DISK WARNING - free space: /sql 78975 MB (6% inode=99%): [12:28:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:29:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:36:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16206.000000 [12:38:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:41:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43572 MB (10% inode=99%): [12:42:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:43:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:00:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 62053 MB (15% inode=99%): [13:17:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 392355 MB (7% inode=39%): [13:22:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.854981/1.95, alarm hl:tmp_free=41357M/100M, alarm hl:np_load_avg=0.895508/2.0, alarm hl:mem_free=286.000000M/350M, alarm hl:available=1/0 [13:27:14] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [13:28:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:29:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:33:15] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.840332/1.95, alarm hl:tmp_free=41333M/100M, alarm hl:np_load_avg=0.868164/2.0, alarm hl:mem_free=294.000000M/350M, alarm hl:available=1/0 [13:36:44] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17347.000000 [13:39:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:42:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:43:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:47:14] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [14:01:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 190655 MB (19% inode=99%): [14:17:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 392073 MB (7% inode=39%): [14:28:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:30:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:36:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15783.000000 [14:39:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:41:15] Hello all [14:43:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:44:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:51:48] DaBPunkt: just a quick note, I noticed that user-store isnt mounted on yarrow [14:52:01] ah ok. I will fix it [15:13:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43363 MB (10% inode=99%): [15:15:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.919922/1.95, alarm hl:tmp_free=41108M/100M, alarm hl:np_load_avg=0.999023/2.0, alarm hl:mem_free=319.000000M/350M, alarm hl:available=1/0 [15:17:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391954 MB (7% inode=39%): [15:18:13] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [15:28:43] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:28:43] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 65396 MB (16% inode=99%): [15:28:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:30:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:37:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14568.000000 [15:39:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:43:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:44:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:51:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.736816/1.95, alarm hl:tmp_free=41035M/100M, alarm hl:np_load_avg=0.673340/2.0, alarm hl:mem_free=272.000000M/350M, alarm hl:available=1/0 [15:53:26] [[Debian/Installation]] 10https://wiki.toolserver.org/w/index.php?diff=7403&oldid=7288&rcid=10002 * Dab * (+38) (/* Pre-puppet */ ) [15:54:12] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [15:58:56] [[Debian/Groups]] 10https://wiki.toolserver.org/w/index.php?diff=7404&oldid=7293&rcid=10003 * Dab * (+33) (/* 300-399 */ ) [16:00:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.832031/1.95, alarm hl:tmp_free=41018M/100M, alarm hl:np_load_avg=0.791016/2.0, alarm hl:mem_free=317.000000M/350M, alarm hl:available=1/0 [16:16:32] [[Debian/Groups]] M 10https://wiki.toolserver.org/w/index.php?diff=7405&oldid=7404&rcid=10004 * Dab * (+42) (/* 200-299 */ ) [16:17:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 391961 MB (7% inode=39%): [16:25:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.963379/1.95, alarm hl:tmp_free=40960M/100M, alarm hl:np_load_avg=0.854492/2.0, alarm hl:mem_free=141.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.963379/2.3, alarm hl:np_load_long=0.820312/2.5, alarm hl:cpu=91.300000/98, alarm hl:mem_free=141.000000M/200M, al [16:28:43] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:29:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:30:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:30:23] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [16:33:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.919434/1.95, alarm hl:tmp_free=40943M/100M, alarm hl:np_load_avg=0.895996/2.0, alarm hl:mem_free=130.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.919434/2.3, alarm hl:np_load_long=0.860352/2.5, alarm hl:cpu=80.800000/98, alarm hl:mem_free=130.000000M/200M, al [16:37:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13988.000000 [16:38:13] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:39:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:43:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:44:13] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [16:44:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:56:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.897949/1.95, alarm hl:tmp_free=40893M/100M, alarm hl:np_load_avg=0.778809/2.0, alarm hl:mem_free=305.000000M/350M, alarm hl:available=1/0 [17:17:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 390159 MB (7% inode=39%): [17:29:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:30:23] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [17:30:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:34:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.604492/1.95, alarm hl:tmp_free=40791M/100M, alarm hl:np_load_avg=0.649414/2.0, alarm hl:mem_free=266.000000M/350M, alarm hl:available=1/0 [17:37:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14928.000000 [17:39:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:42:23] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [17:43:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:44:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:02:02] [[Code snippets]] ! 10https://wiki.toolserver.org/w/index.php?diff=7406&oldid=7002&rcid=10005 * 182.177.140.114 * (+78) (/* Complete IPv6 address formats */ ) [18:09:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 127229 MB (20% inode=99%): [18:17:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 380823 MB (7% inode=38%): [18:27:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.663574/1.95, alarm hl:tmp_free=40690M/100M, alarm hl:np_load_avg=0.650879/2.0, alarm hl:mem_free=300.000000M/350M, alarm hl:available=1/0 [18:29:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:30:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:30:24] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:38:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.607910/1.95, alarm hl:tmp_free=40666M/100M, alarm hl:np_load_avg=0.604004/2.0, alarm hl:mem_free=251.000000M/350M, alarm hl:available=1/0 [18:38:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16079.000000 [18:39:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:43:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:44:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:09:33] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126929 MB (20% inode=99%): [19:17:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 380714 MB (7% inode=38%): [19:29:14] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:30:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:39:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17005.000000 [19:39:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:44:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:55:43] 3(created) [MNT-1251] Changed yarrow's MGMT-password; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1251> (DaB.) [19:55:46] 3(resolved) [MNT-1251] Changed yarrow's MGMT-password <10https://jira.toolserver.org/browse/MNT-1251> (DaB.) [20:00:55] @replag [20:00:55] Akoopal: s1-rr-a: 32s [+0.00 s/s]; s2-user: 1w 23h 13m 36s [+0.39 s/s]; s2-user-c: 4h 47m 59s [+0.26 s/s]; s3-rr-a: 15s [-0.00 s/s]; s3-user: 15s [-0.00 s/s]; s5-user-c: 4h 47m 59s [+0.26 s/s]; s6-rr-a: 14m 4s [-0.00 s/s]; s6-user: 14m 4s [-0.00 s/s] [20:01:23] the replag graphs for S2 don't seem to be updated? [20:09:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126663 MB (20% inode=99%): [20:17:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 380568 MB (7% inode=38%): [20:18:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.163574/1.95, alarm hl:tmp_free=40446M/100M, alarm hl:np_load_avg=0.859863/2.0, alarm hl:mem_free=284.000000M/350M, alarm hl:available=1/0 [20:22:54] DaBPunkt, did you change yarrow ssh key? [20:23:27] Platonides: yes and no. It is reinstalling at the moment. The old key will be back in few minutes [20:23:44] ah [20:24:09] I was a bit confused seeing it wasn't listed at https://fingerprints.toolserver.org/ nor /etc/opt/ts/ssh/ssh_known_hosts [20:30:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:31:24] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:35:07] Platonides: it's now generating the locales, but login should be possible [20:35:25] APT on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:35:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.458008/1.95, alarm hl:tmp_free=40436M/100M, alarm hl:np_load_avg=0.529785/2.0, alarm hl:mem_free=201.000000M/350M, alarm hl:available=1/0 [20:35:25] NTP on yarrow is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset 0.007631 secs [20:35:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:35:32] Sensors on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:35:43] Environment IPMI on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [20:37:36] DaBPunkt, it is [20:37:53] up 6 min, [20:38:42] it is showing password-interactive as a possible login method [20:39:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18477.000000 [20:39:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:42:48] were you able to fix that odd hostkeys issue? [20:43:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:44:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:44:45] 3(commented) [TS-1402] Cronie jobs intermittently failing to run <10https://jira.toolserver.org/browse/TS-1402> (drtrigon) [20:46:40] Platonides: spent the greater part of today on it. Couldn't find the problem yet. the problem is somewhere on yarrow. Login from everywhere to yarrow works (even from nightshade), but login from yarrow doesn't work [20:47:24] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:48:23] NTP on yarrow is OK: NTP OK: Offset 0.048866 secs [20:50:49] it bet it is something very trivial in the end… [21:01:28] got it! [21:06:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43149 MB (10% inode=99%): [21:09:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126398 MB (20% inode=99%): [21:17:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 389412 MB (7% inode=39%): [21:22:48] [[Debian/Installation]] 10https://wiki.toolserver.org/w/index.php?diff=7407&oldid=7403&rcid=10006 * Dab * (-108) (/* OS-Installation */ is now done automaticaly) [21:26:02] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 56988 MB (14% inode=99%): [21:27:54] DaBPunkt, what was the problem? [21:28:02] I still see it fail, though [21:28:27] it still fails for you? [21:28:55] a ssh from yarrow to willow? yes [21:29:20] oh, that way [21:29:24] wait a moment [21:29:35] willow to yarrow worked before [21:29:58] not that I really need it [21:30:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:31:26] the problem was that yarrow full name in shosts.equiv was wrong [21:32:37] fixed for willow, ortelius and wolfsbane too [21:34:17] yep [21:35:35] Sensors on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:35:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:35:43] Environment IPMI on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:36:23] APT on yarrow is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:37:39] I don't see shosts.equiv in /etc, which is a bit odd [21:38:00] * Platonides realises it's in /etc/ssh/ [21:40:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18904.000000 [21:40:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:43:03] Angelika Adam * [Toolserver-l] Toolserver Workshop at Wikipedia Academy 2012 in Berlin [21:43:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:44:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:44:49] MySQL trouble with VARCHAR: If a string has < 255 Unicode points and > 255 UTF-8 bytes it'll silently truncate leaving a possibly corrupt string [21:48:43] Environment IPMI on yarrow is OK: ok: temperature ok fan ok voltage ok chassis ok [21:49:23] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:49:35] Sensors on yarrow is OK: sensor ok [21:55:38] wow, I had a stack of local -> willow -> yarrow -> willow -> yarrow -> willow [22:00:45] Platonides: put in a few wolfsbanes ;) [22:01:27] * Danny_B|backup sends bribes to root to make the switch to apache [22:02:33] Danny_B|backup: should be done in July [22:07:09] come on july, come on, pass away quickly ;-) [22:09:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126149 MB (20% inode=99%): [22:17:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 380206 MB (7% inode=38%): [22:24:01] nacht ts [22:30:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:35:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:40:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18804.000000 [22:41:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:42:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.639648/1.95, alarm hl:tmp_free=40205M/100M, alarm hl:np_load_avg=0.612305/2.0, alarm hl:mem_free=218.000000M/350M, alarm hl:available=1/0 [22:43:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:45:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:47:33] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:55:03] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:03] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:13] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:55:23] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:23] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:34] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:34] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:34] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:45] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:45] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:45] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:54] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:54] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:03] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:03] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:56:04] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:04] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:04] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:04] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:04] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:04] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:13] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:13] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:13] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:13] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:56:23] / on z-dat-s4-a is OK: DISK OK - free space: / 8215 MB (27% inode=85%): [22:56:23] SMF on z-dat-s7-a is OK: OK - all services online [22:56:23] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 1836 MB (99% inode=99%): [22:56:23] SMF on z-dat-s4-a is OK: OK - all services online [22:56:24] Load avg. on z-dat-s7-a is OK: OK - load average: 0.56, 1.93, 2.62 [22:56:24] Load avg. on z-dat-s3-a is OK: OK - load average: 0.57, 1.89, 2.60 [22:56:24] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 60561 MB (14% inode=99%): [22:56:24] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 1834 MB (99% inode=99%): [22:56:24] SMF on z-dat-s3-a is OK: OK - all services online [22:56:33] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:56:33] / on z-dat-s3-a is OK: DISK OK - free space: / 8211 MB (27% inode=85%): [22:56:33] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1819 MB (99% inode=99%): [22:56:34] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 116904 MB (12% inode=98%): [22:56:34] / on z-dat-s7-a is OK: DISK OK - free space: / 8211 MB (27% inode=85%): [22:56:34] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 1820 MB (99% inode=99%): [22:56:34] SMF on z-dat-s6-a is OK: OK - all services online [22:56:43] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 116902 MB (12% inode=98%): [22:56:43] Load avg. on z-dat-s6-a is OK: OK - load average: 1.80, 2.07, 2.64 [22:56:43] Load avg. on z-dat-s4-a is OK: OK - load average: 1.84, 2.07, 2.64 [22:56:43] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 74794 MB (18% inode=99%): [22:56:44] / on z-dat-s6-a is OK: DISK OK - free space: / 8211 MB (27% inode=85%): [22:56:53] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:56:53] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:56:54] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:56:54] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:57:03] SMTP on z-dat-s4-a is OK: SMTP OK - 0.003 sec. response time [23:09:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125977 MB (20% inode=99%): [23:17:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 379123 MB (7% inode=38%): [23:30:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:35:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:40:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19140.000000 [23:41:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:43:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:45:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default