[00:03:11] Load avg. on willow is WARNING: WARNING - load average: 17.98, 15.11, 11.17 [00:03:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.250000/1.95, alarm hl:np_load_avg=1.897461/2.0, alarm hl:mem_free=137.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.250000/2.3, alarm hl:np_load_long=1.402344/2.5, alarm hl:cpu=98.800000/98, alarm hl:mem_free=137.000000M/150M, alarm hl:available=1/0 [00:07:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3619.000000 [00:11:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:11:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:23:10] Load avg. on willow is OK: OK - load average: 13.59, 14.73, 14.52 [00:24:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386567 MB (7% inode=39%): [00:27:21] Load avg. on willow is WARNING: WARNING - load average: 17.43, 16.09, 15.09 [00:34:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:44:39] [[Special:Log/newusers]] create 10 * Karin726g * (New user account) [00:52:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:03:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.032227/1.95, alarm hl:np_load_avg=2.195801/2.0, alarm hl:mem_free=254.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.032227/2.3, alarm hl:np_load_long=2.107422/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=254.000000M/150M, alarm hl:available=1/0 [01:07:41] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6388.000000 [01:11:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:11:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:24:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 388658 MB (7% inode=39%): [01:28:11] Load avg. on willow is WARNING: WARNING - load average: 17.95, 18.05, 17.98 [01:34:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:52:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:53:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:57:11] Load avg. on willow is OK: OK - load average: 11.20, 12.73, 14.88 [02:02:10] Load avg. on willow is WARNING: WARNING - load average: 15.83, 14.77, 15.13 [02:02:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.985840/1.95, alarm hl:np_load_avg=1.858399/2.0, alarm hl:mem_free=246.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.985840/2.3, alarm hl:np_load_long=1.895020/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=246.000000M/150M, alarm hl:available=1/0 [02:07:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9127.000000 [02:11:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:11:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:24:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 387581 MB (7% inode=39%): [02:34:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:43:11] Load avg. on willow is OK: OK - load average: 13.21, 13.80, 14.95 [02:52:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:52:42] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:02:11] Load avg. on willow is WARNING: WARNING - load average: 17.84, 15.80, 14.88 [03:02:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.204590/1.95, alarm hl:np_load_avg=1.981934/2.0, alarm hl:mem_free=263.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.204590/2.3, alarm hl:np_load_long=1.863770/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=263.000000M/150M, alarm hl:available=1/0 [03:08:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11571.000000 [03:11:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:12:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:13:21] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.218750/1.10, alarm hl:np_load_long=0.796875/1.55, alarm hl:mem_free=18144.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.218750/1.00, alarm hl:np_load_long=0.796875/1.50, alarm hl:mem_free=18144.000000M/600M, alarm hl:available=1/0 [03:15:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:25:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386339 MB (7% inode=39%): [03:26:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:29:10] Load avg. on willow is OK: OK - load average: 12.54, 13.89, 14.94 [03:32:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:34:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:37:11] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:37:42] Sun Grid Engine execd on willow is WARNING: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.684082/2.3, alarm hl:np_load_long=1.782715/2.5, alarm hl:cpu=99.900000/98, alarm hl:mem_free=561.000000M/150M, alarm hl:available=1/0 [03:38:42] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:52:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:52:10] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:03:12] Load avg. on willow is WARNING: WARNING - load average: 17.56, 15.56, 14.10 [04:09:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14352.000000 [04:11:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:12:51] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:14:52] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:14:52] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:14:52] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:14:52] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:12] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:12] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:15:12] Load avg. on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:15:13] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:15:13] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:15:13] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:15:21] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:46] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:49] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:49] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:49] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:49] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:49] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:49] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:49] SMTP on hyacinth is OK: SMTP OK - 1.423 sec. response time [04:17:49] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:49] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:49] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:49] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:49] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:49] SMF on z-dat-s7-a is OK: OK - all services online [04:17:49] Load avg. on z-dat-s4-a is OK: OK - load average: 0.43, 2.07, 2.98 [04:17:49] Load avg. on hyacinth is OK: OK - load average: 0.43, 2.07, 2.98 [04:17:49] / on hyacinth is OK: DISK OK - free space: / 8432 MB (28% inode=85%): [04:17:49] Load avg. on z-dat-s3-a is OK: OK - load average: 0.43, 2.07, 2.98 [04:17:49] SMF on z-dat-s3-a is OK: OK - all services online [04:17:49] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 1865 MB (99% inode=99%): [04:17:49] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:17:49] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:17:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.757324/1.95, alarm hl:np_load_avg=2.057129/2.0, alarm hl:mem_free=86.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.757324/2.3, alarm hl:np_load_long=1.961426/2.5, alarm hl:cpu=92.300000/98, alarm hl:mem_free=86.000000M/150M, alarm hl:available=1/0 [04:17:49] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:17:49] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [04:17:49] SMTP on z-dat-s3-a is OK: SMTP OK - 0.006 sec. response time [04:17:49] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 171294 MB (17% inode=99%): [04:17:49] / on z-dat-s3-a is OK: DISK OK - free space: / 8432 MB (28% inode=85%): [04:17:49] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:17:49] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 101220 MB (25% inode=99%): [04:17:49] /tmp on hyacinth is OK: DISK OK - free space: /tmp 2034 MB (99% inode=99%): [04:17:49] Load avg. on z-dat-s7-a is OK: OK - load average: 1.39, 2.17, 2.99 [04:17:49] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:17:49] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:17:49] Load avg. on z-dat-s6-a is OK: OK - load average: 1.40, 2.17, 2.99 [04:17:49] / on z-dat-s7-a is OK: DISK OK - free space: / 8432 MB (28% inode=85%): [04:25:10] Load avg. on willow is OK: OK - load average: 14.20, 14.30, 14.99 [04:25:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386139 MB (7% inode=39%): [04:29:11] Load avg. on willow is WARNING: WARNING - load average: 16.17, 15.01, 15.11 [04:30:12] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7119&oldid=7025&rcid=9417 * 91.198.174.202 * (+1) (updated page) [04:34:12] Load avg. on willow is OK: OK - load average: 11.78, 14.24, 14.91 [04:34:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:52:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:59:11] Load avg. on willow is WARNING: WARNING - load average: 14.37, 14.83, 15.09 [05:01:52] Load avg. on adenia is WARNING: WARNING - load average: 15.82, 14.43, 12.16 [05:09:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17439.000000 [05:11:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:12:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:16:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.974609/1.95, alarm hl:np_load_avg=2.604492/2.0, alarm hl:mem_free=74.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.974609/2.3, alarm hl:np_load_long=2.213867/2.5, alarm hl:cpu=99.500000/98, alarm hl:mem_free=74.000000M/150M, alarm hl:available=1/0 [05:16:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.14, 21.72, 18.09 [05:17:11] Load avg. on willow is WARNING: WARNING - load average: 19.90, 20.04, 17.71 [05:19:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.116211/1.10, alarm hl:np_load_long=0.737305/1.55, alarm hl:mem_free=16127.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.116211/1.00, alarm hl:np_load_long=0.737305/1.50, alarm hl:mem_free=16127.000000M/600M, alarm hl:available=1/0 [05:25:30] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:26:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386077 MB (7% inode=39%): [05:26:51] Load avg. on adenia is OK: OK - load average: 11.75, 14.31, 14.77 [05:33:53] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:34:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:37:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.782226/1.95, alarm hl:np_load_avg=1.782715/2.0, alarm hl:mem_free=212.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.782226/2.3, alarm hl:np_load_long=1.925293/2.5, alarm hl:cpu=99.800000/98, alarm hl:mem_free=212.000000M/150M, alarm hl:available=1/0 [05:39:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:44:20] Load avg. on willow is OK: OK - load average: 12.59, 14.12, 14.98 [05:47:10] Load avg. on willow is WARNING: WARNING - load average: 15.21, 15.91, 15.62 [05:52:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:54:10] Load avg. on willow is OK: OK - load average: 10.93, 13.58, 14.76 [06:10:51] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20535.000000 [06:12:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:13:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:15:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.555664/1.95, alarm hl:np_load_avg=3.219727/2.0, alarm hl:mem_free=355.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.555664/2.3, alarm hl:np_load_long=2.640137/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=355.000000M/150M, alarm hl:available=1/0 [06:26:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386007 MB (7% inode=39%): [06:34:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:50:21] Load avg. on willow is WARNING: WARNING - load average: 20.34, 17.71, 18.39 [06:52:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:08:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.04, 22.51, 20.05 [07:10:50] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23630.000000 [07:12:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:13:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:15:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.196777/1.95, alarm hl:np_load_avg=2.705078/2.0, alarm hl:mem_free=254.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.196777/2.3, alarm hl:np_load_long=2.591309/2.5, alarm hl:cpu=94.300000/98, alarm hl:mem_free=254.000000M/150M, alarm hl:available=1/0 [07:17:20] Load avg. on willow is WARNING: WARNING - load average: 12.38, 18.44, 19.66 [07:24:02] Load avg. on adenia is WARNING: WARNING - load average: 18.79, 12.28, 8.01 [07:26:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385935 MB (7% inode=39%): [07:29:02] Load avg. on adenia is OK: OK - load average: 14.82, 14.77, 10.38 [07:32:02] Load avg. on adenia is WARNING: WARNING - load average: 17.45, 15.68, 11.52 [07:34:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:52:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:53:02] Load avg. on adenia is OK: OK - load average: 7.76, 13.84, 14.82 [08:03:35] [[Special:Log/newusers]] create 10 * Carter625t * (New user account) [08:05:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.89, 23.23, 20.06 [08:10:51] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26649.000000 [08:13:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:14:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:16:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.467773/1.95, alarm hl:np_load_avg=2.987793/2.0, alarm hl:mem_free=132.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.467773/2.3, alarm hl:np_load_long=2.837891/2.5, alarm hl:cpu=95.300000/98, alarm hl:mem_free=132.000000M/150M, alarm hl:available=1/0 [08:20:21] Load avg. on willow is WARNING: WARNING - load average: 12.33, 16.41, 19.73 [08:22:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:27:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386404 MB (7% inode=39%): [08:34:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:36:31] Load avg. on willow is OK: OK - load average: 11.13, 12.12, 14.74 [08:46:52] 3(created) [TS-1362] Replication on rosemary commons not working because of missing table; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1362> (Marlen Caemmerer) [08:48:53] 3(assigned) [TS-1362] Replication on rosemary commons not working because of missing table <10https://jira.toolserver.org/browse/TS-1362> (Marlen Caemmerer) [08:52:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:52:32] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.473633/1.10, alarm hl:np_load_long=0.970703/1.55, alarm hl:mem_free=17656.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.473633/1.00, alarm hl:np_load_long=0.970703/1.50, alarm hl:mem_free=17656.000000M/600M, alarm hl:available=1/0 [08:53:32] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:57:41] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:03:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.759277/1.95, alarm hl:np_load_avg=1.527832/2.0, alarm hl:mem_free=334.000000M/350M, alarm hl:available=1/0 [09:04:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:07:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.629395/1.95, alarm hl:np_load_avg=1.604492/2.0, alarm hl:mem_free=335.000000M/350M, alarm hl:available=1/0 [09:07:20] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:10:50] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29517.000000 [09:11:31] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.356445/1.10, alarm hl:np_load_long=0.324219/1.55, alarm hl:mem_free=356.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.356445/1.00, alarm hl:np_load_long=0.324219/1.50, alarm hl:mem_free=356.000000M/600M, alarm hl:available=1/0 [09:11:51] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:12:31] Load avg. on willow is WARNING: WARNING - load average: 16.59, 15.41, 13.50 [09:12:32] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [09:13:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:14:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:14:30] Load avg. on willow is OK: OK - load average: 11.43, 14.38, 13.39 [09:17:31] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.317383/1.00, alarm hl:np_load_long=0.327637/1.50, alarm hl:mem_free=599.000000M/600M, alarm hl:available=1/0 [09:18:31] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:27:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386264 MB (7% inode=39%): [09:34:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:42:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 126364 MB (10% inode=99%): [09:52:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:57:41] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:04:52] 3(commented) [TS-1362] Replication on rosemary commons not working because of missing table <10https://jira.toolserver.org/browse/TS-1362> (Marlen Caemmerer) [10:05:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.329101/1.95, alarm hl:np_load_avg=1.359863/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.329101/2.3, alarm hl:np_load_long=1.334961/2.5, alarm hl:cpu=81.600000/98, alarm hl:mem_free=125.000000M/150M, alarm hl:available=1/0 [10:08:25] anyone around? [10:10:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31859.000000 [10:12:52] 3(commented) [TS-1362] Replication on rosemary commons not working because of missing table <10https://jira.toolserver.org/browse/TS-1362> (DaB.) [10:13:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:14:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:14:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:16:21] Environment IPMI on thyme is OK: ok: temperature ok fan ok voltage ok chassis ok [10:16:31] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.327637/1.10, alarm hl:np_load_long=0.332520/1.55, alarm hl:mem_free=382.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.327637/1.00, alarm hl:np_load_long=0.332520/1.50, alarm hl:mem_free=382.000000M/600M, alarm hl:available=1/0 [10:17:55] 3(resolved) [TS-1362] Replication on rosemary commons not working because of missing table <10https://jira.toolserver.org/browse/TS-1362> (DaB.) [10:19:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:22:31] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [10:22:41] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:52] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:12] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:12] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:32] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:23:51] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:23:51] SMTP on hyacinth is OK: SMTP OK - 9.237 sec. response time [10:24:01] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 193.000000 [10:24:01] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:24:01] SMTP on z-dat-s3-a is OK: SMTP OK - 0.062 sec. response time [10:24:11] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:24:20] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [10:24:21] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [10:24:32] MySQL slave on z-dat-s6-a is OK: Uptime: 1782230 Threads: 11 Questions: 413378941 Slow queries: 106352 Opens: 4548618 Flush tables: 2 Open tables: 2961 Queries per second avg: 231.944 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 173 [10:24:32] MySQL on z-dat-s6-a is OK: Uptime: 1782230 Threads: 11 Questions: 413378943 Slow queries: 106352 Opens: 4548618 Flush tables: 2 Open tables: 2961 Queries per second avg: 231.944 [10:24:32] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:27:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386166 MB (7% inode=39%): [10:34:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:41:01] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2624.000000 [10:42:02] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1795.000000 [10:52:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:55:31] [[~dispenser/view/Checklinks]] ! 10https://wiki.toolserver.org/w/index.php?diff=7120&oldid=4514&rcid=9419 * 94.202.111.249 * (-8) () [10:55:32] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:56:11] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:57:41] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:08:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.307617/1.95, alarm hl:np_load_avg=1.350098/2.0, alarm hl:mem_free=217.000000M/350M, alarm hl:available=1/0 [11:13:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:14:42] Environment IPMI on thyme is OK: ok: temperature ok fan ok voltage ok chassis ok [11:15:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:16:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:18:01] Load avg. on adenia is WARNING: WARNING - load average: 20.44, 12.33, 7.84 [11:21:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:22:31] Environment IPMI on thyme is OK: ok: temperature ok fan ok voltage ok chassis ok [11:27:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386076 MB (7% inode=39%): [11:33:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.249024/1.95, alarm hl:np_load_avg=1.415039/2.0, alarm hl:mem_free=275.000000M/350M, alarm hl:available=1/0 [11:34:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:35:59] Hello, My account has been expired. How can I resuscitate it? [11:40:11] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:52:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:54:12] Load avg. on adenia is OK: OK - load average: 8.29, 11.34, 14.63 [11:57:14] is there any ts-admin online here? [11:57:42] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:03:42] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.411133/1.10, alarm hl:np_load_long=0.859375/1.55, alarm hl:mem_free=17107.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.411133/1.00, alarm hl:np_load_long=0.859375/1.50, alarm hl:mem_free=17107.000000M/600M, alarm hl:available=1/0 [12:05:42] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:07:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.520996/1.95, alarm hl:np_load_avg=1.592285/2.0, alarm hl:mem_free=220.000000M/350M, alarm hl:available=1/0 [12:08:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:11:41] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.515625/1.10, alarm hl:np_load_long=0.990235/1.55, alarm hl:mem_free=16638.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.515625/1.00, alarm hl:np_load_long=0.990235/1.50, alarm hl:mem_free=16638.000000M/600M, alarm hl:available=1/0 [12:12:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.693359/1.95, alarm hl:np_load_avg=1.618652/2.0, alarm hl:mem_free=273.000000M/350M, alarm hl:available=1/0 [12:13:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:15:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:21:54] 3(created) [TS-1363] recreat my account; Toolserver; Critical Task <10https://jira.toolserver.org/browse/TS-1363> (Ladsgroup) [12:21:58] 3(updated) [TS-1363] recreate "amir" account <10https://jira.toolserver.org/browse/TS-1363> (Ladsgroup) [12:27:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 386004 MB (7% inode=39%): [12:35:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:40:42] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.205078/1.10, alarm hl:np_load_long=0.936523/1.55, alarm hl:mem_free=17188.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.205078/1.00, alarm hl:np_load_long=0.936523/1.50, alarm hl:mem_free=17188.000000M/600M, alarm hl:available=1/0 [12:44:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:47:41] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.769531/1.10, alarm hl:np_load_long=1.107422/1.55, alarm hl:mem_free=17006.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.769531/1.00, alarm hl:np_load_long=1.107422/1.50, alarm hl:mem_free=17006.000000M/600M, alarm hl:available=1/0 [12:52:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:52:51] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.102539/1.10, alarm hl:np_load_long=0.706055/1.55, alarm hl:mem_free=1142.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.102539/1.00, alarm hl:np_load_long=0.706055/1.50, alarm hl:mem_free=1142.000000M/600M, alarm hl:available=1/0 [12:56:51] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [12:57:51] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:00:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:03:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.484375/1.95, alarm hl:np_load_avg=1.341797/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.484375/2.3, alarm hl:np_load_long=1.324707/2.5, alarm hl:cpu=88.900000/98, alarm hl:mem_free=125.000000M/150M, alarm hl:available=1/0 [13:04:52] hello all [13:08:42] DaBPunkt: I'm rather untrusting of TUSC. Can I authenticate users by their watchlist token+a user whitelist to display watcher counts under 30 for special users? http://enwp.org/WP:VPP#Limit_number_of_watchers_to_those_active_in_the_last_.3F.3F_days [13:10:01] wrong pump, http://enwp.org/WP:VPR#Limit_number_of_watchers_to_those_active_in_the_last_.3F.3F_days [13:13:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:15:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:17:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:20:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.294922/1.95, alarm hl:np_load_avg=1.433594/2.0, alarm hl:mem_free=334.000000M/350M, alarm hl:available=1/0 [13:27:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385897 MB (7% inode=39%): [13:30:19] DaBPunkt: ^ [13:32:21] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:35:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:38:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:42:12] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:52:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:57:21] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:57:52] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:13:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:15:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:20:53] 3(created) [MNT-1231] Reboot hyacinth for debugging; Maintenance; Planned work - user impact <10https://jira.toolserver.org/browse/MNT-1231> (Marlen Caemmerer) [14:21:00] 3(assigned) [MNT-1231] Reboot hyacinth for debugging <10https://jira.toolserver.org/browse/MNT-1231> (Marlen Caemmerer) [14:22:52] 3(created) [MNT-1232] More space for cassia s2+s5; Maintenance; Planned work - user impact <10https://jira.toolserver.org/browse/MNT-1232> (Marlen Caemmerer) [14:27:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385811 MB (7% inode=39%): [14:28:02] Marlen Caemmerer * [Toolserver-l] Maintenance on Friday [14:35:00] 3(resolved) [TS-1363] recreate "amir" account <10https://jira.toolserver.org/browse/TS-1363> (DaB.) [14:35:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:52:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:57:52] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:03:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.282226/1.95, alarm hl:np_load_avg=1.232422/2.0, alarm hl:mem_free=338.000000M/350M, alarm hl:available=1/0 [15:04:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:07:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.312012/1.95, alarm hl:np_load_avg=1.246582/2.0, alarm hl:mem_free=225.000000M/350M, alarm hl:available=1/0 [15:09:52] 3(commented) [TS-1363] recreate "amir" account <10https://jira.toolserver.org/browse/TS-1363> (Ladsgroup) [15:13:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:16:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:25:55] 3(commented) [TS-1363] recreate "amir" account <10https://jira.toolserver.org/browse/TS-1363> (DaB.) [15:27:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385717 MB (7% inode=39%): [15:36:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:38:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.092285/1.95, alarm hl:np_load_avg=1.111816/2.0, alarm hl:mem_free=228.000000M/350M, alarm hl:available=1/0 [15:40:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:43:02] DaB. * [Toolserver-announce] Fwd: [Toolserver-l] Maintenance on Friday [15:51:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.179199/1.95, alarm hl:np_load_avg=1.141601/2.0, alarm hl:mem_free=248.000000M/350M, alarm hl:available=1/0 [15:52:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:58:01] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:03:01] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:13:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:16:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:27:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385635 MB (7% inode=39%): [16:36:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.022949/1.95, alarm hl:np_load_avg=1.103027/2.0, alarm hl:mem_free=213.000000M/350M, alarm hl:available=1/0 [16:36:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:39:10] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:52:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:13:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.206055/1.95, alarm hl:np_load_avg=1.208984/2.0, alarm hl:mem_free=312.000000M/350M, alarm hl:available=1/0 [17:13:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:14:21] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:16:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:23:00] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:28:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385720 MB (7% inode=39%): [17:28:41] Environment IPMI on thyme is OK: ok: temperature ok fan ok voltage ok chassis ok [17:37:01] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:39:01] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:52:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:03:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.341309/1.95, alarm hl:np_load_avg=1.251953/2.0, alarm hl:mem_free=147.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.341309/2.3, alarm hl:np_load_long=1.096191/2.5, alarm hl:cpu=73.100000/98, alarm hl:mem_free=147.000000M/150M, alarm hl:available=1/0 [18:04:21] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:11:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.698242/1.95, alarm hl:np_load_avg=1.373535/2.0, alarm hl:mem_free=226.000000M/350M, alarm hl:available=1/0 [18:13:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:14:01] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:16:21] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40291 MB (9% inode=99%): [18:16:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:16:41] Environment IPMI on thyme is OK: ok: temperature ok fan ok voltage ok chassis ok [18:22:20] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:27:00] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [18:27:20] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23825 MB (5% inode=99%): [18:28:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385659 MB (7% inode=39%): [18:37:01] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:38:00] MySQL on adenia is CRITICAL: Cant connect to MySQL server on adenia (146) [18:40:01] MySQL on adenia is OK: Uptime: 45 Threads: 30 Questions: 407 Slow queries: 2 Opens: 56 Flush tables: 1 Open tables: 39 Queries per second avg: 9.44 [18:42:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41362 MB (10% inode=99%): [18:43:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 55480 MB (13% inode=99%): [18:50:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41287 MB (10% inode=99%): [18:53:31] Load avg. on z-dat-s7-a is WARNING: WARNING - load average: 16.41, 14.02, 11.47 [18:53:31] Load avg. on z-dat-s6-a is WARNING: WARNING - load average: 16.45, 14.04, 11.48 [18:53:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:55:11] Load avg. on z-dat-s3-a is WARNING: WARNING - load average: 16.41, 14.74, 12.02 [18:55:11] Load avg. on hyacinth is WARNING: WARNING - load average: 16.41, 14.74, 12.02 [18:55:11] Load avg. on z-dat-s4-a is WARNING: WARNING - load average: 16.46, 14.76, 12.02 [18:58:32] Load avg. on z-dat-s7-a is OK: OK - load average: 13.67, 14.91, 12.68 [18:58:32] Load avg. on z-dat-s6-a is OK: OK - load average: 13.61, 14.89, 12.68 [18:59:11] Load avg. on z-dat-s3-a is OK: OK - load average: 12.26, 14.39, 12.60 [18:59:11] Load avg. on hyacinth is OK: OK - load average: 12.26, 14.39, 12.60 [18:59:11] Load avg. on z-dat-s4-a is OK: OK - load average: 12.25, 14.38, 12.60 [19:13:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:13:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.148926/1.95, alarm hl:np_load_avg=1.170899/2.0, alarm hl:mem_free=309.000000M/350M, alarm hl:available=1/0 [19:14:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:16:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:27:53] 3(created) [DRTRIGON-122] Additional options to config sum_disc.py dealing with thread header; DrTrigon's tools; New Feature <10https://jira.toolserver.org/browse/DRTRIGON-122> (Xqt ) [19:28:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385544 MB (7% inode=39%): [19:37:01] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:39:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.110840/1.95, alarm hl:np_load_avg=1.118652/2.0, alarm hl:mem_free=343.000000M/350M, alarm hl:available=1/0 [19:42:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:53:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:08:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.398926/1.95, alarm hl:np_load_avg=1.360840/2.0, alarm hl:mem_free=227.000000M/350M, alarm hl:available=1/0 [20:14:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:14:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:17:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:28:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385459 MB (7% inode=39%): [20:37:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:39:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.111328/1.95, alarm hl:np_load_avg=1.218262/2.0, alarm hl:mem_free=325.000000M/350M, alarm hl:available=1/0 [20:40:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:43:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.958984/1.95, alarm hl:np_load_avg=1.140137/2.0, alarm hl:mem_free=296.000000M/350M, alarm hl:available=1/0 [20:50:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:53:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:02:12] Environment IPMI on thyme is OK: ok: temperature ok fan ok voltage ok chassis ok [21:08:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:14:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:17:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:28:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385301 MB (7% inode=39%): [21:37:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:42:11] /sql on cassia is WARNING: DISK WARNING - free space: /sql 125359 MB (10% inode=99%): [21:42:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.946289/1.95, alarm hl:np_load_avg=0.960449/2.0, alarm hl:mem_free=332.000000M/350M, alarm hl:available=1/0 [21:43:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:50:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.959961/1.95, alarm hl:np_load_avg=0.981934/2.0, alarm hl:mem_free=333.000000M/350M, alarm hl:available=1/0 [21:53:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:14:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:18:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:23:39] nacht ts [22:28:21] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385200 MB (7% inode=39%): [22:34:20] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:37:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:38:10] Environment IPMI on thyme is OK: ok: temperature ok fan ok voltage ok chassis ok [22:41:21] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:42:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.048828/1.95, alarm hl:np_load_avg=1.103027/2.0, alarm hl:mem_free=304.000000M/350M, alarm hl:available=1/0 [22:45:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:49:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.010742/1.95, alarm hl:np_load_avg=1.096680/2.0, alarm hl:mem_free=289.000000M/350M, alarm hl:available=1/0 [22:54:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:14:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:18:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:28:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 385130 MB (7% inode=39%): [23:30:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.596191/1.95, alarm hl:np_load_avg=1.508301/2.0, alarm hl:mem_free=123.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.596191/2.3, alarm hl:np_load_long=1.333008/2.5, alarm hl:cpu=86.300000/98, alarm hl:mem_free=123.000000M/150M, alarm hl:available=1/0 [23:34:32] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:37:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:53:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:54:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk