[00:00:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:00:48] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [00:01:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:03:08] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 649 MB (17% inode=99%): [00:04:09] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1024 MB (25% inode=99%): [00:10:48] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 472459 MB (8% inode=44%): [00:11:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1382743.000000 [00:12:38] Load avg. on willow is OK: OK - load average: 12.63, 13.74, 14.86 [00:18:38] Load avg. on willow is WARNING: WARNING - load average: 15.59, 14.55, 14.82 [00:21:38] Load avg. on willow is OK: OK - load average: 14.84, 14.65, 14.81 [00:21:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.838867/1.95, alarm hl:np_load_avg=1.827149/2.0, alarm hl:mem_free=135.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.838867/2.3, alarm hl:np_load_long=1.849609/2.5, alarm hl:cpu=89.100000/98, alarm hl:mem_free=135.000000M/150M, alarm hl:available=1/0 [00:26:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:36:08] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1408765 [00:43:37] Load avg. on willow is WARNING: WARNING - load average: 16.24, 16.63, 16.47 [00:50:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:00:47] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:00:47] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [01:01:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:05:37] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.03, 22.59, 20.35 [01:11:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474498 MB (8% inode=44%): [01:11:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1386343.000000 [01:14:38] Load avg. on willow is WARNING: WARNING - load average: 17.61, 19.49, 19.93 [01:22:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.308105/1.95, alarm hl:np_load_avg=2.265137/2.0, alarm hl:mem_free=137.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.308105/2.3, alarm hl:np_load_long=2.374512/2.5, alarm hl:cpu=99.900000/98, alarm hl:mem_free=137.000000M/150M, alarm hl:available=1/0 [01:27:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:37:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1386034 [01:51:47] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:52:47] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.033203/1.00, alarm hl:np_load_long=0.654297/1.50, alarm hl:mem_free=18312.000000M/350M, alarm hl:available=1/0 [01:53:49] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:01:47] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:01:47] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [02:02:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:12:48] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 474413 MB (8% inode=44%): [02:12:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1375387.000000 [02:14:37] Load avg. on willow is WARNING: WARNING - load average: 21.26, 20.19, 19.22 [02:22:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.518555/1.95, alarm hl:np_load_avg=2.476562/2.0, alarm hl:mem_free=488.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.518555/2.3, alarm hl:np_load_long=2.426758/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=488.000000M/150M, alarm hl:available=1/0 [02:28:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:32:37] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.51, 21.12, 20.11 [02:34:49] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.828125/1.10, alarm hl:np_load_long=1.316406/1.55, alarm hl:mem_free=19328.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.828125/1.00, alarm hl:np_load_long=1.316406/1.50, alarm hl:mem_free=19328.000000M/350M, alarm hl:available=1/0 [02:37:07] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1364567 [02:37:48] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:46:28] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35548 MB (8% inode=99%): [02:51:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 20811 MB (5% inode=99%): [02:51:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:00:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 71280 MB (17% inode=99%): [03:01:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:01:48] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [03:02:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:12:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 472681 MB (8% inode=44%): [03:12:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1351226.000000