[00:02:10] Environment IPMI on thyme is OK: ok: temperature ok fan ok voltage ok chassis ok [00:02:32] Load avg. on willow is WARNING: WARNING - load average: 16.79, 13.45, 11.30 [00:03:32] Load avg. on willow is OK: OK - load average: 13.13, 13.00, 11.27 [00:06:32] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:11:31] Load avg. on willow is WARNING: WARNING - load average: 20.22, 16.04, 13.09 [00:15:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:17:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.493652/1.95, alarm hl:np_load_avg=1.741699/2.0, alarm hl:mem_free=296.000000M/350M, alarm hl:available=1/0 [00:18:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:28:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 384849 MB (7% inode=39%): [00:29:32] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:32:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.610351/1.95, alarm hl:np_load_avg=1.702149/2.0, alarm hl:mem_free=290.000000M/350M, alarm hl:available=1/0 [00:33:32] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:37:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:42:21] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:47:10] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:54:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:07:21] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:08:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.576660/1.95, alarm hl:np_load_avg=1.469238/2.0, alarm hl:mem_free=282.000000M/350M, alarm hl:available=1/0 [01:14:33] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:15:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:19:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:21:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.533691/1.95, alarm hl:np_load_avg=1.494141/2.0, alarm hl:mem_free=246.000000M/350M, alarm hl:available=1/0 [01:29:30] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 384710 MB (7% inode=39%): [01:38:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:49:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.329590/1.95, alarm hl:np_load_avg=1.268555/2.0, alarm hl:mem_free=183.000000M/350M, alarm hl:available=1/0 [01:54:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:01:32] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37765 MB (9% inode=99%): [02:05:33] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24267 MB (5% inode=99%): [02:12:41] Load avg. on willow is WARNING: WARNING - load average: 16.54, 15.56, 13.45 [02:14:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 25605 MB (6% inode=99%): [02:14:41] Load avg. on willow is OK: OK - load average: 14.11, 14.97, 13.49 [02:16:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:17:40] Load avg. on willow is WARNING: WARNING - load average: 15.96, 15.77, 14.09 [02:20:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:25:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.589844/1.10, alarm hl:np_load_long=0.749023/1.55, alarm hl:mem_free=17300.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.589844/1.00, alarm hl:np_load_long=0.749023/1.50, alarm hl:mem_free=17300.000000M/600M, alarm hl:available=1/0 [02:26:01] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:29:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 383532 MB (7% inode=39%): [02:38:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:41:31] / on wolfsbane is WARNING: DISK WARNING - free space: / 6286 MB (20% inode=93%): [02:44:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:54:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:02:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.849121/1.95, alarm hl:np_load_avg=1.630371/2.0, alarm hl:mem_free=221.000000M/350M, alarm hl:available=1/0 [03:11:41] Load avg. on willow is WARNING: WARNING - load average: 19.90, 16.86, 14.26 [03:14:18] [[User:Carter625t]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7121&rcid=9420 * Carter625t * (+145) (Created page with "Hey this is an awesome site you need to find. My Website: [http://www.coachmee.de/coaching-in-frankfurt-coach.html Coach Beratung in Frankfurt]") [03:16:11] Joan: Nuke that shit. [03:16:17] So much spam. [03:16:21] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User:Carter625t10]]": spam) [03:16:34] Thanks gurl. [03:16:38] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Carter625t10]] with an expiry time of infinite (account creation disabled): inappropriate behavior) [03:16:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:16:47] Abuse! [03:17:32] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:17:41] Load avg. on willow is OK: OK - load average: 13.76, 14.93, 14.21 [03:17:48] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User:Meetcheese12310]]": spam) [03:18:04] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Meetcheese12310]] with an expiry time of infinite (account creation disabled): inappropriate behavior) [03:18:33] https://wiki.toolserver.org/view/Special:Contributions/Karin726g seems like a sleeper. [03:19:18] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User:Ferrinojv164710]]": spam) [03:19:31] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Ferrinojv164710]] with an expiry time of infinite (account creation disabled): inappropriate behavior) [03:20:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:22:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.606445/1.95, alarm hl:np_load_avg=1.674316/2.0, alarm hl:mem_free=251.000000M/350M, alarm hl:available=1/0 [03:26:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:29:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382703 MB (7% inode=39%): [03:32:42] Load avg. on willow is WARNING: WARNING - load average: 17.62, 16.21, 14.77 [03:34:42] Load avg. on willow is OK: OK - load average: 11.94, 14.68, 14.38 [03:38:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:41:32] / on wolfsbane is WARNING: DISK WARNING - free space: / 6153 MB (20% inode=93%): [03:52:17] [[Special:Log/newusers]] create 10 * Vincci Hui * (New user account) [03:53:21] [[Special:Log/newusers]] create 10 * Annpenguins93z * (New user account) [03:54:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:05:01] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.661133/1.10, alarm hl:np_load_long=0.763672/1.55, alarm hl:mem_free=17077.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.661133/1.00, alarm hl:np_load_long=0.763672/1.50, alarm hl:mem_free=17077.000000M/600M, alarm hl:available=1/0 [04:06:01] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:07:31] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.481934/1.00, alarm hl:np_load_long=0.414062/1.50, alarm hl:mem_free=598.000000M/600M, alarm hl:available=1/0 [04:08:32] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [04:11:31] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.549805/1.10, alarm hl:np_load_long=0.445312/1.55, alarm hl:mem_free=165.000000M/500M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.549805/1.00, alarm hl:np_load_long=0.445312/1.50, alarm hl:mem_free=165.000000M/600M, alarm hl:available=1/0 [04:12:41] Load avg. on willow is WARNING: WARNING - load average: 14.68, 15.06, 14.43 [04:13:01] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.094726/1.00, alarm hl:np_load_long=0.797852/1.50, alarm hl:mem_free=17032.000000M/600M, alarm hl:available=1/0 [04:14:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.660645/1.95, alarm hl:np_load_avg=1.827637/2.0, alarm hl:mem_free=283.000000M/350M, alarm hl:available=1/0 [04:14:41] Load avg. on willow is OK: OK - load average: 12.58, 14.42, 14.29 [04:16:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:20:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:26:47] [[User:Annpenguins93z]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7122&rcid=9429 * Annpenguins93z * (+593) (Created page with "I'm Heather, yet i am a 19 year-old journalism student. I am personally a gamer, comic book adapted reader, sci-fi fanatic, role player, tech lover, all the while what some would...") [04:28:33] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User:Annpenguins93z10]]": spam) [04:28:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:28:47] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Annpenguins93z10]] with an expiry time of infinite (account creation disabled): inappropriate behavior) [04:29:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382664 MB (7% inode=39%): [04:37:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.656738/1.95, alarm hl:np_load_avg=1.688965/2.0, alarm hl:mem_free=269.000000M/350M, alarm hl:available=1/0 [04:38:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:41:31] / on wolfsbane is WARNING: DISK WARNING - free space: / 6027 MB (20% inode=93%): [04:47:41] Load avg. on willow is WARNING: WARNING - load average: 15.71, 14.37, 13.80 [04:48:40] Load avg. on willow is OK: OK - load average: 14.73, 14.33, 13.82 [04:49:41] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:55:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:01:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.380371/1.95, alarm hl:np_load_avg=1.856445/2.0, alarm hl:mem_free=131.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.380371/2.3, alarm hl:np_load_long=1.749024/2.5, alarm hl:cpu=95.800000/98, alarm hl:mem_free=131.000000M/150M, alarm hl:available=1/0 [05:04:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.692383/1.10, alarm hl:np_load_long=0.855469/1.55, alarm hl:mem_free=17569.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.692383/1.00, alarm hl:np_load_long=0.855469/1.50, alarm hl:mem_free=17569.000000M/600M, alarm hl:available=1/0 [05:05:01] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:07:41] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:16:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:20:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:21:41] Load avg. on willow is WARNING: WARNING - load average: 17.72, 16.05, 14.49 [05:29:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382571 MB (7% inode=39%): [05:29:41] Load avg. on willow is OK: OK - load average: 12.03, 14.81, 14.78 [05:30:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.877930/1.95, alarm hl:np_load_avg=2.335449/2.0, alarm hl:mem_free=154.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.877930/2.3, alarm hl:np_load_long=2.011230/2.5, alarm hl:cpu=96.600000/98, alarm hl:mem_free=154.000000M/150M, alarm hl:available=1/0 [05:32:42] Load avg. on willow is WARNING: WARNING - load average: 19.14, 18.42, 16.36 [05:38:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:41:42] / on wolfsbane is WARNING: DISK WARNING - free space: / 5875 MB (19% inode=93%): [05:55:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:12:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.04, 21.69, 20.10 [06:16:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:17:41] Load avg. on willow is WARNING: WARNING - load average: 16.80, 19.78, 19.88 [06:21:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:30:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382492 MB (7% inode=39%): [06:30:41] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.124512/1.95, alarm hl:np_load_avg=2.264648/2.0, alarm hl:mem_free=133.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.124512/2.3, alarm hl:np_load_long=2.294434/2.5, alarm hl:cpu=98.400000/98, alarm hl:mem_free=133.000000M/150M, alarm hl:available=1/0 [06:38:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:41:41] / on wolfsbane is WARNING: DISK WARNING - free space: / 5740 MB (19% inode=93%): [06:41:43] Would installing checkuser or the Tor block extension stop the spam? [06:55:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:00:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.18, 22.34, 20.20 [07:13:53] Load avg. on willow is WARNING: WARNING - load average: 14.45, 18.29, 19.95 [07:16:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:18:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.84, 20.42, 20.28 [07:21:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:30:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382361 MB (7% inode=39%): [07:30:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=5.097656/1.95, alarm hl:np_load_avg=3.290039/2.0, alarm hl:mem_free=85.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=5.097656/2.3, alarm hl:np_load_long=2.851074/2.5, alarm hl:cpu=99.900000/98, alarm hl:mem_free=85.000000M/150M, alarm hl:available=1/0 [07:38:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:41:42] / on wolfsbane is WARNING: DISK WARNING - free space: / 5594 MB (18% inode=93%): [07:55:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:16:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:18:54] Load avg. on willow is CRITICAL: CRITICAL - load average: 14.54, 18.64, 20.79 [08:20:54] Load avg. on willow is WARNING: WARNING - load average: 11.61, 16.24, 19.62 [08:21:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:27:22] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:30:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382283 MB (7% inode=39%): [08:30:54] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.02, 22.03, 20.00 [08:30:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.669922/1.95, alarm hl:np_load_avg=2.745605/2.0, alarm hl:mem_free=134.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.669922/2.3, alarm hl:np_load_long=2.492676/2.5, alarm hl:cpu=99.600000/98, alarm hl:mem_free=134.000000M/150M, alarm hl:available=1/0 [08:33:53] Load avg. on willow is WARNING: WARNING - load average: 16.71, 20.18, 19.70 [08:38:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:41:54] / on wolfsbane is WARNING: DISK WARNING - free space: / 5312 MB (17% inode=93%): [08:42:22] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:50:54] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.20, 23.16, 20.62 [08:51:52] / on wolfsbane is OK: DISK OK - free space: / 6692 MB (22% inode=93%): [08:54:54] Load avg. on willow is WARNING: WARNING - load average: 16.96, 20.07, 19.95 [08:55:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:55:54] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.21, 20.96, 20.29 [09:16:55] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.12, 23.77, 22.95 [09:16:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:21:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:23:55] Load avg. on willow is WARNING: WARNING - load average: 12.47, 16.63, 19.75 [09:25:23] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:25:44] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:44] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:44] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:44] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:44] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:02] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:03] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:03] Load avg. on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:03] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:03] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:03] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:03] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:12] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:13] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:26:33] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:26:34] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:26:34] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 2021 MB (99% inode=99%): [09:26:34] SMTP on z-dat-s4-a is OK: SMTP OK - 0.005 sec. response time [09:26:34] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:26:34] Load avg. on hyacinth is OK: OK - load average: 0.47, 1.44, 1.69 [09:26:35] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:26:35] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 2036 MB (99% inode=99%): [09:26:36] SMF on z-dat-s4-a is OK: OK - all services online [09:26:36] / on hyacinth is OK: DISK OK - free space: / 8427 MB (28% inode=85%): [09:26:42] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [09:26:42] / on z-dat-s4-a is OK: DISK OK - free space: / 8427 MB (28% inode=85%): [09:26:43] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 97493 MB (24% inode=99%): [09:26:53] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [09:26:54] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [09:26:54] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:26:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:27:02] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:27:03] MySQL on z-dat-s3-a is OK: Uptime: 4924604 Threads: 20 Questions: 5568463420 Slow queries: 274435 Opens: 43273290 Flush tables: 1 Open tables: 16384 Queries per second avg: 1130.743 [09:27:03] MySQL slave on z-dat-s3-a is OK: Uptime: 4924604 Threads: 20 Questions: 5568463421 Slow queries: 274435 Opens: 43273290 Flush tables: 1 Open tables: 16384 Queries per second avg: 1130.743 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 197 [09:29:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.333984/1.95, alarm hl:np_load_avg=1.732910/2.0, alarm hl:mem_free=291.000000M/350M, alarm hl:available=1/0 [09:30:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382185 MB (7% inode=39%): [09:38:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:42:34] /sql on cassia is WARNING: DISK WARNING - free space: /sql 123443 MB (10% inode=99%): [09:48:54] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.32, 22.47, 20.14 [09:54:54] Load avg. on willow is WARNING: WARNING - load average: 16.33, 19.68, 19.77 [09:55:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:56:54] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.86, 20.54, 20.09 [10:17:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:21:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:29:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.459473/1.95, alarm hl:np_load_avg=2.696777/2.0, alarm hl:mem_free=262.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.459473/2.3, alarm hl:np_load_long=2.600098/2.5, alarm hl:cpu=99.200000/98, alarm hl:mem_free=262.000000M/150M, alarm hl:available=1/0 [10:31:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382147 MB (7% inode=39%): [10:38:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:39:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 18.45, 18.77, 20.18 [10:44:53] Load avg. on willow is WARNING: WARNING - load average: 16.96, 18.65, 19.94 [10:45:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.13, 20.55, 20.54 [10:53:53] Load avg. on willow is WARNING: WARNING - load average: 14.37, 18.29, 19.66 [10:55:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:13:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.49, 26.04, 23.84 [11:17:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:21:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:27:53] Load avg. on willow is WARNING: WARNING - load average: 16.05, 17.70, 19.80 [11:29:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.636230/1.95, alarm hl:np_load_avg=1.996094/2.0, alarm hl:mem_free=291.000000M/350M, alarm hl:available=1/0 [11:31:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 382042 MB (7% inode=39%): [11:38:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:43:42] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.430664/1.10, alarm hl:np_load_long=0.935547/1.55, alarm hl:mem_free=17283.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.430664/1.00, alarm hl:np_load_long=0.935547/1.50, alarm hl:mem_free=17283.000000M/600M, alarm hl:available=1/0 [11:50:42] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:50:54] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.07, 21.98, 20.07 [11:51:53] 3(commented) [ACCAPP-495] Wikipedia mapping tool in collaboration with Oxford University <10https://jira.toolserver.org/browse/ACCAPP-495> (Gavin Baily) [11:53:54] Load avg. on willow is WARNING: WARNING - load average: 15.73, 20.01, 19.69 [11:55:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:57:43] Load avg. on adenia is WARNING: WARNING - load average: 16.77, 13.83, 9.99 [11:58:43] Load avg. on adenia is OK: OK - load average: 13.05, 13.30, 10.05 [11:59:55] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:00:43] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:23] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:01:43] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:44] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:44] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:54] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:03] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:02:13] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:02:34] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:02:34] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:02:34] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:02:35] SMTP on z-dat-s7-a is OK: SMTP OK - 0.019 sec. response time [12:02:35] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 2117 MB (99% inode=99%): [12:02:43] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:05:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.830078/1.95, alarm hl:np_load_avg=1.692871/2.0, alarm hl:mem_free=262.000000M/350M, alarm hl:available=1/0 [12:09:43] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.005859/1.00, alarm hl:np_load_long=1.049805/1.50, alarm hl:mem_free=16288.000000M/600M, alarm hl:available=1/0 [12:09:53] Load avg. on willow is OK: OK - load average: 12.06, 12.68, 14.96 [12:10:43] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:11:03] hello all [12:13:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:17:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:21:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:22:23] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:30:42] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.177734/1.10, alarm hl:np_load_long=0.977539/1.55, alarm hl:mem_free=16797.000000M/500M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.177734/1.00, alarm hl:np_load_long=0.977539/1.50, alarm hl:mem_free=16797.000000M/600M, alarm hl:available=1/0 [12:32:23] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:32:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 381910 MB (7% inode=39%): [12:32:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.966309/1.95, alarm hl:np_load_avg=1.792480/2.0, alarm hl:mem_free=389.000000M/350M, alarm hl:available=1/0 [12:33:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:38:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:54:54] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:02] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:55:12] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:13] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:23] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:23] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:23] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:23] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:23] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:23] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:23] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:24] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:34] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:55:43] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:55:43] SMF on z-dat-s4-a is OK: OK - all services online [12:55:54] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:55:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:55:54] / on z-dat-s7-a is OK: DISK OK - free space: / 8426 MB (28% inode=85%): [12:55:54] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 100225 MB (24% inode=99%): [12:55:54] Load avg. on z-dat-s7-a is OK: OK - load average: 1.07, 1.08, 1.58 [12:55:54] /tmp on hyacinth is OK: DISK OK - free space: /tmp 1924 MB (99% inode=99%): [12:55:54] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [12:55:55] SMF on hyacinth is OK: OK - all services online [12:55:55] SMF on z-dat-s6-a is OK: OK - all services online [12:55:56] SMF on z-dat-s7-a is OK: OK - all services online [12:55:56] SMF on z-dat-s3-a is OK: OK - all services online [12:56:02] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:02:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.873535/1.95, alarm hl:np_load_avg=1.777832/2.0, alarm hl:mem_free=288.000000M/350M, alarm hl:available=1/0 [13:05:54] Load avg. on willow is WARNING: WARNING - load average: 18.69, 15.80, 14.14 [13:14:53] Load avg. on willow is OK: OK - load average: 11.56, 14.50, 14.45 [13:16:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:17:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:19:43] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.007812/1.00, alarm hl:np_load_long=0.983398/1.50, alarm hl:mem_free=17566.000000M/600M, alarm hl:available=1/0 [13:22:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:24:43] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:30:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.623047/1.95, alarm hl:np_load_avg=1.846191/2.0, alarm hl:mem_free=173.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.623047/2.3, alarm hl:np_load_long=1.743164/2.5, alarm hl:cpu=97.200000/98, alarm hl:mem_free=173.000000M/150M, alarm hl:available=1/0 [13:32:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 381724 MB (7% inode=39%): [13:32:54] Load avg. on willow is WARNING: WARNING - load average: 15.33, 14.89, 14.11 [13:33:54] Load avg. on willow is OK: OK - load average: 12.55, 14.29, 13.96 [13:39:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:48:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:51:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.362305/1.95, alarm hl:np_load_avg=1.365234/2.0, alarm hl:mem_free=222.000000M/350M, alarm hl:available=1/0 [13:55:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:13:03] Load avg. on willow is WARNING: WARNING - load average: 18.68, 15.68, 13.03 [14:17:03] Load avg. on willow is OK: OK - load average: 12.00, 14.43, 13.19 [14:18:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:23:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:32:35] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 381590 MB (7% inode=39%): [14:40:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:55:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:03:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.270996/1.95, alarm hl:np_load_avg=1.343750/2.0, alarm hl:mem_free=300.000000M/350M, alarm hl:available=1/0 [15:07:52] 3(created) [MNT-1233] Update of SGE; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1233> (DaB.) [15:11:56] 3(commented) [MNT-1233] Update of SGE <10https://jira.toolserver.org/browse/MNT-1233> (DaB.) [15:14:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:18:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.099609/1.95, alarm hl:np_load_avg=1.406738/2.0, alarm hl:mem_free=323.000000M/350M, alarm hl:available=1/0 [15:19:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:24:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:33:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 381496 MB (7% inode=39%): [15:40:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:45:01] Any TS admins around? [15:45:15] I'm still getting spammed with query killer emails, even though my account is disabled. [15:48:39] DaBPunkt: perhaps? [15:49:11] I will look at it later tonight [15:55:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:19:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:24:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:33:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 380940 MB (7% inode=39%): [16:40:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:56:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:03:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.231934/1.95, alarm hl:np_load_avg=1.181152/2.0, alarm hl:mem_free=254.000000M/350M, alarm hl:available=1/0 [17:07:37] Anyone available to look at my issue sooner? I'm still getting mass spammed with query killed emails. [17:17:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:19:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:24:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:33:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 380823 MB (7% inode=39%): [17:38:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.152344/1.95, alarm hl:np_load_avg=1.256836/2.0, alarm hl:mem_free=271.000000M/350M, alarm hl:available=1/0 [17:40:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:45:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:48:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.251465/1.95, alarm hl:np_load_avg=1.233887/2.0, alarm hl:mem_free=283.000000M/350M, alarm hl:available=1/0 [17:57:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:13:03] Load avg. on willow is WARNING: WARNING - load average: 15.76, 15.34, 12.94 [18:15:03] Load avg. on willow is OK: OK - load average: 13.16, 14.86, 13.08 [18:19:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:24:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:33:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 380685 MB (7% inode=39%): [18:39:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.129883/1.95, alarm hl:np_load_avg=1.231445/2.0, alarm hl:mem_free=99.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.129883/2.3, alarm hl:np_load_long=1.356445/2.5, alarm hl:cpu=79.000000/98, alarm hl:mem_free=99.000000M/150M, alarm hl:available=1/0 [18:40:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:41:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:45:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.016113/1.95, alarm hl:np_load_avg=1.147949/2.0, alarm hl:mem_free=265.000000M/350M, alarm hl:available=1/0 [18:57:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:19:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:24:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:33:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 380608 MB (7% inode=39%): [19:37:01] DaBPunkt: Is there a way to stop the spam mails in the meantime? [19:39:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:40:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:40:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:40:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:57:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:00:26] Hello anyone ? I'm being flooded with failures from SGE suddently [20:00:32] I haven't touched anything in days/weeks [20:00:38] /opt/local/bin/cronsub[4]: /sge62/default/common/settings.sh: not found [20:04:39] I've just had one of those too [20:05:12] I'm getting them every minute for 4 different bots [20:19:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:24:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:32:23] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:34:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 379833 MB (7% inode=39%): [20:37:23] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:40:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:40:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:40:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:40:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:57:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:19:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:22:01] hi all [21:22:14] I need an answer :) [21:22:46] what is the value of the environment variable SGE_ROOT ? [21:23:48] in my case that variable is not set [21:24:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:24:55] Nettrom: and you manage to use the qstat command ? [21:25:39] Grimlock-fr: no, I just get an error saying I should set it [21:25:46] okay [21:26:48] according to DaBPunkt's email, there's some SGE maintenance going on right now [21:27:26] Thurs/Fri 19-23 UTC [21:27:27] okay [21:27:38] that's the reason [21:27:50] I read the mail, but forgot the containt ^^ [21:34:00] thanks a lot ! [21:35:15] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 379725 MB (7% inode=39%): [21:35:58] thanks as well, came here for the same reason grimlock-fr [21:40:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:40:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:40:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:40:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:42:43] /sql on cassia is WARNING: DISK WARNING - free space: /sql 124612 MB (10% inode=99%): [21:57:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:07:53] you're most welcome, happy I could help you out :) [22:19:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:24:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:35:14] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 379641 MB (7% inode=39%): [22:40:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:40:24] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:40:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:40:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:48:43] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:54] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:48:54] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:50:02] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:02] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:43] /home on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:52:48] there is a problem with the nfs-service. I look for it [22:54:48] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.284 second response time [22:54:48] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.294 second response time [22:54:57] Sun Grid Engine execd on ortelius is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [22:54:57] /home on hemlock is OK: DISK OK - free space: /home 18502 MB (36% inode=87%): [22:54:57] Sun Grid Engine execd on willow is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [22:54:57] Sun Grid Engine execd on wolfsbane is UNKNOWN: Cannot execute /sge62/bin/sol-amd64/qstat [22:57:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:58:38] NTP on turnera is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.000526 secs [23:00:38] Sun Grid Engine execd on willow is OK: medium-sol@willow disabled: longrun-sol@willow disabled [23:01:38] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius disabled: medium-sol@ortelius disabled [23:01:38] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane disabled: medium-sol@wolfsbane disabled [23:06:39] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (201412 kB) free! [23:07:38] Free Memory on damiana is OK: OK - 8.3% (348300 kB) free. [23:12:38] NTP on turnera is OK: NTP OK: Offset -0.005048 secs [23:19:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:22:38] Free Memory on damiana is WARNING: WARNING - 6.2% (261076 kB) free! [23:22:53] DaBPunkt: Is there a way to stop the spam mails in the meantime? [23:24:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:27:38] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (182564 kB) free! [23:29:38] Free Memory on damiana is WARNING: WARNING - 5.2% (219500 kB) free! [23:31:02] DaB. * Re: [Toolserver-announce] [Toolserver-l] SGE-Maintenance at Thursday and Friday [23:35:39] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 379581 MB (7% inode=39%): [23:35:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.937988/1.95, alarm hl:np_load_avg=0.898438/2.0, alarm hl:mem_free=210.000000M/350M, alarm hl:available=1/0 [23:37:22] Yetanotherx: please forward me such a mail to ts@dabpunkt.eu [23:40:06] nacht ts [23:40:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:41:39] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:47:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.029297/1.95, alarm hl:np_load_avg=0.991699/2.0, alarm hl:mem_free=297.000000M/350M, alarm hl:available=1/0 [23:56:38] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:57:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk