[00:00:37] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.94, 20.13, 17.01 [00:01:29] Load avg. on willow is WARNING: WARNING - load average: 25.02, 19.78, 17.07 [00:11:26] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.496582/1.75, alarm hl:np_load_avg=2.260254/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.496582/1.9, alarm hl:np_load_long=2.141602/2.25, alarm hl:mem_free=125.000000M/200M, alarm hl:available=1/0 [00:23:07] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:23:47] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [00:24:06] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:27:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:31:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [00:32:40] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 577210 MB (10% inode=49%): [00:33:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:33:47] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:33:57] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:43:48] [[File permissions]] 10https://wiki.toolserver.org/w/index.php?diff=6998&oldid=4848&rcid=9218 * Ceradon * (+6) () [00:44:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347875.000000 [00:48:38] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:53:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:54:17] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [00:58:38] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:01:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.822754/1.75, alarm hl:np_load_avg=2.400391/2.0, alarm hl:mem_free=596.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.822754/1.9, alarm hl:np_load_long=2.258301/2.25, alarm hl:mem_free=596.000000M/200M, alarm hl:available=1/0 [01:01:39] Load avg. on willow is WARNING: WARNING - load average: 21.37, 19.40, 18.18 [01:04:29] [[Special:Log/newusers]] create 10 * CarruthTuel * (New user account) [01:24:51] 3(created) [UTRS-97] Newlines are not escaped in JavaScript string values; UTRS: Main Interface; Bug <10https://jira.toolserver.org/browse/UTRS-97> (Chris Howie) [01:27:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:30:54] 3(updated) [DBQ-180] Database Query (question) <10https://jira.toolserver.org/browse/DBQ-180> (Franz Herbach) [01:32:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [01:32:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 577160 MB (10% inode=49%): [01:33:37] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:33:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:34:37] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:37:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.771484/1.75, alarm hl:np_load_avg=2.005371/2.0, alarm hl:mem_free=994.000000M/350M, alarm hl:available=1/0 [01:38:37] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:44:17] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 351475.000000 [01:49:38] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:53:27] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:54:16] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:00:37] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.449219/1.75, alarm hl:np_load_avg=2.442871/2.0, alarm hl:mem_free=1522.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.449219/1.9, alarm hl:np_load_long=2.259766/2.25, alarm hl:mem_free=1522.000000M/200M, alarm hl:available=1/0 [02:00:47] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.00, 22.84, 19.27 [02:01:50] Load avg. on willow is WARNING: WARNING - load average: 21.73, 21.07, 18.88 [02:09:37] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:12:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.753906/1.75, alarm hl:np_load_avg=1.876465/2.0, alarm hl:mem_free=652.000000M/350M, alarm hl:available=1/0 [02:14:38] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:22:59] [[Venice Travel is an excellent source of advice]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6999&rcid=9220 * CarruthTuel * (+2238) (Created page with "Do you realize that Venice is a car-free location? An element that is unconventional within this modern day age, but that's Venice. So, you'll want comfy footwear that will acco...") [02:23:47] Load avg. on willow is OK: OK - load average: 11.70, 13.19, 14.89 [02:27:37] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:32:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [02:32:48] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575441 MB (10% inode=49%): [02:32:48] Load avg. on willow is WARNING: WARNING - load average: 16.60, 15.39, 15.02 [02:33:37] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:33:48] Load avg. on willow is OK: OK - load average: 12.47, 14.42, 14.70 [02:34:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:44:17] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355075.000000 [02:49:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:53:27] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:54:17] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:16:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.920410/1.75, alarm hl:np_load_avg=1.801758/2.0, alarm hl:mem_free=276.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.920410/1.9, alarm hl:np_load_long=1.733887/2.25, alarm hl:mem_free=276.000000M/200M, alarm hl:available=1/0 [03:18:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:27:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:32:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [03:32:48] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575410 MB (10% inode=49%): [03:33:47] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:34:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:44:17] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 358674.000000 [03:49:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:53:37] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:54:17] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:02:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.846191/1.75, alarm hl:np_load_avg=1.842774/2.0, alarm hl:mem_free=770.000000M/350M, alarm hl:available=1/0 [04:03:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:06:49] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.087891/1.00, alarm hl:np_load_long=0.586914/1.50, alarm hl:mem_free=18035.000000M/350M, alarm hl:available=1/0 [04:07:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.062500/1.75, alarm hl:np_load_avg=1.938476/2.0, alarm hl:mem_free=596.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.062500/1.9, alarm hl:np_load_long=1.745605/2.25, alarm hl:mem_free=596.000000M/200M, alarm hl:available=1/0 [04:07:50] Load avg. on willow is WARNING: WARNING - load average: 16.14, 15.47, 13.98 [04:08:49] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:08:49] Load avg. on willow is OK: OK - load average: 13.00, 14.79, 13.83 [04:12:48] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.352539/1.10, alarm hl:np_load_long=0.764648/1.55, alarm hl:mem_free=17757.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.352539/1.00, alarm hl:np_load_long=0.764648/1.50, alarm hl:mem_free=17757.000000M/350M, alarm hl:available=1/0 [04:12:48] Load avg. on willow is WARNING: WARNING - load average: 18.62, 17.22, 15.09 [04:28:47] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:32:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [04:32:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575376 MB (10% inode=49%): [04:33:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:34:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:44:17] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 362276.000000 [04:49:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:53:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:54:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:54:58] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:55:47] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:15:12] * jeremyb waves. someone still there? joan? [05:15:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.440430/1.75, alarm hl:np_load_avg=2.378418/2.0, alarm hl:mem_free=206.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.440430/1.9, alarm hl:np_load_long=2.139160/2.25, alarm hl:mem_free=206.000000M/200M, alarm hl:available=1/0 [05:24:48] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.063476/1.00, alarm hl:np_load_long=0.720703/1.50, alarm hl:mem_free=18660.000000M/350M, alarm hl:available=1/0 [05:25:47] Load avg. on willow is WARNING: WARNING - load average: 14.55, 15.97, 16.65 [05:26:47] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:27:46] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:29:47] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:32:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [05:32:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575284 MB (10% inode=49%): [05:34:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:34:47] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:36:47] Load avg. on willow is OK: OK - load average: 12.80, 13.34, 14.88 [05:44:17] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 365878.000000 [05:44:47] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.103516/1.10, alarm hl:np_load_long=0.751953/1.55, alarm hl:mem_free=19397.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.103516/1.00, alarm hl:np_load_long=0.751953/1.50, alarm hl:mem_free=19397.000000M/350M, alarm hl:available=1/0 [05:47:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.161133/1.75, alarm hl:np_load_avg=1.894531/2.0, alarm hl:mem_free=687.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.161133/1.9, alarm hl:np_load_long=1.845703/2.25, alarm hl:mem_free=687.000000M/200M, alarm hl:available=1/0 [05:47:48] Load avg. on willow is WARNING: WARNING - load average: 17.81, 15.39, 14.86 [05:49:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:54:37] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:55:17] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:56:21] hey Tanvir, have an account? [05:56:56] oh, i guess that was an autojoin [06:04:48] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:12:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.272949/1.75, alarm hl:np_load_avg=2.284668/2.0, alarm hl:mem_free=446.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.272949/1.9, alarm hl:np_load_long=2.124512/2.25, alarm hl:mem_free=446.000000M/200M, alarm hl:available=1/0 [06:20:47] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.19, 23.79, 20.13 [06:21:47] Load avg. on willow is WARNING: WARNING - load average: 16.13, 21.55, 19.57 [06:29:47] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:29:47] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:30:48] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.81, 21.52, 19.49 [06:32:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [06:32:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.250488/1.75, alarm hl:np_load_avg=2.419434/2.0, alarm hl:mem_free=762.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.250488/1.9, alarm hl:np_load_long=2.362305/2.25, alarm hl:mem_free=762.000000M/200M, alarm hl:available=1/0 [06:33:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575249 MB (10% inode=49%): [06:34:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:34:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:39:48] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:44:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 369482.000000 [06:48:48] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.902344/1.10, alarm hl:np_load_long=1.092774/1.55, alarm hl:mem_free=19560.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.902344/1.00, alarm hl:np_load_long=1.092774/1.50, alarm hl:mem_free=19560.000000M/350M, alarm hl:available=1/0 [06:50:47] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:50:47] Load avg. on willow is WARNING: WARNING - load average: 14.83, 15.04, 16.05 [06:50:57] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:54:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:55:16] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:59:48] Load avg. on willow is OK: OK - load average: 11.76, 13.44, 14.80 [07:02:49] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.498047/1.10, alarm hl:np_load_long=0.953125/1.55, alarm hl:mem_free=18368.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.498047/1.00, alarm hl:np_load_long=0.953125/1.50, alarm hl:mem_free=18368.000000M/350M, alarm hl:available=1/0 [07:02:49] Load avg. on willow is WARNING: WARNING - load average: 14.29, 15.17, 15.36 [07:07:49] Load avg. on willow is OK: OK - load average: 12.41, 14.30, 14.98 [07:28:48] Load avg. on ortelius is WARNING: WARNING - load average: 22.84, 12.74, 7.32 [07:29:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:30:48] Load avg. on ortelius is OK: OK - load average: 9.68, 12.09, 7.82 [07:32:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [07:33:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575218 MB (10% inode=49%): [07:34:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:35:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:37:47] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.007812/1.10, alarm hl:np_load_long=1.614258/1.55, alarm hl:mem_free=19011.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.007812/1.00, alarm hl:np_load_long=1.614258/1.50, alarm hl:mem_free=19011.000000M/350M, alarm hl:available=1/0 [07:40:47] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:44:27] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 373086.000000 [07:45:47] Load avg. on willow is WARNING: WARNING - load average: 21.47, 18.24, 16.53 [07:48:47] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.179688/1.10, alarm hl:np_load_long=1.730469/1.55, alarm hl:mem_free=19726.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.179688/1.00, alarm hl:np_load_long=1.730469/1.50, alarm hl:mem_free=19726.000000M/350M, alarm hl:available=1/0 [07:51:17] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:52:48] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:54:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:55:19] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:58:48] Load avg. on willow is OK: OK - load average: 11.28, 13.56, 14.98 [08:02:48] Load avg. on willow is WARNING: WARNING - load average: 13.93, 15.28, 15.46 [08:04:06] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:09:48] Load avg. on willow is OK: OK - load average: 13.80, 14.36, 14.96 [08:29:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:32:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [08:33:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575058 MB (10% inode=49%): [08:34:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:35:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:38:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.951660/1.75, alarm hl:np_load_avg=1.791992/2.0, alarm hl:mem_free=791.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.951660/1.9, alarm hl:np_load_long=1.760254/2.25, alarm hl:mem_free=791.000000M/200M, alarm hl:available=1/0 [08:39:47] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:44:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 376687.000000 [08:47:55] 3(created) [ACCAPP-491] Support WIWOSM; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-491> (Simon Legner) [08:51:17] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:54:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:56:17] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:02:48] Load avg. on willow is WARNING: WARNING - load average: 16.10, 16.49, 15.19 [09:02:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.020020/1.75, alarm hl:np_load_avg=2.063477/2.0, alarm hl:mem_free=1075.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.020020/1.9, alarm hl:np_load_long=1.898926/2.25, alarm hl:mem_free=1075.000000M/200M, alarm hl:available=1/0 [09:04:06] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:05:01] Load avg. on willow is OK: OK - load average: 11.73, 14.77, 14.71 [09:07:57] Load avg. on willow is WARNING: WARNING - load average: 14.88, 15.58, 15.10 [09:12:58] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.192383/1.10, alarm hl:np_load_long=0.840820/1.55, alarm hl:mem_free=19154.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.192383/1.00, alarm hl:np_load_long=0.840820/1.50, alarm hl:mem_free=19154.000000M/350M, alarm hl:available=1/0 [09:13:58] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:29:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:31:57] Load avg. on willow is WARNING: WARNING - load average: 16.87, 16.15, 16.04 [09:32:27] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [09:33:47] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:33:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 575059 MB (10% inode=49%): [09:35:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:36:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:36:40] [[Special:Log/newusers]] create 10 * AbercrombieJMBoy * (New user account) [09:44:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 380304.000000 [09:45:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.321289/1.10, alarm hl:np_load_long=0.911133/1.55, alarm hl:mem_free=19142.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.321289/1.00, alarm hl:np_load_long=0.911133/1.50, alarm hl:mem_free=19142.000000M/350M, alarm hl:available=1/0 [09:47:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:52:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.258301/1.75, alarm hl:np_load_avg=2.230469/2.0, alarm hl:mem_free=1036.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.258301/1.9, alarm hl:np_load_long=2.152344/2.25, alarm hl:mem_free=1036.000000M/200M, alarm hl:available=1/0 [09:52:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:54:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:56:23] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:23:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.90, 22.25, 19.86 [10:30:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:33:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [10:34:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 574822 MB (10% inode=49%): [10:36:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:36:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:44:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 383905.000000 [10:52:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.945312/1.75, alarm hl:np_load_avg=5.212402/2.0, alarm hl:mem_free=690.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.945312/1.9, alarm hl:np_load_long=4.475098/2.25, alarm hl:mem_free=690.000000M/200M, alarm hl:available=1/0 [10:52:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:54:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:55:53] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:54] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:56:32] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:56:32] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [10:57:22] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:58:12] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:12] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:04:12] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:23:53] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:24:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 51.25, 48.14, 43.39 [11:30:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:33:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [11:33:52] 3(created) [OSM-12] OSM ignores the scale-parameter; OSM; Minor Bug <10https://jira.toolserver.org/browse/OSM-12> [11:34:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 574161 MB (10% inode=49%): [11:36:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:36:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:44:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 387505.000000 [11:44:55] 3(created) [OSM-13] Add map layers from the Norwegian Mapping Authority; OSM; Minor New Feature <10https://jira.toolserver.org/browse/OSM-13> [11:46:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.870117/1.10, alarm hl:np_load_long=1.285156/1.55, alarm hl:mem_free=18832.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.870117/1.00, alarm hl:np_load_long=1.285156/1.50, alarm hl:mem_free=18832.000000M/350M, alarm hl:available=1/0 [11:49:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:52:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=7.680176/1.75, alarm hl:np_load_avg=6.542969/2.0, alarm hl:mem_free=291.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=7.680176/1.9, alarm hl:np_load_long=5.646973/2.25, alarm hl:mem_free=291.000000M/200M, alarm hl:available=1/0 [11:52:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:55:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:57:32] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:03:01] 3(created) [OSM-14] Symlink no to nb; OSM; Minor Improvement <10https://jira.toolserver.org/browse/OSM-14> [12:25:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.05, 28.53, 33.91 [12:30:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:33:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [12:34:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573228 MB (10% inode=49%): [12:36:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:36:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:41:53] 3(commented) [OSM-13] Add map layers from the Norwegian Mapping Authority <10https://jira.toolserver.org/browse/OSM-13> [12:44:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 391107.000000 [12:52:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.850098/1.75, alarm hl:np_load_avg=4.274902/2.0, alarm hl:mem_free=306.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.850098/1.9, alarm hl:np_load_long=4.152344/2.25, alarm hl:mem_free=306.000000M/200M, alarm hl:available=1/0 [12:52:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:56:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:57:33] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:19:12] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:25:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.75, 40.94, 41.04 [13:30:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:33:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [13:33:42] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:34:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 571617 MB (10% inode=49%): [13:36:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:37:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:45:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 394768.000000 [13:52:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.313477/1.75, alarm hl:np_load_avg=4.936523/2.0, alarm hl:mem_free=240.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.313477/1.9, alarm hl:np_load_long=4.901855/2.25, alarm hl:mem_free=240.000000M/200M, alarm hl:available=1/0 [13:53:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:56:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:57:32] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [14:00:12] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:01:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.997559/1.75, alarm hl:np_load_avg=4.696777/2.0, alarm hl:mem_free=1401.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.997559/1.9, alarm hl:np_load_long=4.778809/2.25, alarm hl:mem_free=1401.000000M/200M, alarm hl:available=1/0 [14:04:03] hello all [14:04:23] hi DaBPunkt [14:22:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.936524/1.10, alarm hl:np_load_long=0.986328/1.55, alarm hl:mem_free=18859.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.936524/1.00, alarm hl:np_load_long=0.986328/1.50, alarm hl:mem_free=18859.000000M/350M, alarm hl:available=1/0 [14:24:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:25:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.89, 34.27, 35.73 [14:31:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:33:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [14:34:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 574452 MB (10% inode=49%): [14:36:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:37:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:45:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 398368.000000 [14:53:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:56:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:58:32] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [15:01:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.305664/1.75, alarm hl:np_load_avg=3.913086/2.0, alarm hl:mem_free=1033.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.305664/1.9, alarm hl:np_load_long=3.917969/2.25, alarm hl:mem_free=1033.000000M/200M, alarm hl:available=1/0 [15:10:59] 3(commented) [OSM-12] OSM ignores the scale-parameter <10https://jira.toolserver.org/browse/OSM-12> (Kolossos ) [15:11:04] 3(assigned) [OSM-11] WIWOSM https <10https://jira.toolserver.org/browse/OSM-11> (Kolossos ) [15:12:53] 3(assigned) [OSM-13] Add map layers from the Norwegian Mapping Authority <10https://jira.toolserver.org/browse/OSM-13> (Kolossos ) [15:18:55] 3(commented) [OSM-13] Add map layers from the Norwegian Mapping Authority <10https://jira.toolserver.org/browse/OSM-13> (Kolossos ) [15:25:13] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.21, 25.91, 28.20 [15:25:54] 3(resolved) [OSM-14] Symlink no to nb <10https://jira.toolserver.org/browse/OSM-14> (Kolossos ) [15:31:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:33:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [15:34:13] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 574231 MB (10% inode=49%): [15:36:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:37:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:46:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 402029.000000 [15:53:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:56:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:58:33] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [15:59:54] 3(created) [TS-1350] Add new wikis (bewikisource, guwikisource, lezwiki, slwikiversity); Toolserver: Toolserver-DB; Task <10https://jira.toolserver.org/browse/TS-1350> (Jesse Plamondon-Willard) [16:01:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.830078/1.75, alarm hl:np_load_avg=2.504395/2.0, alarm hl:mem_free=651.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.830078/1.9, alarm hl:np_load_long=2.711914/2.25, alarm hl:mem_free=651.000000M/200M, alarm hl:available=1/0 [16:17:12] Load avg. on willow is WARNING: WARNING - load average: 17.97, 19.21, 19.96 [16:30:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:31:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:33:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.937500/1.75, alarm hl:np_load_avg=2.120605/2.0, alarm hl:mem_free=994.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.937500/1.9, alarm hl:np_load_long=2.236816/2.25, alarm hl:mem_free=994.000000M/200M, alarm hl:available=1/0 [16:33:54] Hmm, is there any particular reason file_get_contents( 'http://status.toolserver.org/json' ) would time out? [16:34:13] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 574051 MB (10% inode=49%): [16:34:13] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.748047/1.10, alarm hl:np_load_long=0.779297/1.55, alarm hl:mem_free=18340.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.748047/1.00, alarm hl:np_load_long=0.779297/1.50, alarm hl:mem_free=18340.000000M/350M, alarm hl:available=1/0 [16:34:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [16:35:12] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [16:35:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:36:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:38:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:46:12] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.084961/1.00, alarm hl:np_load_long=0.861328/1.50, alarm hl:mem_free=18104.000000M/350M, alarm hl:available=1/0 [16:46:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 405629.000000 [16:54:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:56:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:57:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.092285/1.75, alarm hl:np_load_avg=2.113281/2.0, alarm hl:mem_free=755.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.092285/1.9, alarm hl:np_load_long=2.126953/2.25, alarm hl:mem_free=755.000000M/200M, alarm hl:available=1/0 [16:59:32] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [17:09:22] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:13:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.214844/1.75, alarm hl:np_load_avg=2.210449/2.0, alarm hl:mem_free=402.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.214844/1.9, alarm hl:np_load_long=2.165039/2.25, alarm hl:mem_free=402.000000M/200M, alarm hl:available=1/0 [17:18:12] Load avg. on willow is WARNING: WARNING - load average: 13.34, 15.42, 16.51 [17:18:22] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:28:22] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.082031/1.00, alarm hl:np_load_long=0.816406/1.50, alarm hl:mem_free=18476.000000M/350M, alarm hl:available=1/0 [17:30:22] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:31:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:34:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573957 MB (10% inode=49%): [17:34:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [17:36:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:38:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:45:22] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.416992/1.10, alarm hl:np_load_long=0.938477/1.55, alarm hl:mem_free=18358.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.416992/1.00, alarm hl:np_load_long=0.938477/1.50, alarm hl:mem_free=18358.000000M/350M, alarm hl:available=1/0 [17:47:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 409239.000000 [17:52:13] Load avg. on willow is OK: OK - load average: 12.32, 14.28, 14.91 [17:54:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:56:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:59:32] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [18:02:14] Load avg. on willow is WARNING: WARNING - load average: 20.45, 18.45, 16.34 [18:04:52] 3(commented) [OSM-12] OSM ignores the scale-parameter <10https://jira.toolserver.org/browse/OSM-12> [18:13:22] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.020508/1.00, alarm hl:np_load_long=0.732422/1.50, alarm hl:mem_free=18261.000000M/350M, alarm hl:available=1/0 [18:17:22] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:21:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.187500/1.75, alarm hl:np_load_avg=2.170898/2.0, alarm hl:mem_free=597.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.187500/1.9, alarm hl:np_load_long=2.166504/2.25, alarm hl:mem_free=597.000000M/200M, alarm hl:available=1/0 [18:27:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.416992/1.10, alarm hl:np_load_long=0.958985/1.55, alarm hl:mem_free=17930.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.416992/1.00, alarm hl:np_load_long=0.958985/1.50, alarm hl:mem_free=17930.000000M/350M, alarm hl:available=1/0 [18:31:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:34:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573870 MB (10% inode=49%): [18:34:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [18:36:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:38:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:47:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 412838.000000 [18:52:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.575195/1.10, alarm hl:np_load_long=1.003906/1.55, alarm hl:mem_free=18683.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.575195/1.00, alarm hl:np_load_long=1.003906/1.50, alarm hl:mem_free=18683.000000M/350M, alarm hl:available=1/0 [18:53:43] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:54:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:56:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:59:42] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:00:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.767578/1.10, alarm hl:np_load_long=1.042969/1.55, alarm hl:mem_free=18597.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.767578/1.00, alarm hl:np_load_long=1.042969/1.50, alarm hl:mem_free=18597.000000M/350M, alarm hl:available=1/0 [19:02:22] Load avg. on willow is WARNING: WARNING - load average: 20.35, 19.05, 18.41 [19:02:42] [[User servers]] !M 10https://wiki.toolserver.org/w/index.php?diff=7000&oldid=5190&rcid=9222 * Wiegels * (-1) (grammar) [19:13:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.75, 21.82, 20.06 [19:16:22] Load avg. on willow is WARNING: WARNING - load average: 19.07, 20.77, 19.97 [19:21:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.099121/1.75, alarm hl:np_load_avg=2.303711/2.0, alarm hl:mem_free=414.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.099121/1.9, alarm hl:np_load_long=2.387695/2.25, alarm hl:mem_free=414.000000M/200M, alarm hl:available=1/0 [19:29:12] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:31:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:34:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573781 MB (10% inode=49%): [19:34:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [19:37:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:38:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:45:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.25, 21.61, 19.73 [19:46:21] Load avg. on willow is WARNING: WARNING - load average: 24.84, 21.65, 19.86 [19:47:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 416440.000000 [19:48:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.48, 22.12, 20.23 [19:55:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:56:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:59:43] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:08:22] Load avg. on willow is WARNING: WARNING - load average: 18.26, 19.38, 19.28 [20:10:32] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:32] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:42] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:02] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:13] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:13] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:23] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:23] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:23] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:32] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [20:11:32] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:32] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:32] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:44] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [20:11:53] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:53] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:11:53] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:03] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [20:12:03] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [20:12:12] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [20:12:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.35, 21.34, 20.04 [20:13:02] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [20:13:03] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [20:14:12] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 2743 MB (99% inode=99%): [20:14:12] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 181319 MB (18% inode=99%): [20:14:12] Load avg. on z-dat-s6-a is OK: OK - load average: 0.14, 0.82, 1.71 [20:14:12] MySQL slave on z-dat-s6-a is OK: Uptime: 348815 Threads: 33 Questions: 89680984 Slow queries: 22687 Opens: 794107 Flush tables: 2 Open tables: 1802 Queries per second avg: 257.101 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 377 [20:14:12] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 249.000000 [20:14:13] SMF on z-dat-s6-a is OK: OK - all services online [20:14:13] SMF on z-dat-s4-a is OK: OK - all services online [20:14:14] SMF on hyacinth is OK: OK - all services online [20:14:14] SMF on z-dat-s7-a is OK: OK - all services online [20:14:22] MySQL slave on z-dat-s3-a is OK: Uptime: 3408243 Threads: 15 Questions: 3771412144 Slow queries: 210462 Opens: 26619506 Flush tables: 1 Open tables: 16384 Queries per second avg: 1106.556 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 373 [20:14:22] MySQL on z-dat-s3-a is OK: Uptime: 3408243 Threads: 15 Questions: 3771412147 Slow queries: 210462 Opens: 26619506 Flush tables: 1 Open tables: 16384 Queries per second avg: 1106.556 [20:14:22] / on z-dat-s6-a is OK: DISK OK - free space: / 8511 MB (28% inode=85%): [20:14:22] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:14:22] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:14:31] MySQL slave on z-dat-s7-a is OK: Uptime: 3840674 Threads: 14 Questions: 862617559 Slow queries: 116509 Opens: 6552883 Flush tables: 1 Open tables: 7113 Queries per second avg: 224.600 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 341 [20:14:31] MySQL on z-dat-s7-a is OK: Uptime: 3840674 Threads: 14 Questions: 862617566 Slow queries: 116509 Opens: 6552883 Flush tables: 1 Open tables: 7113 Queries per second avg: 224.600 [20:14:31] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:14:32] SMF on z-dat-s3-a is OK: OK - all services online [20:14:42] MySQL on z-dat-s6-a is OK: Uptime: 348841 Threads: 26 Questions: 89683907 Slow queries: 22722 Opens: 794165 Flush tables: 2 Open tables: 1802 Queries per second avg: 257.91 [20:14:43] SMTP on z-dat-s7-a is OK: SMTP OK - 0.043 sec. response time [20:14:44] SMTP on z-dat-s6-a is OK: SMTP OK - 0.101 sec. response time [20:14:54] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:19:54] 3(commented) [ACCAPP-481] I am active user in http://fa.wiktionary.org and i want this account to run bot on this wiki also updating it's statics. <10https://jira.toolserver.org/browse/ACCAPP-481> (DaB.) [20:21:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.617188/1.75, alarm hl:np_load_avg=2.733887/2.0, alarm hl:mem_free=733.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.617188/1.9, alarm hl:np_load_long=2.614746/2.25, alarm hl:mem_free=733.000000M/200M, alarm hl:available=1/0 [20:23:52] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:31:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:35:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573686 MB (10% inode=49%): [20:35:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [20:37:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:38:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:40:05] ? [20:41:32] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:12] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:42:22] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:42:31] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:32] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:40] borked? [20:42:41] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:42:43] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:42:43] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:42:53] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:43:03] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:43:03] SMF on z-dat-s6-a is OK: OK - all services online [20:43:12] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 2726 MB (99% inode=99%): [20:43:12] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 183071 MB (18% inode=99%): [20:43:12] Load avg. on z-dat-s6-a is OK: OK - load average: 0.67, 1.26, 1.66 [20:43:22] / on z-dat-s6-a is OK: DISK OK - free space: / 8511 MB (28% inode=85%): [20:43:22] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:43:22] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:43:22] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:47:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 420043.000000 [20:55:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:57:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:59:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:07:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.647461/1.10, alarm hl:np_load_long=0.970703/1.55, alarm hl:mem_free=18167.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.647461/1.00, alarm hl:np_load_long=0.970703/1.50, alarm hl:mem_free=18167.000000M/350M, alarm hl:available=1/0 [21:12:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.37, 22.09, 21.93 [21:21:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.637207/1.75, alarm hl:np_load_avg=2.825684/2.0, alarm hl:mem_free=139.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.637207/1.9, alarm hl:np_load_long=2.779785/2.25, alarm hl:mem_free=139.000000M/200M, alarm hl:available=1/0 [21:31:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:35:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [21:35:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 573527 MB (10% inode=49%): [21:37:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:38:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:47:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 423647.000000 [21:51:22] / on ortelius is WARNING: DISK WARNING - free space: / 5687 MB (18% inode=93%): [21:56:52] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:57:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:59:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [22:04:43] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [22:09:12] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:12:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 19.65, 19.47, 20.43 [22:19:22] Load avg. on willow is WARNING: WARNING - load average: 17.22, 18.73, 19.79 [22:21:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.229980/1.75, alarm hl:np_load_avg=2.331055/2.0, alarm hl:mem_free=420.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.229980/1.9, alarm hl:np_load_long=2.452637/2.25, alarm hl:mem_free=420.000000M/200M, alarm hl:available=1/0 [22:32:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.74, 21.19, 20.01 [22:32:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:35:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [22:35:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 564498 MB (10% inode=49%): [22:36:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.556641/1.10, alarm hl:np_load_long=1.054688/1.55, alarm hl:mem_free=19259.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.556641/1.00, alarm hl:np_load_long=1.054688/1.50, alarm hl:mem_free=19259.000000M/350M, alarm hl:available=1/0 [22:38:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:38:45] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:38:53] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [22:38:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:39:22] Load avg. on willow is WARNING: WARNING - load average: 18.44, 19.87, 19.88 [22:40:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.92, 21.39, 20.38 [22:47:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 427248.000000 [22:51:23] / on ortelius is WARNING: DISK WARNING - free space: / 5830 MB (19% inode=93%): [22:56:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:57:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:00:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [23:09:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:12:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.203613/1.75, alarm hl:np_load_avg=2.150391/2.0, alarm hl:mem_free=238.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.203613/1.9, alarm hl:np_load_long=2.220703/2.25, alarm hl:mem_free=238.000000M/200M, alarm hl:available=1/0 [23:14:46] nacht ts [23:20:33] Load avg. on willow is WARNING: WARNING - load average: 19.66, 16.86, 17.26 [23:20:52] 3(created) [MNT-1229] space on / at ortelius became small; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1229> (DaB.) [23:20:57] 3(updated) [MNT-1229] space on / at ortelius became small <10https://jira.toolserver.org/browse/MNT-1229> (DaB.) [23:22:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:32:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:35:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [23:35:53] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 564034 MB (10% inode=49%): [23:38:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:38:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:44:53] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.977539/1.10, alarm hl:np_load_long=0.903320/1.55, alarm hl:mem_free=19921.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.977539/1.00, alarm hl:np_load_long=0.903320/1.50, alarm hl:mem_free=19921.000000M/350M, alarm hl:available=1/0 [23:47:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 430855.000000 [23:47:53] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [23:51:31] / on ortelius is WARNING: DISK WARNING - free space: / 5789 MB (19% inode=93%): [23:52:32] Load avg. on willow is OK: OK - load average: 11.06, 13.66, 14.88 [23:57:01] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:57:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default