[08:15:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:16:57] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [08:16:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:18:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:26:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450102 MB (8% inode=43%): [08:27:15] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 346408.000000 [08:29:56] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.123047/1.95, alarm hl:np_load_avg=2.298828/2.0, alarm hl:mem_free=582.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.123047/2.3, alarm hl:np_load_long=2.541016/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=582.000000M/150M, alarm hl:available=1/0 [08:31:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.21, 22.79, 21.78 [08:46:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 338318 [08:50:35] Load avg. on willow is WARNING: WARNING - load average: 16.14, 18.25, 19.73 [09:06:55] 3(created) [ACCAPP-497] tools for el.wikipedia; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-497> (geraki) [09:15:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:17:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:17:56] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [09:18:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:26:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449962 MB (8% inode=43%): [09:27:15] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 327972.000000 [09:29:56] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.113281/1.95, alarm hl:np_load_avg=2.228027/2.0, alarm hl:mem_free=127.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.113281/2.3, alarm hl:np_load_long=2.331055/2.5, alarm hl:cpu=94.700000/98, alarm hl:mem_free=127.000000M/150M, alarm hl:available=1/0 [09:30:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.55, 22.55, 20.23 [09:33:36] Load avg. on willow is WARNING: WARNING - load average: 16.89, 20.01, 19.67 [09:40:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.44, 20.67, 19.63 [09:46:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:50:05] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 323244 [10:00:35] Load avg. on willow is WARNING: WARNING - load average: 27.75, 19.26, 18.29 [10:15:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:18:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:18:05] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [10:19:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:26:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449848 MB (8% inode=43%): [10:27:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 314718.000000 [10:29:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.095215/1.95, alarm hl:np_load_avg=2.270996/2.0, alarm hl:mem_free=305.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.095215/2.3, alarm hl:np_load_long=2.315918/2.5, alarm hl:cpu=99.400000/98, alarm hl:mem_free=305.000000M/150M, alarm hl:available=1/0 [10:45:02] Danny B. * Re: [Toolserver-l] Rules clarifications [10:47:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:50:26] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 309001 [10:59:45] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:00:36] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.28, 21.04, 19.55 [11:01:35] Load avg. on willow is WARNING: WARNING - load average: 23.93, 20.48, 19.45 [11:09:04] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41171 MB (10% inode=99%): [11:10:05] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 47395 MB (11% inode=99%): [11:12:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.41, 21.50, 20.08 [11:15:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:18:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:18:16] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [11:18:36] Load avg. on willow is WARNING: WARNING - load average: 17.66, 19.72, 19.82 [11:19:35] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:20:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:27:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 296189.000000 [11:27:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449705 MB (8% inode=43%): [11:28:05] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40898 MB (10% inode=99%): [11:30:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.139648/1.95, alarm hl:np_load_avg=2.245117/2.0, alarm hl:mem_free=290.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.139648/2.3, alarm hl:np_load_long=2.346191/2.5, alarm hl:cpu=99.300000/98, alarm hl:mem_free=290.000000M/150M, alarm hl:available=1/0 [11:30:35] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.95, 20.93, 19.73 [11:34:29] [[Special:Log/newusers]] create 10 * Search * (New user account) [11:36:03] [[User:Search]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7024&rcid=9256 * Search * (+2710) (Created page with "في الشرح السابق تكلمت عن كيف نكسب باك لينك قوي من خلال التسجيل في منتديات ثم وضع رابط موقعنا بالمل...") [11:40:55] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.235352/1.10, alarm hl:np_load_long=0.794922/1.55, alarm hl:mem_free=18186.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.235352/1.00, alarm hl:np_load_long=0.794922/1.50, alarm hl:mem_free=18186.000000M/350M, alarm hl:available=1/0 [11:47:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:50:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 285608 [11:50:36] Load avg. on willow is WARNING: WARNING - load average: 23.92, 20.95, 19.84 [11:57:55] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:00:56] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.099609/1.00, alarm hl:np_load_long=1.507812/1.50, alarm hl:mem_free=18221.000000M/350M, alarm hl:available=1/0 [12:03:55] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:15:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:18:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:18:15] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [12:20:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:27:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 269495.000000 [12:27:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 446691 MB (8% inode=43%): [12:30:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.160156/1.95, alarm hl:np_load_avg=2.338379/2.0, alarm hl:mem_free=126.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.160156/2.3, alarm hl:np_load_long=2.355469/2.5, alarm hl:cpu=99.500000/98, alarm hl:mem_free=126.000000M/150M, alarm hl:available=1/0 [12:47:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:50:44] Load avg. on willow is WARNING: WARNING - load average: 26.68, 20.00, 18.84 [12:51:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 260245 [13:00:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:03:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.131836/1.95, alarm hl:np_load_avg=2.213867/2.0, alarm hl:mem_free=494.000000M/350M, alarm hl:available=1/0 [13:12:55] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.155274/1.10, alarm hl:np_load_long=0.894531/1.55, alarm hl:mem_free=18068.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.155274/1.00, alarm hl:np_load_long=0.894531/1.50, alarm hl:mem_free=18068.000000M/350M, alarm hl:available=1/0 [13:13:56] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:16:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:18:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:18:15] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [13:20:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:27:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447861 MB (8% inode=43%): [13:28:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 249456.000000 [13:30:56] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.128906/1.10, alarm hl:np_load_long=0.915039/1.55, alarm hl:mem_free=18422.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.128906/1.00, alarm hl:np_load_long=0.915039/1.50, alarm hl:mem_free=18422.000000M/350M, alarm hl:available=1/0 [13:47:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:50:45] Load avg. on willow is WARNING: WARNING - load average: 21.11, 17.99, 17.62 [13:51:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 242700 [13:57:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:03:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.072754/1.95, alarm hl:np_load_avg=2.135742/2.0, alarm hl:mem_free=724.000000M/350M, alarm hl:available=1/0 [14:05:16] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:13:05] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.398438/1.10, alarm hl:np_load_long=0.888672/1.55, alarm hl:mem_free=18160.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.398438/1.00, alarm hl:np_load_long=0.888672/1.50, alarm hl:mem_free=18160.000000M/350M, alarm hl:available=1/0 [14:14:05] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:16:14] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:18:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:18:15] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [14:20:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:28:15] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.888184/1.95, alarm hl:np_load_avg=1.952149/2.0, alarm hl:mem_free=273.000000M/350M, alarm hl:available=1/0 [14:28:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 240707.000000 [14:28:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447625 MB (8% inode=43%): [14:35:15] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:43:16] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.849609/1.95, alarm hl:np_load_avg=1.840820/2.0, alarm hl:mem_free=263.000000M/350M, alarm hl:available=1/0 [14:47:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:51:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 239171 [14:51:45] Load avg. on willow is WARNING: WARNING - load average: 16.64, 16.38, 15.96 [15:13:05] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:06] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:06] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:14] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:36] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:36] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:36] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:36] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:36] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:36] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:46] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:46] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:46] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:46] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:46] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:46] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:46] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:13:56] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:05] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:05] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:05] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:05] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:05] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:06] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:15] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [15:14:15] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [15:14:15] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [15:14:15] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:25] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:25] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:25] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:25] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:14:25] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:15:06] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [15:15:06] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [15:15:25] SMF on z-dat-s7-a is OK: OK - all services online [15:15:25] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 2558 MB (99% inode=99%): [15:15:25] / on z-dat-s3-a is OK: DISK OK - free space: / 8488 MB (28% inode=85%): [15:15:25] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 2559 MB (99% inode=99%): [15:15:26] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 179762 MB (18% inode=99%): [15:15:26] Load avg. on z-dat-s4-a is OK: OK - load average: 2.32, 2.77, 2.98 [15:15:26] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 2558 MB (99% inode=99%): [15:15:26] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 99371 MB (24% inode=99%): [15:15:26] Load avg. on z-dat-s7-a is OK: OK - load average: 2.32, 2.77, 2.98 [15:15:27] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 103726 MB (25% inode=99%): [15:15:27] MySQL on z-dat-s3-a is OK: Uptime: 4081513 Threads: 23 Questions: 4641302501 Slow queries: 228059 Opens: 34226627 Flush tables: 1 Open tables: 16384 Queries per second avg: 1137.152 [15:15:28] MySQL slave on z-dat-s3-a is OK: Uptime: 4081513 Threads: 23 Questions: 4641302502 Slow queries: 228059 Opens: 34226627 Flush tables: 1 Open tables: 16384 Queries per second avg: 1137.152 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 312 [15:15:35] MySQL on z-dat-s7-a is OK: Uptime: 4513937 Threads: 12 Questions: 1040715487 Slow queries: 130642 Opens: 8237914 Flush tables: 1 Open tables: 7309 Queries per second avg: 230.556 [15:15:35] MySQL slave on z-dat-s7-a is OK: Uptime: 4513937 Threads: 12 Questions: 1040715488 Slow queries: 130642 Opens: 8237914 Flush tables: 1 Open tables: 7309 Queries per second avg: 230.556 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 289 [15:15:35] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 145.000000 [15:15:35] / on z-dat-s6-a is OK: DISK OK - free space: / 8490 MB (28% inode=85%): [15:15:35] Load avg. on z-dat-s6-a is OK: OK - load average: 2.49, 2.77, 2.97 [15:15:35] Load avg. on z-dat-s3-a is OK: OK - load average: 2.49, 2.77, 2.97 [15:15:35] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:15:36] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:15:56] SMTP on z-dat-s4-a is OK: SMTP OK - 0.003 sec. response time [15:15:56] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 179759 MB (18% inode=99%): [15:15:56] / on z-dat-s4-a is OK: DISK OK - free space: / 8488 MB (28% inode=85%): [15:15:56] SMTP on z-dat-s6-a is OK: SMTP OK - 0.002 sec. response time [15:15:56] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:15:56] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 2635 MB (99% inode=99%): [15:15:56] / on z-dat-s7-a is OK: DISK OK - free space: / 8488 MB (28% inode=85%): [15:16:05] SMTP on z-dat-s3-a is OK: SMTP OK - 0.004 sec. response time [15:16:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:18:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:18:14] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [15:20:14] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:20:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:25:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.865723/1.95, alarm hl:np_load_avg=1.933594/2.0, alarm hl:mem_free=287.000000M/350M, alarm hl:available=1/0 [15:27:14] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:28:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 228469.000000 [15:29:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447413 MB (8% inode=43%): [15:32:00] hello all [15:47:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:51:15] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.755371/1.95, alarm hl:np_load_avg=1.908203/2.0, alarm hl:mem_free=345.000000M/350M, alarm hl:available=1/0 [15:51:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 221045 [15:51:45] Load avg. on willow is WARNING: WARNING - load average: 12.84, 14.86, 15.44 [15:52:14] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:54:46] Load avg. on willow is OK: OK - load average: 12.41, 13.68, 14.86 [15:55:15] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.672363/1.95, alarm hl:np_load_avg=1.719238/2.0, alarm hl:mem_free=173.000000M/350M, alarm hl:available=1/0 [15:58:15] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:02:46] Load avg. on willow is WARNING: WARNING - load average: 15.45, 15.75, 15.26 [16:09:45] Load avg. on willow is OK: OK - load average: 13.58, 14.71, 14.96 [16:16:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:18:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:18:15] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [16:20:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.766601/1.95, alarm hl:np_load_avg=1.890137/2.0, alarm hl:mem_free=207.000000M/350M, alarm hl:available=1/0 [16:20:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:23:05] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 600 MB (16% inode=99%): [16:26:04] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 206 MB (6% inode=99%): [16:27:05] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 527 MB (15% inode=99%): [16:29:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 200214.000000 [16:29:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447348 MB (8% inode=43%): [16:30:15] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:30:25] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:31:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:31:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.010254/1.95, alarm hl:np_load_avg=1.921387/2.0, alarm hl:mem_free=505.000000M/350M, alarm hl:available=1/0 [16:31:45] Load avg. on willow is WARNING: WARNING - load average: 13.96, 14.93, 15.25 [16:36:15] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:38:45] Load avg. on willow is OK: OK - load average: 13.93, 14.55, 14.97 [16:42:46] Load avg. on willow is WARNING: WARNING - load average: 16.59, 15.42, 15.21 [16:47:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:51:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 191890 [16:59:45] Load avg. on willow is OK: OK - load average: 13.00, 13.98, 14.87 [17:02:46] Load avg. on willow is WARNING: WARNING - load average: 14.15, 14.92, 15.16 [17:04:46] Load avg. on willow is OK: OK - load average: 13.46, 14.36, 14.91 [17:13:39] @replag [17:13:39] Merlissimo: s2-user: 36s [+0.00 s/s]; s2-user-c: 12s [+0.00 s/s]; s3-rr-a: 33s [+0.00 s/s]; s3-user: 33s [+0.00 s/s]; s5-rr-a: 5m 44s [+0.00 s/s]; s5-user: 5m 44s [+0.00 s/s]; s5-user-c: 12s [+0.00 s/s]; s6-rr-a: 1m 29s [+0.00 s/s] [17:13:40] Merlissimo: s6-user: 1m 29s [+0.00 s/s] [17:16:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:18:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:18:15] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [17:29:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 178592.000000 [17:29:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447253 MB (8% inode=43%): [17:31:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:33:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.520996/1.95, alarm hl:np_load_avg=1.687500/2.0, alarm hl:mem_free=124.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.520996/2.3, alarm hl:np_load_long=1.740723/2.5, alarm hl:cpu=86.700000/98, alarm hl:mem_free=124.000000M/150M, alarm hl:available=1/0 [17:47:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:49:25] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:51:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 173345 [17:52:26] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.503906/1.95, alarm hl:np_load_avg=1.644043/2.0, alarm hl:mem_free=175.000000M/350M, alarm hl:available=1/0 [17:57:46] Load avg. on willow is WARNING: WARNING - load average: 15.01, 14.50, 14.10 [17:58:45] Load avg. on willow is OK: OK - load average: 14.91, 14.57, 14.16 [18:02:45] Load avg. on willow is WARNING: WARNING - load average: 18.89, 17.44, 15.46 [18:03:05] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 579 MB (16% inode=99%): [18:04:05] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 905 MB (23% inode=99%): [18:04:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:16:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:18:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:18:25] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [18:24:25] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [18:29:05] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 120 MB (3% inode=99%): [18:29:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 445487 MB (8% inode=43%): [18:30:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 162231.000000 [18:31:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:32:05] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 715 MB (19% inode=99%): [18:34:05] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 173 MB (5% inode=99%): [18:35:46] Load avg. on willow is WARNING: WARNING - load average: 17.54, 16.33, 17.49 [18:36:05] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:05] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [18:39:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:41:24] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:44:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.031738/1.95, alarm hl:np_load_avg=1.968750/2.0, alarm hl:mem_free=131.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.031738/2.3, alarm hl:np_load_long=2.079590/2.5, alarm hl:cpu=99.500000/98, alarm hl:mem_free=131.000000M/150M, alarm hl:available=1/0 [18:46:05] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:05] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:56] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [18:46:56] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.002 second response time [18:47:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:47:55] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1907815s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1907815s failure [18:52:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 154823 [19:17:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:18:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:18:24] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [19:20:56] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [19:22:57] 3(created) [ACCAPP-498] I'm active user in fa.wiki and I need this account to run bot on this wiki; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-498> (Javad) [19:23:25] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:25:45] Load avg. on willow is OK: OK - load average: 9.57, 12.50, 14.77 [19:26:52] 3(commented) [ACCAPP-498] I'm active user in fa.wiki and I need this account to run bot on this wiki <10https://jira.toolserver.org/browse/ACCAPP-498> (Darafsh Kaviyani) [19:27:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.958008/1.95, alarm hl:np_load_avg=1.395020/2.0, alarm hl:mem_free=271.000000M/350M, alarm hl:available=1/0 [19:30:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138082.000000 [19:30:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 443493 MB (8% inode=43%): [19:31:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:35:53] 3(commented) [ACCAPP-498] I'm active user in fa.wiki and I need this account to run bot on this wiki <10https://jira.toolserver.org/browse/ACCAPP-498> (Javad) [19:36:05] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 155 MB (4% inode=99%): [19:38:14] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=5.630859/1.10, alarm hl:np_load_long=1.687500/1.55, alarm hl:mem_free=18310.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=5.630859/1.00, alarm hl:np_load_long=1.687500/1.50, alarm hl:mem_free=18310.000000M/350M, alarm hl:available=1/0 [19:38:57] Sun Grid Engine execd on wolfsbane is CRITICAL: short-sol@wolfsbane in error state: QERROR as result of job 1907801s failure: medium-sol@wolfsbane in error state: QERROR as result of job 1907801s failure [19:40:06] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 380 MB (11% inode=99%): [19:42:05] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 942 MB (24% inode=99%): [19:42:15] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [19:47:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:52:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 132878 [19:53:15] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.270508/1.10, alarm hl:np_load_long=1.197266/1.55, alarm hl:mem_free=18401.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.270508/1.00, alarm hl:np_load_long=1.197266/1.50, alarm hl:mem_free=18401.000000M/350M, alarm hl:available=1/0 [19:57:25] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:00:55] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [20:02:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.723633/1.95, alarm hl:np_load_avg=1.536133/2.0, alarm hl:mem_free=189.000000M/350M, alarm hl:available=1/0 [20:03:26] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:18:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:18:25] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [20:18:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:30:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116388.000000 [20:30:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 443317 MB (8% inode=43%): [20:31:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:45:15] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.069336/1.00, alarm hl:np_load_long=0.853515/1.50, alarm hl:mem_free=18329.000000M/350M, alarm hl:available=1/0 [20:47:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:48:15] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:52:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 109438 [21:18:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:18:25] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [21:18:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:30:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99238.000000 [21:30:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 443148 MB (8% inode=43%): [21:31:26] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:45:15] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.147461/1.10, alarm hl:np_load_long=0.775390/1.55, alarm hl:mem_free=19095.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.147461/1.00, alarm hl:np_load_long=0.775390/1.50, alarm hl:mem_free=19095.000000M/350M, alarm hl:available=1/0 [21:46:15] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:47:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:51:46] nacht ts [21:52:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 92212 [22:18:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:19:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:19:26] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [22:30:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80655.000000 [22:30:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 442952 MB (8% inode=43%): [22:31:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:47:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:52:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73571 [23:03:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.382324/1.95, alarm hl:np_load_avg=1.413086/2.0, alarm hl:mem_free=261.000000M/350M, alarm hl:available=1/0 [23:06:26] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:18:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:19:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:19:25] APT on yarrow is CRITICAL: APT CRITICAL: 6 packages available for upgrade (6 critical updates). [23:25:21] wow, what hapenned to willow, it's much faster now, yay! [23:30:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53318.000000 [23:31:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:31:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 442865 MB (8% inode=43%): [23:41:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.412598/1.95, alarm hl:np_load_avg=1.371582/2.0, alarm hl:mem_free=174.000000M/350M, alarm hl:available=1/0 [23:44:27] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:44:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:47:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:47:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.524414/1.95, alarm hl:np_load_avg=1.447266/2.0, alarm hl:mem_free=318.000000M/350M, alarm hl:available=1/0 [23:49:15] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:50:27] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:52:28] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43512