[00:02:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:02:43] Load avg. on willow is WARNING: WARNING - load average: 17.72, 15.23, 11.52 [00:02:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.252930/1.95, alarm hl:tmp_free=36525M/100M, alarm hl:np_load_avg=1.908691/2.0, alarm hl:mem_free=361.000000M/350M, alarm hl:available=1/0 [00:03:44] Load avg. on willow is OK: OK - load average: 14.00, 14.60, 11.54 [00:06:43] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [00:09:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.103516/1.10, alarm hl:np_load_long=0.618164/1.55, alarm hl:mem_free=10903.000000M/500M, alarm hl:tmp_free=13505M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.103516/1.00, alarm hl:np_load_long=0.618164/1.50, alarm hl:mem_free=10903.000000M/600M, alarm hl:tmp_free [00:10:43] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [00:12:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.150879/1.95, alarm hl:tmp_free=36490M/100M, alarm hl:np_load_avg=1.336914/2.0, alarm hl:mem_free=292.000000M/350M, alarm hl:available=1/0 [00:12:44] 3(commented) [TS-1377] Cannot access a number of DBs on the toolserver <10https://jira.toolserver.org/browse/TS-1377> (Dario Taraborelli) [00:13:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:17:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:20:43] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:28:23] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1920.000000 [00:28:32] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1928.000000 [00:29:23] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1977.000000 [00:29:23] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1978.000000 [00:33:33] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5148 [00:34:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:34:33] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1755.000000 [00:39:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321837 MB (6% inode=35%): [00:43:23] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1793.000000 [00:56:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [00:57:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3659.000000 [00:59:03] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [01:00:23] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [01:02:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:02:44] Load avg. on willow is WARNING: WARNING - load average: 15.38, 13.75, 12.09 [01:02:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.092773/1.95, alarm hl:tmp_free=36328M/100M, alarm hl:np_load_avg=1.726562/2.0, alarm hl:mem_free=597.000000M/350M, alarm hl:available=1/0 [01:03:43] Load avg. on willow is OK: OK - load average: 13.64, 13.51, 12.11 [01:03:45] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [01:07:43] Sun Grid Engine execd on willow is WARNING: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.856934/2.3, alarm hl:np_load_long=1.613770/2.5, alarm hl:cpu=98.800000/98, alarm hl:mem_free=525.000000M/200M, alarm hl:tmp_free=36313M/100M, alarm hl:available=1/0 [01:12:43] Load avg. on willow is WARNING: WARNING - load average: 15.13, 14.54, 13.30 [01:13:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:17:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:20:43] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:33:34] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6221 [01:34:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:36:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.415527/1.95, alarm hl:tmp_free=36224M/100M, alarm hl:np_load_avg=1.383789/2.0, alarm hl:mem_free=187.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.415527/2.3, alarm hl:np_load_long=1.456543/2.5, alarm hl:cpu=80.400000/98, alarm hl:mem_free=187.000000M/200M, al [01:39:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324186 MB (6% inode=35%): [01:39:44] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [01:43:43] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:53:12] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:56:34] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7204.000000 [01:58:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7320.000000 [01:59:02] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [02:00:33] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [02:02:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:12:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.750488/1.95, alarm hl:tmp_free=36113M/100M, alarm hl:np_load_avg=1.620117/2.0, alarm hl:mem_free=262.000000M/350M, alarm hl:available=1/0 [02:13:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:16:43] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [02:17:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:20:43] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:32:43] Load avg. on willow is WARNING: WARNING - load average: 18.98, 16.00, 13.70 [02:33:34] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7709 [02:33:44] Sun Grid Engine execd on willow is WARNING: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.818359/2.3, alarm hl:np_load_long=1.696777/2.5, alarm hl:cpu=98.300000/98, alarm hl:mem_free=372.000000M/200M, alarm hl:tmp_free=36060M/100M, alarm hl:available=1/0 [02:34:43] Load avg. on willow is OK: OK - load average: 11.25, 14.23, 13.32 [02:35:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:39:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41095 MB (10% inode=99%): [02:40:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 322212 MB (6% inode=35%): [02:49:33] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24462 MB (5% inode=99%): [02:54:34] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 88511 MB (21% inode=99%): [02:57:33] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10864.000000 [02:58:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10921.000000 [02:59:03] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [03:01:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [03:02:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:02:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.801758/1.95, alarm hl:tmp_free=35974M/100M, alarm hl:np_load_avg=1.550781/2.0, alarm hl:mem_free=300.000000M/350M, alarm hl:available=1/0 [03:03:44] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [03:08:44] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:12:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.916504/1.95, alarm hl:tmp_free=35943M/100M, alarm hl:np_load_avg=1.617676/2.0, alarm hl:mem_free=212.000000M/350M, alarm hl:available=1/0 [03:13:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:17:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:20:43] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:34:33] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7699 [03:35:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:40:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 322162 MB (6% inode=35%): [03:44:44] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41406 MB (10% inode=99%): [03:54:44] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22757 MB (5% inode=99%): [03:57:44] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14476.000000 [03:58:44] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14535.000000 [03:59:03] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [04:01:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [04:02:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:04:44] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.015625/1.00, alarm hl:np_load_long=0.577148/1.50, alarm hl:mem_free=10283.000000M/600M, alarm hl:tmp_free=14831M/100M, alarm hl:available=1/0 [04:05:45] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:06:44] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37705 MB (9% inode=99%): [04:07:44] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 64654 MB (15% inode=99%): [04:09:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.552246/1.95, alarm hl:tmp_free=35775M/100M, alarm hl:np_load_avg=1.357422/2.0, alarm hl:mem_free=272.000000M/350M, alarm hl:available=1/0 [04:12:44] Load avg. on willow is CRITICAL: CRITICAL - load average: 50.62, 22.13, 14.80 [04:13:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:14:44] Load avg. on willow is WARNING: WARNING - load average: 24.54, 21.88, 15.64 [04:17:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:18:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 44.60, 27.46, 19.09 [04:21:43] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:25:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.265625/1.10, alarm hl:np_load_long=0.833008/1.55, alarm hl:mem_free=9630.000000M/500M, alarm hl:tmp_free=14824M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.265625/1.00, alarm hl:np_load_long=0.833008/1.50, alarm hl:mem_free=9630.000000M/600M, alarm hl:tmp_free=1 [04:34:43] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8894 [04:36:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:40:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 322089 MB (6% inode=35%): [04:49:44] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.43, 30.06, 26.45 [04:55:22] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:56:03] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=3.272461/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.519043/1.95, alarm hl:tmp_free=35643M/100M, alarm hl:np_load_avg=3.272461/2.0, alarm hl:mem_free=1777.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.519043/2.3, alarm hl:np_load_long= [04:57:43] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18079.000000 [04:59:13] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [04:59:44] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18197.000000 [05:01:44] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [05:02:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:13:03] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:13:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:15:03] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:18:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:22:03] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:34:52] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9472 [05:36:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:40:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 322005 MB (6% inode=35%): [05:49:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.86, 31.55, 28.83 [05:56:59] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=4.530762/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.884277/1.95, alarm hl:tmp_free=35381M/100M, alarm hl:np_load_avg=4.530762/2.0, alarm hl:mem_free=2003.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.884277/2.3, alarm hl:np_load_long= [05:57:50] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21681.000000 [05:59:19] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [05:59:50] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21799.000000 [06:01:49] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [06:02:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:12:19] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129829 MB (10% inode=99%): [06:14:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:18:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:19:59] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.150391/1.10, alarm hl:np_load_long=0.845703/1.55, alarm hl:mem_free=10972.000000M/500M, alarm hl:tmp_free=14711M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.150391/1.00, alarm hl:np_load_long=0.845703/1.50, alarm hl:mem_free=10972.000000M/600M, alarm hl:tmp_free [06:20:59] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:22:19] /sql on cassia is OK: DISK OK - free space: /sql 134286 MB (11% inode=99%): [06:23:00] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:35:49] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10661 [06:37:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:41:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321870 MB (6% inode=35%): [06:42:43] 3(created) [MAGNUS-322] CommonsHelper adds a lot of not needed templates when copying from en-wiki; Magnus' tools: CommonsHelper; Bug <10https://jira.toolserver.org/browse/MAGNUS-322> (Michael Andersen) [06:43:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.268555/1.10, alarm hl:np_load_long=0.873047/1.55, alarm hl:mem_free=11401.000000M/500M, alarm hl:tmp_free=14687M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.268555/1.00, alarm hl:np_load_long=0.873047/1.50, alarm hl:mem_free=11401.000000M/600M, alarm hl:tmp_free [06:48:09] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:50:00] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.16, 30.27, 31.50 [06:57:10] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=3.216309/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.176758/1.95, alarm hl:tmp_free=34971M/100M, alarm hl:np_load_avg=3.216309/2.0, alarm hl:mem_free=1801.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.176758/2.3, alarm hl:np_load_long= [06:57:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25293.000000 [06:59:29] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [06:59:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25412.000000 [07:00:09] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:00:20] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:01:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:01:09] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=3.787109/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=6.879395/1.95, alarm hl:tmp_free=34958M/100M, alarm hl:np_load_avg=3.787109/2.0, alarm hl:mem_free=2436.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=6.879395/2.3, alarm hl:np_load_long= [07:02:00] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [07:02:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:14:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:18:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:20:09] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:20:48] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [07:23:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:36:00] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12771 [07:36:59] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41715 MB (10% inode=99%): [07:40:58] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 74348 MB (18% inode=99%): [07:41:08] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321778 MB (6% inode=35%): [07:50:09] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.70, 25.60, 26.59 [07:58:09] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28905.000000 [07:59:30] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [07:59:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29014.000000 [08:01:29] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=5.672852/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=10.026855/1.95, alarm hl:tmp_free=34675M/100M, alarm hl:np_load_avg=5.672852/2.0, alarm hl:mem_free=1095.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=10.026855/2.3, alarm hl:np_load_lon [08:01:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:01:59] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [08:02:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:14:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:19:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:24:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:30:09] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:30:59] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:31:09] Load avg. on willow is CRITICAL: CRITICAL - load average: 56.47, 35.49, 33.41 [08:31:29] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=4.552734/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=7.140625/1.95, alarm hl:tmp_free=34609M/100M, alarm hl:np_load_avg=4.552734/2.0, alarm hl:mem_free=1177.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=7.140625/2.3, alarm hl:np_load_long= [08:35:59] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14579 [08:41:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321208 MB (6% inode=35%): [08:58:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32506.000000 [08:59:29] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [08:59:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32616.000000 [09:01:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:01:59] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [09:02:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:15:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:19:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:24:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:31:19] Load avg. on willow is CRITICAL: CRITICAL - load average: 76.59, 38.96, 31.09 [09:32:40] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=4.180176/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=4.088867/1.95, alarm hl:tmp_free=34415M/100M, alarm hl:np_load_avg=4.180176/2.0, alarm hl:mem_free=1355.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=4.088867/2.3, alarm hl:np_load_long= [09:36:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16484 [09:41:19] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321050 MB (6% inode=35%): [09:58:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36110.000000 [09:59:39] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [10:00:09] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36223.000000 [10:01:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:02:10] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [10:03:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:13:09] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40168 MB (9% inode=99%): [10:15:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:19:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:23:08] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23942 MB (5% inode=99%): [10:25:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:29:09] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 100460 MB (24% inode=99%): [10:31:29] Load avg. on willow is CRITICAL: CRITICAL - load average: 60.33, 34.54, 27.62 [10:32:40] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=3.711426/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.350098/1.95, alarm hl:tmp_free=34227M/100M, alarm hl:np_load_avg=3.711426/2.0, alarm hl:mem_free=1390.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.350098/2.3, alarm hl:np_load_long= [10:36:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18326 [10:41:19] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 320859 MB (6% inode=35%): [10:58:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39712.000000 [10:58:29] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.025391/1.00, alarm hl:np_load_long=0.872070/1.50, alarm hl:mem_free=11813.000000M/600M, alarm hl:tmp_free=14341M/100M, alarm hl:available=1/0 [10:59:29] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [10:59:49] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [11:00:09] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39823.000000 [11:01:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:03:09] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [11:03:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:15:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:19:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:25:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:31:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 42.84, 29.77, 26.91 [11:32:41] Sun Grid Engine execd on willow is WARNING: testqueue@willow exceedes load threshold: alarm hl:np_load_avg=3.251465/2.75: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.706543/1.95, alarm hl:tmp_free=34032M/100M, alarm hl:np_load_avg=3.251465/2.0, alarm hl:mem_free=1885.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.706543/2.3, alarm hl:np_load_long= [11:36:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20112 [11:41:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 320698 MB (6% inode=35%): [11:50:43] Hey, any guys out there who speak PHP and are interessted in working on a interessting modular project? But we're also looking for some users who can test our software or reply questions to our mail or social networks accounts. If anybody wants to look into this: http://www.modovi.de/ (english) [11:58:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43313.000000 [11:59:49] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [12:00:19] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43430.000000 [12:00:59] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:01:11] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:01:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.928223/1.95, alarm hl:tmp_free=33940M/100M, alarm hl:np_load_avg=2.642578/2.0, alarm hl:mem_free=2739.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.928223/2.3, alarm hl:np_load_long=2.787109/2.5, alarm hl:cpu=65.400000/98, alarm hl:mem_free=2739.000000M/200M, [12:01:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:03:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:03:19] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [12:05:40] Load avg. on willow is WARNING: WARNING - load average: 15.90, 16.48, 19.96 [12:05:40] Sun Grid Engine execd on willow is OK: testqueue@willow OK: medium-sol@willow OK: longrun-sol@willow OK [12:06:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 66.71, 27.61, 23.51 [12:13:43] 3(resolved) [MAGNUS-322] CommonsHelper adds a lot of not needed templates when copying from en-wiki <10https://jira.toolserver.org/browse/MAGNUS-322> (Magnus Manske) [12:15:43] 3(created) [MNT-1240] Rebooted willow; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1240> (Marlen Caemmerer) [12:15:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:16:40] Load avg. on willow is OK: OK - load average: 2.04, 0.66, 0.23 [12:17:08] no notice before the reboot just now? [12:17:39] hello [12:17:48] i just rebooted willow [12:18:23] no notice in advance? [12:20:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:25:40] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:32:42] 3(updated) [MNT-1240] Rebooted willow <10https://jira.toolserver.org/browse/MNT-1240> (Marlen Caemmerer) [12:36:29] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21935 [12:41:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 320505 MB (6% inode=35%): [12:55:11] damn, somebody rebooted willow.. [12:58:29] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46923.000000 [12:59:34] willow has been restarted? [12:59:49] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [13:00:29] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47043.000000 [13:02:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:03:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:03:31] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:03:52] nosy: around? [13:04:00] yes [13:04:06] yes [13:04:11] willow has been restarted [13:04:28] that was unplanned I assume? [13:06:56] yes [13:07:27] seems that @reboot on cronie still does not work :-/ [13:07:30] but willows system components were failing [13:07:56] when we'll be back to Linux? [13:08:25] cron on Linux did not have any problems [13:08:33] on @reboot [13:10:57] afais it should not be too long [13:15:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:20:33] * wikibooker is crossing fingers  [13:20:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:20:50] hopefully it will be before hackathon if not earlier [13:25:50] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:36:39] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23613 [13:41:39] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 320264 MB (6% inode=35%): [13:46:09] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:10] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:47:00] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [13:48:09] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [13:48:39] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:53:39] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:58:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50536.000000 [13:59:46] 3(resolved) [TS-1371] Add user "sactage" to MMT group "cvn" and "swmtbot" <10https://jira.toolserver.org/browse/TS-1371> (Marlen Caemmerer) [13:59:59] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [14:00:39] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50650.000000 [14:01:43] 3(commented) [TS-1380] Crontab for MMT cvn on willow is not running <10https://jira.toolserver.org/browse/TS-1380> (Marlen Caemmerer) [14:03:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:03:39] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [14:03:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:08:39] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:13:43] 3(created) [TS-1381] s4 does not replicate anymore; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1381> (Marlen Caemmerer) [14:13:44] 3(assigned) [TS-1381] s4 does not replicate anymore <10https://jira.toolserver.org/browse/TS-1381> (Marlen Caemmerer) [14:16:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:21:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:26:10] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:26:47] hey all! [14:27:03] is the quota warning with the ultimatum on the toolserver new? [14:28:46] omg [14:28:51] its the dschwen! [14:28:57] hello dschwen :) [14:30:19] you ought to hang out on irc more :p [14:31:41] oh [14:31:45] uhm, hi [14:31:54] are you WhiteCat? [14:31:55] hi [14:32:04] hi! [14:33:01] so, anybody having quota problems here? [14:33:18] FYI region information will be filled into my database automatically for https://commons.wikimedia.org/wiki/Commons:Bots/Requests/Emijrpbot_%28Task_7%29 [14:33:27] I'm trying to track down what is filling up my home dir, thought I moved all the big stuff to /mnt/user-store [14:34:10] I will implement displaying categories um the WMA next [14:34:38] I don't know why I haven't done it earlier. Should be easy. [14:34:43] dschwen yup [14:34:56] all I know is that Neko means cat [14:35:02] I am a certain white cat [14:35:04] from My Neighbor Totoro [14:35:05] how about [W] icons like in Google map [14:35:17] とある白い猫 for short :p [14:35:22] dschwen hah! :D [14:35:27] I can do that now easily sice I switched to JSON data transfer [14:35:46] but I actually hate the mystery meat navigation [14:36:00] so about them templates, we can discuss any question you may have if you like :p [14:36:07] you have to hover/or click every icon if you want to know what it means [14:36:30] the W icons will probably come when I display the labeled original OSM tiles [14:36:31] One temlate a day keeps the doctor away as he is too busy figuring out which } is breaking the parser function chain [14:36:39] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25206 [14:36:45] haha [14:37:07] But its hard to find spots to drag the map because of all the labels [14:37:30] oh, yeah, I need to do something about that [14:37:57] actually... [14:38:09] ... icn Chrome you can drag anywhere, including on labels [14:38:21] that does not work on Firefox? [14:38:26] I wonder why. [14:38:37] probably something related to ondragstart [14:38:46] hmmm [14:41:49] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 319996 MB (6% inode=35%): [14:42:22] han, that is terrible, WMA sucks on Firefox [14:42:42] you cannot really test if you are the developer [14:42:57] I automatically drag on empty spots, it's like a reflex [14:43:06] grrr [14:43:25] ok, suspend project "categories", focus on basic usability [14:43:29] Install the Firebug and Web Developer extensions, combined they're better than what's built into Chrome [14:43:41] I have, and I disagree ;-) [14:44:01] probably because I'm so used to the webkit inspector [14:44:10] I think it is amazing! [14:44:22] the console alone is way better than firebugs [14:44:44] Only took 15 years to get decent debugger into web browser :-( [14:45:10] only? [14:45:19] its not over 9000 days, I am not impressed [14:45:42] 9000/365.25 = 24.640657 years [14:45:42] 3(commented) [MNT-668] Add missing quotas <10https://jira.toolserver.org/browse/MNT-668> (Marlen Caemmerer) [14:46:10] .640657*365.25 = 233.999969 days [14:46:36] As a matter of fact I now wish I had as nice a debugging environment for my C/C++ stuff as i have for my JS stuff [14:47:03] Today is September 6832, 1993 [14:47:23] * ToAruShiroiNeko dispenses Dispenser [14:47:48] 24 years 234 days is what I count as the cut off point for "a long time" [14:47:59] I used to use kdevelop a long time ago, and it was ok. DDD is pretty good too, but the combined heap and cpu profiling, breakpoints and inspection with console, is just great. [14:48:14] Eclipse? [14:49:18] mm [14:49:23] Java makes me cry [14:49:32] another thing, how comes no browser supports target="_parent" [14:49:51] dschwen thats a product of popups [14:49:53] You wouldn't know it is Java [14:49:55] does _top still work? [14:49:58] yes [14:50:02] _top works [14:50:04] thankfully [14:50:23] oh, right I remember, _parent throws a crossdomain security exception [14:50:24] dschwen I java your momma :p [14:50:42] http://www.youtube.com/watch?v=7_rBidCkJxo [14:50:54] No, I mean, it may be written in Java, but it does not really show. And I would only use it for C/C++ develpoing [14:51:05] poing! [14:52:33] * ToAruShiroiNeko bounces [14:56:53] hm if I bind a return false to ondragstart on the body element I can drag on labels in the WMA [14:57:07] but as soon as i release the MB the label gets a click event [14:57:16] not so in chrome [14:57:26] mpf. [14:58:49] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54146.000000 [14:58:57] also, what is best practice, binding events to document or to the body element? [15:00:09] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [15:00:39] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54252.000000 [15:03:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:03:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:04:38] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [15:17:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:17:16] ok, fixed (i think) [15:17:32] it is now possible to drag the map anywhere, even on labels [15:20:09] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 16.044 second response time [15:21:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:26:18] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:33:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:36:40] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26911 [15:41:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 319816 MB (6% inode=34%): [15:47:08] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.820312/1.10, alarm hl:np_load_long=0.845703/1.55, alarm hl:mem_free=11190.000000M/500M, alarm hl:tmp_free=13895M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.820312/1.00, alarm hl:np_load_long=0.845703/1.50, alarm hl:mem_free=11190.000000M/600M, alarm hl:tmp_free [15:50:09] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:58:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57750.000000 [16:00:10] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [16:00:40] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57854.000000 [16:03:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:03:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:04:39] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [16:17:19] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.911 second response time [16:18:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:20:19] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 274 bytes in 20.172 second response time [16:21:19] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.939 second response time [16:21:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:26:19] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:36:49] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29000 [16:41:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 319629 MB (6% inode=34%): [16:47:19] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:53:39] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:58:59] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61354.000000 [16:59:29] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:00:09] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [17:00:10] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [17:00:37] Where did it go [17:00:42] (The tool server [17:00:44] .topic [17:00:49] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61461.000000 [17:03:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:04:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:04:49] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [17:13:29] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:18:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:21:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:26:19] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:28:42] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:34:44] 3(commented) [ACCAPP-477] Account for hosting Wikipedia-OpenStreetMap integration tools <10https://jira.toolserver.org/browse/ACCAPP-477> [17:36:49] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30727 [17:42:08] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 319492 MB (6% inode=34%): [17:47:29] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:51:43] 3(resolved) [DBQ-127] List of Categories assigned to Commons images of the Wikisource PSM project and uploaded by me. <10https://jira.toolserver.org/browse/DBQ-127> (Hoo man) [17:59:09] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64961.000000 [18:00:19] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [18:00:50] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65065.000000 [18:03:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:04:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:05:48] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [18:18:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:21:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:26:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:31:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.556152/1.95, alarm hl:tmp_free=34388M/100M, alarm hl:np_load_avg=1.430176/2.0, alarm hl:mem_free=297.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.556152/2.3, alarm hl:np_load_long=1.092285/2.5, alarm hl:cpu=52.800000/98, alarm hl:mem_free=297.000000M/200M, al [18:32:29] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:35:45] 3(commented) [DBQ-127] List of Categories assigned to Commons images of the Wikisource PSM project and uploaded by me. <10https://jira.toolserver.org/browse/DBQ-127> (Ineuw) [18:36:49] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32733 [18:42:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 319306 MB (6% inode=34%): [18:48:29] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:50:29] Load avg. on willow is WARNING: WARNING - load average: 15.73, 13.28, 10.34 [18:51:29] Load avg. on willow is OK: OK - load average: 12.36, 12.71, 10.32 [18:59:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68563.000000 [19:00:19] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [19:00:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68668.000000 [19:02:29] Load avg. on willow is WARNING: WARNING - load average: 16.10, 15.52, 12.65 [19:03:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.963867/1.95, alarm hl:tmp_free=34322M/100M, alarm hl:np_load_avg=1.928223/2.0, alarm hl:mem_free=614.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.963867/2.3, alarm hl:np_load_long=1.601562/2.5, alarm hl:cpu=99.600000/98, alarm hl:mem_free=614.000000M/200M, al [19:03:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:04:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:05:49] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [19:11:39] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [19:14:39] Sun Grid Engine execd on willow is WARNING: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.618164/2.3, alarm hl:np_load_long=1.735840/2.5, alarm hl:cpu=99.500000/98, alarm hl:mem_free=463.000000M/200M, alarm hl:tmp_free=34306M/100M, alarm hl:available=1/0 [19:19:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:21:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:26:40] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:36:49] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34730 [19:42:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 320447 MB (6% inode=35%): [19:46:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317886 MB (5% inode=34%): [19:47:43] 3(commented) [TS-1381] s4 does not replicate anymore <10https://jira.toolserver.org/browse/TS-1381> (Marlen Caemmerer) [19:48:39] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:50:19] Where did the toolserver go? [19:51:01] I suppose it is still in its rack [19:51:49] Well, you never know [19:52:44] I'm logged in on willow [19:59:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72174.000000 [20:00:19] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [20:00:49] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72267.000000 [20:03:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:04:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:05:59] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [20:16:24] Rhw other then [20:19:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:20:40] [[Special:Log/newusers]] create 10 * Boberger * (New user account) [20:21:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:26:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:32:42] Load avg. on willow is WARNING: WARNING - load average: 14.02, 15.72, 13.96 [20:32:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.910645/1.95, alarm hl:tmp_free=34186M/100M, alarm hl:np_load_avg=2.000488/2.0, alarm hl:mem_free=1095.000000M/350M, alarm hl:available=1/0 [20:33:40] Load avg. on willow is OK: OK - load average: 9.53, 14.11, 13.50 [20:33:49] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:36:59] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37540 [20:40:43] 3(resolved) [TS-1381] s4 does not replicate anymore <10https://jira.toolserver.org/browse/TS-1381> (Marlen Caemmerer) [20:46:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317346 MB (5% inode=34%): [20:48:49] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:59:19] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45437.000000 [21:00:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43020.000000 [21:01:19] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42256 [21:04:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:04:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:05:43] 3(commented) [TS-1380] Crontab for MMT cvn on willow is not running <10https://jira.toolserver.org/browse/TS-1380> (Krinkle) [21:06:00] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40954 [21:19:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:27:00] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:29:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.844238/1.95, alarm hl:tmp_free=33690M/100M, alarm hl:np_load_avg=1.010254/2.0, alarm hl:mem_free=194.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.844238/2.3, alarm hl:np_load_long=1.086914/2.5, alarm hl:cpu=58.300000/98, alarm hl:mem_free=194.000000M/200M, al [21:32:00] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [21:33:02] DaB. * [Toolserver-announce] No-interwiki-bot-rule is active now [21:33:43] uh-oh [21:36:11] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3297.000000 [21:36:39] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2723 [21:37:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40382 [21:37:40] MySQL slave on daphne is OK: Uptime: 6378139 Threads: 6 Questions: 463292207 Slow queries: 242289 Opens: 722 Flush tables: 2 Open tables: 309 Queries per second avg: 72.637 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1315 [21:38:09] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 552.000000 [21:46:39] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317011 MB (5% inode=34%): [21:48:59] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:58:44] 3(commented) [ACCAPP-496] GhalyBot on toolserver <10https://jira.toolserver.org/browse/ACCAPP-496> (Ghaly) [21:58:45] 3(updated) [ACCAPP-496] GhalyBot on toolserver <10https://jira.toolserver.org/browse/ACCAPP-496> (Ghaly) [21:59:39] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9016.000000 [22:04:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:04:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:06:20] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3361 [22:06:39] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3129.000000 [22:07:43] 3(commented) [ACCAPP-496] GhalyBot on toolserver <10https://jira.toolserver.org/browse/ACCAPP-496> (Ghaly) [22:07:45] 3(updated) [ACCAPP-496] GhalyBot on toolserver <10https://jira.toolserver.org/browse/ACCAPP-496> (Ghaly) [22:08:19] MySQL slave on z-dat-s4-a is OK: Uptime: 1561653 Threads: 4 Questions: 181352159 Slow queries: 26539 Opens: 25839 Flush tables: 1 Open tables: 821 Queries per second avg: 116.128 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1350 [22:08:39] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 892.000000 [22:18:02] Danny B. * Re: [Toolserver-l] No-interwiki-bot-rule is active now [22:19:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:21:03] Krinkle * Re: [Toolserver-l] Out of memory/segfaults? [22:22:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:26:02] Krinkle * Re: [Toolserver-l] Fairness on the toolserver [22:27:21] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:37:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42748 [22:40:02] Marcin Cieslak * Re: [Toolserver-l] Fairness on the toolserver [22:46:39] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317259 MB (5% inode=34%): [22:49:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:00:48] [[Special:Log/newusers]] create 10 * Snlhhni32 * (New user account) [23:04:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:05:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:13:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.125000/1.95, alarm hl:tmp_free=33364M/100M, alarm hl:np_load_avg=1.613281/2.0, alarm hl:mem_free=451.000000M/350M, alarm hl:available=1/0 [23:13:45] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:14:24] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [23:20:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:23:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:24:15] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:24:44] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:25:35] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [23:26:18] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [23:28:05] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:37:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45503 [23:38:26] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:47:12] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317763 MB (5% inode=34%): [23:50:06] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:53:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.