[00:00:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 276572 MB (5% inode=31%): [00:02:45] / on willow is WARNING: DISK WARNING - free space: / 19831 MB (18% inode=98%): [00:02:54] /tmp on willow is WARNING: DISK WARNING - free space: / 19829 MB (18% inode=98%): [00:05:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.145020/1.95, alarm hl:tmp_free=19826M/100M, alarm hl:np_load_avg=1.187012/2.0, alarm hl:mem_free=338.000000M/350M, alarm hl:available=1/0 [00:23:45] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:24:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:27:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:38:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:39:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:47:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.878906/1.95, alarm hl:tmp_free=19752M/100M, alarm hl:np_load_avg=0.812012/2.0, alarm hl:mem_free=249.000000M/350M, alarm hl:available=1/0 [00:48:44] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [00:53:54] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103793 [01:00:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 276553 MB (5% inode=31%): [01:02:44] / on willow is WARNING: DISK WARNING - free space: / 19663 MB (18% inode=98%): [01:02:54] /tmp on willow is WARNING: DISK WARNING - free space: / 19663 MB (18% inode=98%): [01:08:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.032715/1.95, alarm hl:tmp_free=19573M/100M, alarm hl:np_load_avg=1.101562/2.0, alarm hl:mem_free=281.000000M/350M, alarm hl:available=1/0 [01:16:45] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [01:19:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.633789/1.95, alarm hl:tmp_free=19284M/100M, alarm hl:np_load_avg=0.802246/2.0, alarm hl:mem_free=291.000000M/350M, alarm hl:available=1/0 [01:24:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:24:44] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:27:04] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:27:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:38:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:39:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:53:57] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 106034 [02:00:56] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 276501 MB (5% inode=31%): [02:02:55] / on willow is WARNING: DISK WARNING - free space: / 18016 MB (17% inode=98%): [02:02:55] /tmp on willow is WARNING: DISK WARNING - free space: / 18016 MB (17% inode=98%): [02:16:23] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:16:56] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:17:24] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:18:44] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [02:24:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:24:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:27:04] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:27:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.224609/1.95, alarm hl:tmp_free=17891M/100M, alarm hl:np_load_avg=0.946777/2.0, alarm hl:mem_free=253.000000M/350M, alarm hl:available=1/0 [02:28:55] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:31:45] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:31:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.270996/1.95, alarm hl:tmp_free=17776M/100M, alarm hl:np_load_avg=1.063965/2.0, alarm hl:mem_free=342.000000M/350M, alarm hl:available=1/0 [02:38:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:39:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:54:55] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107223 [02:55:55] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129792 MB (10% inode=99%): [02:56:54] /sql on cassia is OK: DISK OK - free space: /sql 130436 MB (11% inode=99%): [02:58:51] [[User:Aviralkumartiwari]] ! 10https://wiki.toolserver.org/w/index.php?diff=7268&oldid=6352&rcid=9698 * 110.234.51.116 * (+5143) () [02:59:11] [[User:Aviralkumartiwari]] ! 10https://wiki.toolserver.org/w/index.php?diff=7269&oldid=7268&rcid=9699 * 110.234.51.116 * (+1) () [02:59:26] [[User:Aviralkumartiwari]] ! 10https://wiki.toolserver.org/w/index.php?diff=7270&oldid=7269&rcid=9700 * 110.234.51.116 * (+1) () [03:00:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 274660 MB (5% inode=31%): [03:02:55] / on willow is WARNING: DISK WARNING - free space: / 17080 MB (16% inode=98%): [03:02:55] /tmp on willow is WARNING: DISK WARNING - free space: / 17080 MB (16% inode=98%): [03:07:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.002930/1.95, alarm hl:tmp_free=16943M/100M, alarm hl:np_load_avg=1.109375/2.0, alarm hl:mem_free=239.000000M/350M, alarm hl:available=1/0 [03:13:55] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [03:19:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.867676/1.95, alarm hl:tmp_free=16601M/100M, alarm hl:np_load_avg=0.952148/2.0, alarm hl:mem_free=309.000000M/350M, alarm hl:available=1/0 [03:24:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:25:55] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:27:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:38:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:39:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:42:30] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User:Aviralkumartiwari10]]": doesn't appear to be within project scope) [03:42:54] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129836 MB (10% inode=99%): [03:44:55] /sql on cassia is OK: DISK OK - free space: /sql 130714 MB (11% inode=99%): [03:50:54] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129851 MB (10% inode=99%): [03:55:04] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 109067 [04:01:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 274611 MB (5% inode=31%): [04:03:04] / on willow is WARNING: DISK WARNING - free space: / 15998 MB (15% inode=98%): [04:03:04] /tmp on willow is WARNING: DISK WARNING - free space: / 15998 MB (15% inode=98%): [04:08:14] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.077149/1.00, alarm hl:np_load_long=0.712890/1.50, alarm hl:mem_free=10478.000000M/600M, alarm hl:tmp_free=14981M/100M, alarm hl:available=1/0 [04:09:13] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:12:04] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.337402/1.00, alarm hl:np_load_long=0.399414/1.50, alarm hl:mem_free=538.000000M/600M, alarm hl:tmp_free=13072M/100M, alarm hl:available=1/0 [04:13:54] /sql on cassia is OK: DISK OK - free space: /sql 131566 MB (11% inode=99%): [04:16:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.580078/1.95, alarm hl:tmp_free=15970M/100M, alarm hl:np_load_avg=1.161621/2.0, alarm hl:mem_free=133.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.580078/2.3, alarm hl:np_load_long=0.982422/2.5, alarm hl:cpu=93.800000/98, alarm hl:mem_free=133.000000M/200M, al [04:19:04] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [04:22:04] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.351074/1.00, alarm hl:np_load_long=0.376465/1.50, alarm hl:mem_free=519.000000M/600M, alarm hl:tmp_free=13056M/100M, alarm hl:available=1/0 [04:22:04] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:24:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:25:55] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129833 MB (10% inode=99%): [04:26:04] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:27:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:28:55] /sql on cassia is OK: DISK OK - free space: /sql 130808 MB (11% inode=99%): [04:36:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.341309/1.95, alarm hl:tmp_free=15930M/100M, alarm hl:np_load_avg=1.416504/2.0, alarm hl:mem_free=188.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.341309/2.3, alarm hl:np_load_long=1.287109/2.5, alarm hl:cpu=92.200000/98, alarm hl:mem_free=188.000000M/200M, al [04:39:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:39:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:40:38] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1901 [04:40:39] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1905.000000 [04:40:57] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129748 MB (10% inode=99%): [04:45:40] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:55:39] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 109926 [04:56:57] /sql on cassia is OK: DISK OK - free space: /sql 130899 MB (11% inode=99%): [05:01:38] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 274557 MB (5% inode=31%): [05:03:39] /tmp on willow is WARNING: DISK WARNING - free space: / 15873 MB (15% inode=98%): [05:03:39] / on willow is WARNING: DISK WARNING - free space: / 15873 MB (15% inode=98%): [05:08:57] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129739 MB (10% inode=99%): [05:10:57] /sql on cassia is OK: DISK OK - free space: /sql 130591 MB (11% inode=99%): [05:12:07] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:16:47] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:17:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.605957/1.95, alarm hl:tmp_free=15837M/100M, alarm hl:np_load_avg=1.501953/2.0, alarm hl:mem_free=152.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.605957/2.3, alarm hl:np_load_long=1.397461/2.5, alarm hl:cpu=94.700000/98, alarm hl:mem_free=152.000000M/200M, al [05:20:48] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [05:24:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:26:47] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:27:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:28:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [05:28:37] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [05:31:08] can anyone access to https://phpmyadmin.toolserver.org/ ? my password doesn't seem to work [05:31:48] Load avg. on willow is WARNING: WARNING - load average: 19.44, 15.17, 12.46 [05:31:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.404785/1.95, alarm hl:tmp_free=15804M/100M, alarm hl:np_load_avg=1.893066/2.0, alarm hl:mem_free=509.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.404785/2.3, alarm hl:np_load_long=1.557129/2.5, alarm hl:cpu=91.300000/98, alarm hl:mem_free=509.000000M/200M, al [05:32:48] Load avg. on willow is OK: OK - load average: 11.39, 13.68, 12.11 [05:39:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:39:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:42:40] liangent@clematis:~$ qcronsub -l h_rt=0:59:00 -l virtual_free=200M -b y -N php_interwikiFromCategorySort_zhwiki_FishBaseLatinName /usr/bin/php $HOME/code/gerrit/mediawiki/core/maintenance/interwikiFromCategorySort.php --wiki=zhwiki_p --category=FishBaseLatinName --lang=en,ca,es,nl,fr,pt [05:42:41] Unable to run job: master got unknown command from JSV: "ERROR". [05:42:41] Exiting. [05:42:45] what does this message mean? [05:50:48] Load avg. on willow is WARNING: WARNING - load average: 19.06, 14.76, 12.84 [05:50:57] /sql on cassia is WARNING: DISK WARNING - free space: /sql 126679 MB (10% inode=99%): [05:56:37] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 109974 [06:01:47] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 274475 MB (5% inode=31%): [06:02:57] /sql on cassia is OK: DISK OK - free space: /sql 131593 MB (11% inode=99%): [06:04:38] /tmp on willow is WARNING: DISK WARNING - free space: / 15587 MB (14% inode=98%): [06:04:38] / on willow is WARNING: DISK WARNING - free space: / 15587 MB (14% inode=98%): [06:05:56] /sql on cassia is WARNING: DISK WARNING - free space: /sql 127793 MB (10% inode=99%): [06:24:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:25:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.876465/1.95, alarm hl:tmp_free=15378M/100M, alarm hl:np_load_avg=2.170410/2.0, alarm hl:mem_free=416.000000M/350M, alarm hl:available=1/0 [06:26:48] Load avg. on willow is WARNING: WARNING - load average: 17.43, 17.68, 17.04 [06:26:48] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:28:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:28:49] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5555 [06:28:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5554.000000 [06:29:48] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:32:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.428711/1.95, alarm hl:tmp_free=15299M/100M, alarm hl:np_load_avg=2.212402/2.0, alarm hl:mem_free=937.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.428711/2.3, alarm hl:np_load_long=2.136230/2.5, alarm hl:cpu=99.800000/98, alarm hl:mem_free=937.000000M/200M, al [06:34:47] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:39:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:39:47] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:53:47] Load avg. on willow is OK: OK - load average: 13.16, 13.98, 14.99 [06:56:37] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110509 [07:01:48] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 274386 MB (5% inode=31%): [07:04:47] / on willow is WARNING: DISK WARNING - free space: / 14947 MB (14% inode=98%): [07:04:48] /tmp on willow is WARNING: DISK WARNING - free space: / 14947 MB (14% inode=98%): [07:05:48] Load avg. on willow is WARNING: WARNING - load average: 17.04, 15.14, 14.69 [07:06:47] Load avg. on willow is OK: OK - load average: 12.58, 14.24, 14.40 [07:20:58] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:57] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [07:24:47] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:26:48] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:27:07] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:28:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:28:50] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6985 [07:28:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6985.000000 [07:37:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.051270/1.95, alarm hl:tmp_free=14757M/100M, alarm hl:np_load_avg=1.130859/2.0, alarm hl:mem_free=293.000000M/350M, alarm hl:available=1/0 [07:38:48] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [07:39:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:39:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:42:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.055664/1.95, alarm hl:tmp_free=14733M/100M, alarm hl:np_load_avg=1.111816/2.0, alarm hl:mem_free=290.000000M/350M, alarm hl:available=1/0 [07:46:48] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:56:37] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110978 [08:01:48] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 274219 MB (5% inode=31%): [08:04:48] / on willow is WARNING: DISK WARNING - free space: / 14638 MB (13% inode=98%): [08:04:48] /tmp on willow is WARNING: DISK WARNING - free space: / 14638 MB (13% inode=98%): [08:22:57] /sql on cassia is OK: DISK OK - free space: /sql 132398 MB (11% inode=99%): [08:24:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:25:58] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129981 MB (10% inode=99%): [08:26:57] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:27:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.485351/1.95, alarm hl:tmp_free=14529M/100M, alarm hl:np_load_avg=1.318359/2.0, alarm hl:mem_free=319.000000M/350M, alarm hl:available=1/0 [08:28:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:28:59] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8524 [08:29:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8523.000000 [08:29:58] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [08:37:58] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.291016/1.95, alarm hl:tmp_free=14462M/100M, alarm hl:np_load_avg=1.297851/2.0, alarm hl:mem_free=232.000000M/350M, alarm hl:available=1/0 [08:39:57] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:39:58] /sql on cassia is OK: DISK OK - free space: /sql 130133 MB (11% inode=99%): [08:39:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:56:37] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111819 [09:01:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 274016 MB (5% inode=31%): [09:04:58] /tmp on willow is WARNING: DISK WARNING - free space: / 14341 MB (13% inode=98%): [09:04:58] / on willow is WARNING: DISK WARNING - free space: / 14341 MB (13% inode=98%): [09:13:07] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.204590/1.95, alarm hl:tmp_free=14307M/100M, alarm hl:np_load_avg=1.323242/2.0, alarm hl:mem_free=297.000000M/350M, alarm hl:available=1/0 [09:14:07] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:16:07] /sql on cassia is WARNING: DISK WARNING - free space: /sql 127867 MB (10% inode=99%): [09:20:07] /sql on cassia is OK: DISK OK - free space: /sql 131987 MB (11% inode=99%): [09:23:07] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129773 MB (10% inode=99%): [09:25:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:26:57] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:29:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:29:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9694.000000 [09:29:58] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9698 [09:40:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:40:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:56:37] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111804 [09:57:44] 3(commented) [UTRS-104] Display ticket date in search interface <10https://jira.toolserver.org/browse/UTRS-104> (Martijn Hoekstra) [10:02:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 273853 MB (5% inode=31%): [10:04:58] /tmp on willow is WARNING: DISK WARNING - free space: / 13997 MB (13% inode=98%): [10:04:58] / on willow is WARNING: DISK WARNING - free space: / 13997 MB (13% inode=98%): [10:15:07] /sql on cassia is OK: DISK OK - free space: /sql 132904 MB (11% inode=99%): [10:23:07] /sql on cassia is WARNING: DISK WARNING - free space: /sql 130026 MB (10% inode=99%): [10:25:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:27:07] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:29:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:30:37] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8550.000000 [10:30:57] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8520 [10:37:08] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:40:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:55:07] /sql on cassia is OK: DISK OK - free space: /sql 132731 MB (11% inode=99%): [10:56:37] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111292 [11:01:07] /sql on cassia is WARNING: DISK WARNING - free space: /sql 130051 MB (10% inode=99%): [11:01:56] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:02:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 273718 MB (5% inode=31%): [11:02:07] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.031250/1.00, alarm hl:np_load_long=0.569336/1.50, alarm hl:mem_free=11152.000000M/600M, alarm hl:tmp_free=14406M/100M, alarm hl:available=1/0 [11:04:59] /tmp on willow is WARNING: DISK WARNING - free space: / 13756 MB (13% inode=98%): [11:05:01] / on willow is WARNING: DISK WARNING - free space: / 13756 MB (13% inode=98%): [11:08:07] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.161133/1.95, alarm hl:tmp_free=13751M/100M, alarm hl:np_load_avg=1.143555/2.0, alarm hl:mem_free=210.000000M/350M, alarm hl:available=1/0 [11:14:07] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:14:07] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [11:17:07] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.051758/1.00, alarm hl:np_load_long=0.941406/1.50, alarm hl:mem_free=11290.000000M/600M, alarm hl:tmp_free=14422M/100M, alarm hl:available=1/0 [11:25:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:28:08] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:29:37] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:30:08] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:30:58] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8720 [11:31:07] /sql on cassia is OK: DISK OK - free space: /sql 131842 MB (11% inode=99%): [11:31:18] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8715.000000 [11:35:07] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129909 MB (10% inode=99%): [11:40:07] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:40:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:56:47] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111050 [11:57:17] /sql on cassia is OK: DISK OK - free space: /sql 130237 MB (11% inode=99%): [12:02:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 273549 MB (5% inode=31%): [12:04:58] /tmp on willow is WARNING: DISK WARNING - free space: / 13656 MB (13% inode=98%): [12:05:56] / on willow is WARNING: DISK WARNING - free space: / 13652 MB (13% inode=98%): [12:07:07] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.097168/1.95, alarm hl:tmp_free=13644M/100M, alarm hl:np_load_avg=1.137695/2.0, alarm hl:mem_free=298.000000M/350M, alarm hl:available=1/0 [12:14:17] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [12:25:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:26:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.182617/1.95, alarm hl:tmp_free=13623M/100M, alarm hl:np_load_avg=1.019043/2.0, alarm hl:mem_free=199.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.182617/2.3, alarm hl:np_load_long=1.005371/2.5, alarm hl:cpu=84.600000/98, alarm hl:mem_free=199.000000M/200M, al [12:28:17] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:29:37] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:30:56] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6581 [12:32:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6582.000000 [12:40:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:40:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:42:22] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198106 MB (20% inode=99%): [12:57:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111329 [13:00:44] [[Special:Log/newusers]] create 10 * Lnagels * (New user account) [13:02:21] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 272085 MB (5% inode=31%): [13:05:26] /tmp on willow is WARNING: DISK WARNING - free space: / 13556 MB (12% inode=98%): [13:06:25] / on willow is WARNING: DISK WARNING - free space: / 13554 MB (12% inode=98%): [13:25:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:28:31] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:29:25] /sql on cassia is WARNING: DISK WARNING - free space: /sql 130063 MB (10% inode=99%): [13:29:26] /sql on thyme is OK: DISK OK - free space: /sql 234952 MB (24% inode=99%): [13:29:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:31:27] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6317 [13:32:24] /sql on cassia is OK: DISK OK - free space: /sql 130455 MB (11% inode=99%): [13:32:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6258.000000 [13:40:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:40:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:57:54] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111323 [14:02:35] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 271942 MB (5% inode=31%): [14:05:34] /tmp on willow is WARNING: DISK WARNING - free space: / 13466 MB (12% inode=98%): [14:06:35] / on willow is WARNING: DISK WARNING - free space: / 13466 MB (12% inode=98%): [14:08:34] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129875 MB (10% inode=99%): [14:12:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.315918/1.95, alarm hl:tmp_free=13458M/100M, alarm hl:np_load_avg=1.264160/2.0, alarm hl:mem_free=274.000000M/350M, alarm hl:available=1/0 [14:13:44] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [14:19:34] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43333 MB (10% inode=99%): [14:21:35] /sql on cassia is OK: DISK OK - free space: /sql 130230 MB (11% inode=99%): [14:24:34] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129963 MB (10% inode=99%): [14:25:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:28:45] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:30:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:31:35] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4411 [14:32:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4395.000000 [14:36:45] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23792 MB (5% inode=99%): [14:40:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:40:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:42:13] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:46:54] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:47:45] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38657 MB (9% inode=99%): [14:50:45] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 51208 MB (12% inode=99%): [14:57:54] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111526 [15:02:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 271750 MB (5% inode=31%): [15:05:45] /tmp on willow is WARNING: DISK WARNING - free space: / 13379 MB (12% inode=98%): [15:06:44] / on willow is WARNING: DISK WARNING - free space: / 13377 MB (12% inode=98%): [15:14:51] [[Special:Log/newusers]] create 10 * FixToolbox * (New user account) [15:22:44] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3598 [15:24:11] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3571.000000 [15:25:45] /sql on cassia is WARNING: DISK WARNING - free space: /sql 129580 MB (10% inode=99%): [15:25:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:28:44] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:30:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:32:44] [[Special:Log/newusers]] create 10 * आनन्द विवेक सतपथी * (New user account) [15:35:16] [[User talk:आनन्द विवेक सतपथी]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7271&rcid=9705 * आनन्द विवेक सतपथी * (+68) (Created page with "आनन्द विवेक सतपथी Anand Vivek Satpathi") [15:35:44] [[User:आनन्द विवेक सतपथी]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7272&rcid=9706 * आनन्द विवेक सतपथी * (+47) (Created page with "आनन्द विवेक सतपथी") [15:41:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:41:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:58:04] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 112134 [16:02:46] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 271502 MB (5% inode=31%): [16:06:44] /tmp on willow is WARNING: DISK WARNING - free space: / 13289 MB (12% inode=98%): [16:07:43] / on willow is WARNING: DISK WARNING - free space: / 13289 MB (12% inode=98%): [16:18:44] 3(commented) [UTRS-104] Display ticket date in search interface <10https://jira.toolserver.org/browse/UTRS-104> (Fluffernutter) [16:22:44] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2457 [16:24:04] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2474.000000 [16:25:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:28:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:30:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:41:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:41:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:42:14] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:46:14] [[User:FixToolbox]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7273&rcid=9707 * FixToolbox * (+296) (Created page with "Fix Toolbox, Inc. is easy and powerful program, distributing data recovery tools and services worldwide. Get Access database fix tool if you are interested to repair corrupted ...") [16:47:14] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:54:14] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1773.000000 [16:54:43] MySQL slave on thyme is OK: Uptime: 479398 Threads: 11 Questions: 169820461 Slow queries: 60994 Opens: 33256 Flush tables: 1 Open tables: 2801 Queries per second avg: 354.236 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1742 [16:58:14] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 112739 [17:02:55] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 271295 MB (5% inode=31%): [17:06:44] /tmp on willow is WARNING: DISK WARNING - free space: / 13204 MB (12% inode=98%): [17:07:44] / on willow is WARNING: DISK WARNING - free space: / 13202 MB (12% inode=98%): [17:25:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:28:55] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:30:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:41:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:41:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:57:14] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:58:14] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113343 [18:04:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 270924 MB (5% inode=31%): [18:06:44] /tmp on willow is WARNING: DISK WARNING - free space: / 12928 MB (12% inode=98%): [18:07:44] / on willow is WARNING: DISK WARNING - free space: / 12923 MB (12% inode=98%): [18:11:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.547363/1.95, alarm hl:tmp_free=12912M/100M, alarm hl:np_load_avg=1.373047/2.0, alarm hl:mem_free=263.000000M/350M, alarm hl:available=1/0 [18:13:05] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:18:02] Christopher David Howie * [Toolserver-l] Block mail from dysgo.org? [18:18:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.166016/1.95, alarm hl:tmp_free=12874M/100M, alarm hl:np_load_avg=1.313476/2.0, alarm hl:mem_free=315.000000M/350M, alarm hl:available=1/0 [18:26:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:29:05] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:30:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:42:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:42:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:58:42] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 114204 [19:04:42] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 270719 MB (5% inode=31%): [19:07:42] /tmp on willow is WARNING: DISK WARNING - free space: / 12644 MB (12% inode=98%): [19:08:40] / on willow is WARNING: DISK WARNING - free space: / 12643 MB (12% inode=98%): [19:09:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.312988/1.95, alarm hl:tmp_free=12642M/100M, alarm hl:np_load_avg=1.323730/2.0, alarm hl:mem_free=280.000000M/350M, alarm hl:available=1/0 [19:14:40] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [19:17:40] Load avg. on ortelius is WARNING: WARNING - load average: 19.07, 10.37, 5.89 [19:17:40] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.746094/1.10, alarm hl:np_load_long=1.477539/1.55, alarm hl:mem_free=10286.000000M/500M, alarm hl:tmp_free=13716M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=4.746094/1.00, alarm hl:np_load_long=1.477539/1.50, alarm hl:mem_free=10286.000000M/600M, alarm hl:tmp_free [19:19:41] Load avg. on ortelius is OK: OK - load average: 7.35, 9.42, 6.12 [19:21:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [19:26:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:29:41] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:31:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:33:58] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.353516/1.10, alarm hl:np_load_long=1.189453/1.55, alarm hl:mem_free=10476.000000M/500M, alarm hl:tmp_free=13682M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.353516/1.00, alarm hl:np_load_long=1.189453/1.50, alarm hl:mem_free=10476.000000M/600M, alarm hl:tmp_free [19:43:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:43:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:48:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.150879/1.95, alarm hl:tmp_free=12375M/100M, alarm hl:np_load_avg=1.318848/2.0, alarm hl:mem_free=261.000000M/350M, alarm hl:available=1/0 [19:49:38] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [19:57:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.181152/1.95, alarm hl:tmp_free=12278M/100M, alarm hl:np_load_avg=1.219238/2.0, alarm hl:mem_free=254.000000M/350M, alarm hl:available=1/0 [19:59:08] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 116233 [20:05:40] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 270519 MB (5% inode=31%): [20:08:39] /tmp on willow is WARNING: DISK WARNING - free space: / 12170 MB (11% inode=98%): [20:09:39] / on willow is WARNING: DISK WARNING - free space: / 12168 MB (11% inode=98%): [20:27:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:30:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:32:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:43:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:43:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:59:08] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 118672 [21:05:38] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 268981 MB (5% inode=31%): [21:08:40] /tmp on willow is WARNING: DISK WARNING - free space: / 11884 MB (11% inode=98%): [21:09:40] / on willow is WARNING: DISK WARNING - free space: / 11882 MB (11% inode=98%): [21:22:18] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:26:57] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [21:27:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:31:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:32:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:43:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:43:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:58:43] @replag [21:58:43] Magog_the_Ogre: s1-rr-a: 39s [-]; s1-user: 39s [-]; s3-rr-a: 41s [+0.00 s/s]; s3-user: 41s [+0.00 s/s]; s5-rr-a: 1m 46s [+0.00 s/s]; s5-user: 1m 46s [+0.00 s/s]; s6-rr-a: 1d 9h 20m 8s [+0.22 s/s]; s6-user: 1d 9h 20m 8s [+0.22 s/s] [21:58:50] yikes [21:59:08] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120003 [22:05:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 267482 MB (5% inode=31%): [22:08:49] /tmp on willow is WARNING: DISK WARNING - free space: / 11726 MB (11% inode=98%): [22:09:49] / on willow is WARNING: DISK WARNING - free space: / 11724 MB (11% inode=98%): [22:10:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.715332/1.95, alarm hl:tmp_free=11724M/100M, alarm hl:np_load_avg=1.364258/2.0, alarm hl:mem_free=151.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.715332/2.3, alarm hl:np_load_long=1.299316/2.5, alarm hl:cpu=96.300000/98, alarm hl:mem_free=151.000000M/200M, al [22:14:48] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:26:48] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.630859/1.10, alarm hl:np_load_long=0.571289/1.55, alarm hl:mem_free=10691.000000M/500M, alarm hl:tmp_free=13456M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.630859/1.00, alarm hl:np_load_long=0.571289/1.50, alarm hl:mem_free=10691.000000M/600M, alarm hl:tmp_free [22:27:58] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [22:27:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:31:58] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:32:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:43:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:43:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:47:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.185059/1.95, alarm hl:tmp_free=11603M/100M, alarm hl:np_load_avg=1.253906/2.0, alarm hl:mem_free=307.000000M/350M, alarm hl:available=1/0 [22:49:58] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:59:08] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121942 [23:06:48] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 267387 MB (5% inode=30%): [23:08:48] /tmp on willow is WARNING: DISK WARNING - free space: / 11537 MB (11% inode=98%): [23:10:49] / on willow is WARNING: DISK WARNING - free space: / 11530 MB (11% inode=98%): [23:11:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.349609/1.95, alarm hl:tmp_free=11531M/100M, alarm hl:np_load_avg=1.256836/2.0, alarm hl:mem_free=217.000000M/350M, alarm hl:available=1/0 [23:12:49] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11525 MB (10% inode=98%): [23:12:49] / on willow is CRITICAL: DISK CRITICAL - free space: / 11525 MB (10% inode=98%): [23:16:58] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [23:28:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:31:58] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:32:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:41:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.648926/1.95, alarm hl:tmp_free=11446M/100M, alarm hl:np_load_avg=1.376465/2.0, alarm hl:mem_free=268.000000M/350M, alarm hl:available=1/0 [23:42:42] 3(created) [MAGNUS-323] Catscan not picking up templates from talk pages; Magnus' tools; Bug <10https://jira.toolserver.org/browse/MAGNUS-323> (Allen Watkins Smith, Ph.D.) [23:43:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:44:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:47:07] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [23:59:08] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123607