[00:03:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:05:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:07:29] [[Special:Log/newusers]] create 10 * Hghplushcg * (New user account) [00:11:14] [[User:Hghplushcg]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7297&rcid=9755 * Hghplushcg * (+2000) (Created page with "Folks have at all times dreamed of one's existence of an antiaging agent, or some sort of rejuvenation. From your starting of time, obtaining the "magic bullet" has taken many ty...") [00:12:01] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37452 [00:12:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.526367/1.95, alarm hl:tmp_free=42588M/100M, alarm hl:np_load_avg=0.520996/2.0, alarm hl:mem_free=308.000000M/350M, alarm hl:available=1/0 [00:13:30] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [00:13:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:14:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37481.000000 [00:15:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121689 MB (20% inode=99%): [00:19:31] [[Special:Log/delete]] delete 10 * Betacommand * (deleted "[[02User:Hghplushcg10]]": Spam) [00:19:53] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:Hghplushcg10]] with an expiry time of infinite (account creation disabled, e-mail blocked): Spamming links to external sites) [00:37:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.358887/1.95, alarm hl:tmp_free=42564M/100M, alarm hl:np_load_avg=0.345703/2.0, alarm hl:mem_free=336.000000M/350M, alarm hl:available=1/0 [00:37:47] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:37:48] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 223778 MB (4% inode=27%): [00:46:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12705.000000 [00:52:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:55:44] 3(commented) [DBQ-184] I would like to know edit history(name of editor, page title and # of edits) for all pages in a category and related pages over a defined time period <10https://jira.toolserver.org/browse/DBQ-184> (Arjuna Rao Chavala) [00:56:29] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:03:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:05:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:12:59] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38237 [01:13:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:14:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38264.000000 [01:15:00] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:15:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121569 MB (20% inode=99%): [01:34:31] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 219962 MB (4% inode=26%): [01:46:20] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15607.000000 [01:53:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:56:50] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:04:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:06:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:13:15] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39503 [02:14:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:15:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39548.000000 [02:16:05] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121626 MB (20% inode=99%): [02:27:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.229980/1.95, alarm hl:tmp_free=42201M/100M, alarm hl:np_load_avg=1.566895/2.0, alarm hl:mem_free=358.000000M/350M, alarm hl:available=1/0 [02:31:10] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [02:34:56] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220080 MB (4% inode=26%): [02:46:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11915.000000 [02:53:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:57:05] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:04:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:06:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:14:04] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40688 [03:14:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:15:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40698.000000 [03:16:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121577 MB (20% inode=99%): [03:28:05] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3311.000000 [03:31:05] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1299.000000 [03:35:12] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 219702 MB (4% inode=26%): [03:45:31] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.784180/1.10, alarm hl:np_load_long=0.896485/1.55, alarm hl:mem_free=8541.000000M/500M, alarm hl:tmp_free=15016M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.784180/1.00, alarm hl:np_load_long=0.896485/1.50, alarm hl:mem_free=8541.000000M/600M, alarm hl:tmp_free=1 [03:48:31] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:53:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:57:32] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:57:32] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [03:57:32] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [04:02:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.023926/1.95, alarm hl:tmp_free=41823M/100M, alarm hl:np_load_avg=0.868652/2.0, alarm hl:mem_free=342.000000M/350M, alarm hl:available=1/0 [04:03:33] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:04:41] MySQL slave on z-dat-s7-a is OK: Uptime: 559 Threads: 3 Questions: 3793 Slow queries: 1 Opens: 196 Flush tables: 1 Open tables: 187 Queries per second avg: 6.785 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 541 [04:04:42] MySQL on z-dat-s7-a is OK: Uptime: 559 Threads: 4 Questions: 3790 Slow queries: 1 Opens: 196 Flush tables: 1 Open tables: 187 Queries per second avg: 6.779 [04:05:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:06:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:14:08] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41955 [04:14:58] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:15:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41979.000000 [04:16:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121518 MB (20% inode=99%): [04:25:59] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.031250/1.00, alarm hl:np_load_long=0.810547/1.50, alarm hl:mem_free=9218.000000M/600M, alarm hl:tmp_free=15000M/100M, alarm hl:available=1/0 [04:26:58] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:35:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 224999 MB (4% inode=27%): [04:53:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:58:04] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:05:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:07:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:14:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43581 [05:14:33] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:15:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:15:34] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:16:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43619.000000 [05:17:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121430 MB (20% inode=99%): [05:22:58] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.575195/1.95, alarm hl:tmp_free=41565M/100M, alarm hl:np_load_avg=0.525391/2.0, alarm hl:mem_free=318.000000M/350M, alarm hl:available=1/0 [05:25:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:32:07] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [05:35:37] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:35:46] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 222163 MB (4% inode=27%): [05:43:08] [[Special:Log/newusers]] create 10 * Jsoneu1282 * (New user account) [05:52:36] /sql on cassia is WARNING: DISK WARNING - free space: /sql 119651 MB (10% inode=99%): [05:54:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:58:08] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:59:14] [[User:Jsoneu1282]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7298&rcid=9759 * Jsoneu1282 * (+485) (Created page with "The inclusion of unlimited hosting for your website is a welcome idea. The mere thought of having free reign over the Internet world is a Napoleonic dream. Never again will you...") [06:06:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:07:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:14:27] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44126 [06:15:07] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:16:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44100.000000 [06:17:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121216 MB (19% inode=99%): [06:30:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.632324/1.95, alarm hl:tmp_free=41130M/100M, alarm hl:np_load_avg=0.719238/2.0, alarm hl:mem_free=302.000000M/350M, alarm hl:available=1/0 [06:32:09] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:36:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 218007 MB (4% inode=26%): [06:54:17] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:59:08] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:03:07] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.768555/1.95, alarm hl:tmp_free=40910M/100M, alarm hl:np_load_avg=1.313476/2.0, alarm hl:mem_free=207.000000M/350M, alarm hl:available=1/0 [07:04:07] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [07:06:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:07:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:14:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43943 [07:15:07] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:16:18] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43914.000000 [07:17:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120966 MB (19% inode=99%): [07:34:27] Can anyone tell me if any user preference information is available on the database mirrors? [07:36:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 215539 MB (4% inode=26%): [07:43:45] 3(commented) [OSM-17] pg_dump is too old <10https://jira.toolserver.org/browse/OSM-17> (Kai Krueger) [07:54:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:59:07] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:06:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:07:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:10:47] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:14:38] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43471 [08:15:07] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:16:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43435.000000 [08:18:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120565 MB (19% inode=99%): [08:20:26] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:36:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 213610 MB (4% inode=26%): [08:54:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:58:40] [[Special:Log/newusers]] create 10 * Tinatianw * (New user account) [08:59:16] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:02:43] [[User:Tinatianw]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7299&rcid=9761 * Tinatianw * (+335) (Created page with "[http://www.thomassabobraceletuk.com thomas sabo online ] was initially developed in Germany and the Thomas Sabo Charm Club Collection is now renowned all over the world. This [h...") [09:06:26] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:07:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:08:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.773438/1.95, alarm hl:tmp_free=40427M/100M, alarm hl:np_load_avg=0.806152/2.0, alarm hl:mem_free=316.000000M/350M, alarm hl:available=1/0 [09:09:10] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:13:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.221680/1.95, alarm hl:tmp_free=40407M/100M, alarm hl:np_load_avg=1.103027/2.0, alarm hl:mem_free=309.000000M/350M, alarm hl:available=1/0 [09:14:40] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42386 [09:15:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:16:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42359.000000 [09:18:10] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120318 MB (19% inode=99%): [09:36:11] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 206406 MB (3% inode=25%): [09:48:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.769531/1.95, alarm hl:tmp_free=40197M/100M, alarm hl:np_load_avg=0.771484/2.0, alarm hl:mem_free=301.000000M/350M, alarm hl:available=1/0 [09:51:11] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:55:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:59:23] [[Special:Log/newusers]] create 10 * Pipes4me420 * (New user account) [10:00:11] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:06:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:08:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:14:20] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:14:40] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42335 [10:15:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:16:10] SMTP on willow is OK: SMTP OK - 0.147 sec. response time [10:16:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42282.000000 [10:18:20] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120019 MB (19% inode=99%): [10:37:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 201605 MB (3% inode=25%): [10:46:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.790527/1.95, alarm hl:tmp_free=39884M/100M, alarm hl:np_load_avg=0.701172/2.0, alarm hl:mem_free=277.000000M/350M, alarm hl:available=1/0 [10:47:20] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [10:48:21] /sql on ptolemy is OK: DISK OK - free space: /sql 133891 MB (22% inode=99%): [10:55:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:00:20] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:06:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:08:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:14:50] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41025 [11:15:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:16:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40960.000000 [11:37:31] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 212501 MB (3% inode=26%): [11:44:45] 3(commented) [ACCAPP-506] ArticleNet: Development of a grafical tool to show relations between WP articles <10https://jira.toolserver.org/browse/ACCAPP-506> (Marlen Caemmerer) [11:55:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:00:31] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1873.000000 [12:00:31] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:07:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:08:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:13:40] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.107422/1.10, alarm hl:np_load_long=0.748047/1.55, alarm hl:mem_free=9324.000000M/500M, alarm hl:tmp_free=14575M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.107422/1.00, alarm hl:np_load_long=0.748047/1.50, alarm hl:mem_free=9324.000000M/600M, alarm hl:tmp_free=1 [12:14:39] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:14:50] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39989 [12:15:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:16:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39929.000000 [12:23:40] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.291016/1.10, alarm hl:np_load_long=0.843750/1.55, alarm hl:mem_free=9608.000000M/500M, alarm hl:tmp_free=14569M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.291016/1.00, alarm hl:np_load_long=0.843750/1.50, alarm hl:mem_free=9608.000000M/600M, alarm hl:tmp_free=1 [12:29:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.412109/1.95, alarm hl:tmp_free=39513M/100M, alarm hl:np_load_avg=0.446777/2.0, alarm hl:mem_free=271.000000M/350M, alarm hl:available=1/0 [12:31:31] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [12:38:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 209673 MB (3% inode=25%): [12:45:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3639.000000 [12:55:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:00:31] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:07:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:08:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:15:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:15:49] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38693 [13:16:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38731.000000 [13:25:50] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:34:12] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41390 MB (10% inode=99%): [13:38:12] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 77440 MB (19% inode=99%): [13:38:42] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 208432 MB (3% inode=25%): [13:46:44] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4130.000000 [13:56:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:01:13] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:07:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:09:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:15:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:16:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37462 [14:17:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37432.000000 [14:27:43] 3(commented) [ACCAPP-506] ArticleNet: Development of a grafical tool to show relations between WP articles <10https://jira.toolserver.org/browse/ACCAPP-506> (Claus Colloseus) [14:39:42] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 208284 MB (3% inode=25%): [14:39:43] 3(commented) [TS-1369] SSL certificate problem, bad/outdated HTTPS CA <10https://jira.toolserver.org/browse/TS-1369> (Marlen Caemmerer) [14:46:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5843.000000 [14:50:12] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41825 MB (10% inode=99%): [14:56:11] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 65378 MB (16% inode=99%): [14:56:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:01:25] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:07:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:09:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:16:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:16:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35677 [15:17:11] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35619.000000 [15:35:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.450195/1.95, alarm hl:tmp_free=38834M/100M, alarm hl:np_load_avg=0.442383/2.0, alarm hl:mem_free=341.000000M/350M, alarm hl:available=1/0 [15:38:21] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [15:39:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 208169 MB (3% inode=25%): [15:46:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5976.000000 [15:56:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:01:42] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:08:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:09:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:16:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:16:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33753 [16:17:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33742.000000 [16:30:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.608398/1.95, alarm hl:tmp_free=38605M/100M, alarm hl:np_load_avg=0.611328/2.0, alarm hl:mem_free=214.000000M/350M, alarm hl:available=1/0 [16:32:22] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [16:40:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 207618 MB (3% inode=25%): [16:47:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4693.000000 [16:56:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:01:50] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:04:11] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3597.000000 [17:08:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:09:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:16:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:17:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31899 [17:17:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31898.000000 [17:22:11] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1779.000000 [17:22:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.470703/1.95, alarm hl:tmp_free=38383M/100M, alarm hl:np_load_avg=0.434082/2.0, alarm hl:mem_free=345.000000M/350M, alarm hl:available=1/0 [17:28:30] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [17:41:11] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 207288 MB (3% inode=25%): [17:52:50] /sql on cassia is WARNING: DISK WARNING - free space: /sql 118177 MB (9% inode=99%): [17:55:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:57:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:00:31] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [18:02:50] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:07:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.552246/1.95, alarm hl:tmp_free=38060M/100M, alarm hl:np_load_avg=0.560547/2.0, alarm hl:mem_free=267.000000M/350M, alarm hl:available=1/0 [18:08:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:09:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:14:30] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:14:39] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:15:30] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [18:16:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:17:31] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31300 [18:17:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31300.000000 [18:20:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:37:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.400391/1.95, alarm hl:tmp_free=37917M/100M, alarm hl:np_load_avg=0.550781/2.0, alarm hl:mem_free=289.000000M/350M, alarm hl:available=1/0 [18:39:39] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:42:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 207396 MB (3% inode=25%): [18:52:01] @replag [18:52:02] Dispenser: s1-user: 8h 29m 53s [-0.40 s/s]; s2-user: 6h 23m 56s [+0.53 s/s]; s2-user-c: 23m 19s [-0.15 s/s]; s3-rr-a: 1m 19s [+0.00 s/s]; s3-user: 1m 19s [+0.00 s/s]; s5-user-c: 23m 19s [-0.15 s/s] [18:57:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:02:51] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:08:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:10:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:16:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:17:40] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30185 [19:17:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30186.000000 [19:19:10] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1855.000000 [19:32:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.732910/1.95, alarm hl:tmp_free=37715M/100M, alarm hl:np_load_avg=0.642090/2.0, alarm hl:mem_free=332.000000M/350M, alarm hl:available=1/0 [19:33:10] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1782.000000 [19:33:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [19:42:28] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 207259 MB (3% inode=25%): [19:50:06] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:50:56] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [19:58:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:03:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:08:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:10:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:17:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:17:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29654.000000 [20:18:28] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29646 [20:21:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.670898/1.95, alarm hl:tmp_free=37534M/100M, alarm hl:np_load_avg=0.598633/2.0, alarm hl:mem_free=233.000000M/350M, alarm hl:available=1/0 [20:25:48] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1940.000000 [20:42:46] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229446 MB (4% inode=27%): [20:46:54] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:55:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.740723/1.95, alarm hl:tmp_free=37402M/100M, alarm hl:np_load_avg=0.621094/2.0, alarm hl:mem_free=185.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.740723/2.3, alarm hl:np_load_long=0.606934/2.5, alarm hl:cpu=63.200000/98, alarm hl:mem_free=185.000000M/200M, al [20:58:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:58:54] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [21:04:04] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:09:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:11:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:13:02] Marcin Cieslak * Re: [Toolserver-l] Toolserver user phe account expired [21:17:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:18:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28676.000000 [21:18:46] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28668 [21:24:46] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3608.000000 [21:42:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 226896 MB (4% inode=27%): [21:42:50] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3542.000000 [21:58:51] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1728.000000 [21:59:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:00:02] Platonides * Re: [Toolserver-l] Toolserver user phe account expired [22:04:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:09:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:11:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:18:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:18:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28385.000000 [22:18:51] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28385 [22:29:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.330078/1.95, alarm hl:tmp_free=36961M/100M, alarm hl:np_load_avg=0.368164/2.0, alarm hl:mem_free=244.000000M/350M, alarm hl:available=1/0 [22:32:30] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:36:11] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41279 MB (10% inode=99%): [22:42:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 224364 MB (4% inode=27%): [22:45:10] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 74072 MB (18% inode=99%): [22:59:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:02:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.745606/1.95, alarm hl:tmp_free=36862M/100M, alarm hl:np_load_avg=0.624023/2.0, alarm hl:mem_free=343.000000M/350M, alarm hl:available=1/0 [23:03:41] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [23:04:41] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:07:39] [[Special:Log/newusers]] create 10 * Fiallochuck1 * (New user account) [23:09:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:11:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:15:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.045899/1.95, alarm hl:tmp_free=36810M/100M, alarm hl:np_load_avg=0.700684/2.0, alarm hl:mem_free=265.000000M/350M, alarm hl:available=1/0 [23:18:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:19:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28511.000000 [23:19:50] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28511 [23:42:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 221325 MB (4% inode=27%):