[00:00:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65607 MB (6% inode=99%): [00:10:51] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.795898/2.00, alarm hl:mem_free=201.000000M/250M [00:12:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [00:12:51] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:35:10] nacht ts [00:43:29] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.295899/1.00, alarm hl:mem_free=12837.000000M/100M [00:46:30] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [00:55:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:56:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:57:50] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.665039/2.00, alarm hl:mem_free=204.000000M/250M [00:58:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 222716 MB (4% inode=28%): [00:58:50] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:59:29] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:00:30] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65492 MB (6% inode=99%): [01:12:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:55:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:57:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:58:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 222540 MB (4% inode=28%): [01:59:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:00:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65378 MB (6% inode=99%): [02:12:46] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.721191/2.00, alarm hl:mem_free=219.000000M/250M [02:13:46] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:55:55] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:57:55] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:59:28] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 222541 MB (4% inode=28%): [02:59:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:00:46] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65262 MB (6% inode=99%): [03:56:06] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:58:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:59:35] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 222418 MB (4% inode=28%): [03:59:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:00:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66312 MB (6% inode=99%): [04:12:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:57:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:59:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:59:55] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:00:37] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 222058 MB (4% inode=28%): [05:00:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65980 MB (6% inode=99%): [05:07:15] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.756836/2.00, alarm hl:mem_free=213.000000M/250M [05:08:14] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:12:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:57:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:59:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:59:57] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:00:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65873 MB (6% inode=99%): [06:01:35] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220032 MB (4% inode=28%): [06:07:17] zzz [06:12:27] Load avg. on willow is WARNING: WARNING - load average: 16.10, 12.63, 10.33 [06:12:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:13:16] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.564941/1.50, alarm hl:mem_free=382.000000M/100M [06:13:25] Load avg. on willow is OK: OK - load average: 11.91, 12.04, 10.27 [06:14:14] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:18:16] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.322266/2.00, alarm hl:mem_free=139.000000M/250M [06:45:56] Sun Grid Engine execd on wolfsbane is WARNING: all.q@wolfsbane exceedes load threshold: alarm hl:np_load_avg=1.096191/1.00, alarm hl:mem_free=4071.000000M/100M [06:55:57] Sun Grid Engine execd on wolfsbane is OK: all.q@wolfsbane OK [06:57:16] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:59:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:59:57] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:00:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65738 MB (6% inode=99%): [07:01:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220260 MB (4% inode=28%): [07:14:45] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 79400 MB (8% inode=99%): [07:42:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:57:15] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:59:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:00:56] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:01:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220172 MB (4% inode=28%): [08:01:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65624 MB (6% inode=99%): [08:42:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [08:57:16] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:59:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:00:56] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:01:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 220086 MB (4% inode=28%): [09:01:55] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65521 MB (6% inode=99%): [09:41:14] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.121094/1.00, alarm hl:mem_free=14273.000000M/100M [09:42:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:46:15] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [09:58:14] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.225586/1.00, alarm hl:mem_free=13753.000000M/100M [09:58:15] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:59:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:01:44] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 219964 MB (4% inode=28%): [10:01:56] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:02:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65401 MB (6% inode=99%): [10:58:16] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:59:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:01:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 219821 MB (4% inode=27%): [11:02:07] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:03:55] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65288 MB (6% inode=99%): [11:12:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:58:17] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:59:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:01:46] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 218560 MB (4% inode=27%): [12:03:05] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:03:55] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65183 MB (6% inode=99%): [12:12:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:48:25] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.626465/1.50, alarm hl:mem_free=1423.000000M/100M [12:50:24] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [12:53:25] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.503418/1.50, alarm hl:mem_free=1464.000000M/100M [12:57:04] Load avg. on nightshade is WARNING: WARNING - load average: 23.00, 18.15, 13.46 [12:58:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:59:25] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:01:04] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 33.44, 23.46, 16.52 [13:02:15] Load avg. on nightshade is WARNING: WARNING - load average: 26.70, 23.80, 17.13 [13:02:46] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 213644 MB (4% inode=27%): [13:03:05] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:04:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65999 MB (6% inode=99%): [13:12:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:18:16] Load avg. on nightshade is OK: OK - load average: 7.05, 11.57, 14.55 [13:18:25] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [13:58:34] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:59:35] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:02:56] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 213517 MB (4% inode=27%): [14:03:15] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:05:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65881 MB (6% inode=99%): [14:12:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:58:37] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:59:37] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:03:06] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 213251 MB (4% inode=27%): [15:03:25] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:05:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65744 MB (6% inode=99%): [15:12:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:26:14] /sql on adenia is WARNING: DISK WARNING - free space: /sql 70509 MB (10% inode=99%): [15:58:36] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:59:15] /sql on adenia is OK: DISK OK - free space: /sql 72378 MB (11% inode=99%): [15:59:37] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:03:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 213083 MB (4% inode=27%): [16:04:25] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:05:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65648 MB (6% inode=99%): [16:42:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:58:46] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:59:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:03:06] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 212832 MB (4% inode=27%): [17:04:36] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:05:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65534 MB (6% inode=99%): [17:42:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:50:25] Load avg. on nightshade is WARNING: WARNING - load average: 27.66, 15.86, 10.87 [17:51:24] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 32.40, 18.89, 12.25 [17:51:36] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=2.379883/1.50, alarm hl:mem_free=1512.000000M/100M [17:52:25] Load avg. on nightshade is WARNING: WARNING - load average: 23.84, 18.73, 12.61 [17:58:46] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:59:44] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:03:06] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 212721 MB (4% inode=27%): [18:04:37] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:05:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65412 MB (6% inode=99%): [18:06:46] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [18:42:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:58:54] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:59:55] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:04:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 212450 MB (3% inode=27%): [19:05:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:06:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65333 MB (6% inode=99%): [19:15:05] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 78597 MB (8% inode=99%): [19:41:05] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:41:25] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:42:06] MySQL on z-dat-s7-a is OK: Uptime: 189 Threads: 4 Questions: 2254 Slow queries: 0 Opens: 254 Flush tables: 1 Open tables: 244 Queries per second avg: 11.925 [19:42:26] MySQL slave on z-dat-s7-a is OK: Uptime: 208 Threads: 5 Questions: 6370 Slow queries: 0 Opens: 331 Flush tables: 1 Open tables: 320 Queries per second avg: 30.625 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78 [19:42:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:58:55] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:59:55] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:04:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 212343 MB (3% inode=27%): [20:05:37] ok, so I have a long-running query and it keeps getting killed off [20:05:45] ideas for making it work? [20:06:16] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65386 MB (6% inode=99%): [20:06:17] query is: "SELECT * FROM image WHERE img_width=0 AND img_height=0" [20:06:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:09:37] hexmode: https://wiki.toolserver.org/view/Database_access#Slow_queries_and_the_query_killer [20:10:55] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.591797/2.00, alarm hl:mem_free=167.000000M/250M [20:11:55] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [20:13:41] DaBPunkt: thanks... only skimmed that page :P let me try that [20:14:34] hexmode: afair there is a link in the email you get ;) [20:14:58] which email? does the query killer send one? [20:15:11] yes [20:15:13] I haven't checked my email yet [20:15:43] * hexmode thinks maybe he shouldn't bury his head in the sand [20:22:16] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1957 [20:29:45] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1941 [20:35:05] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:35:44] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:36:56] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1961 [20:38:56] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.616699/2.00, alarm hl:mem_free=222.000000M/250M [20:39:58] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [20:42:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:50:17] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3623 [20:51:49] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1921 [20:58:50] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3602 [20:58:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:59:58] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:04:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 212245 MB (3% inode=27%): [21:06:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65064 MB (6% inode=99%): [21:06:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:06:57] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3660 [21:21:50] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3645 [21:42:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [21:50:17] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7195 [21:58:49] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6888 [21:58:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:59:57] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:05:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 212079 MB (3% inode=27%): [22:06:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:06:58] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5003 [22:07:17] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 64941 MB (6% inode=99%): [22:09:58] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3367 [22:13:58] MySQL slave on z-dat-s4-a is OK: Uptime: 2421004 Threads: 16 Questions: 103039870 Slow queries: 24415 Opens: 16528 Flush tables: 1 Open tables: 415 Queries per second avg: 42.560 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1654 [22:21:50] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6900 [22:42:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [22:50:37] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10660 [22:58:59] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9609 [22:59:07] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:00:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:05:57] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 211926 MB (3% inode=27%): [23:07:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:07:28] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65778 MB (6% inode=99%): [23:22:10] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9427 [23:28:32] nacht ts [23:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:50:37] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13708 [23:59:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6682 [23:59:17] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default