[00:00:30] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [00:06:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10183 [00:08:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10262.000000 [00:14:59] Free Memory on turnera is WARNING: WARNING - 5.1% (424520 kB) free! [00:16:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422864 kB) free! [00:17:59] Free Memory on turnera is WARNING: WARNING - 5.1% (426004 kB) free! [00:28:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3522.000000 [00:33:09] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:33:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:33:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 46088 MB (7% inode=99%): [00:39:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7366.000000 [00:40:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:43:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:43:58] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:43:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 509219 MB (9% inode=43%): [00:45:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:46:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:52:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.1% (339516 kB) free! [00:53:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:53:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [00:55:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:58:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:59:29] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [01:01:29] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [01:06:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13469 [01:06:58] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1787.000000 [01:08:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13546.000000 [01:16:58] Free Memory on turnera is WARNING: WARNING - 5.1% (428276 kB) free! [01:17:58] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422264 kB) free! [01:18:58] Free Memory on turnera is WARNING: WARNING - 5.1% (430480 kB) free! [01:33:26] Krinkle * Re: [Toolserver-l] Future of the toolserver [01:33:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45416 MB (7% inode=99%): [01:34:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:34:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:40:29] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8806.000000 [01:40:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:43:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:43:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 509132 MB (9% inode=43%): [01:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:46:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:47:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:50:59] Free Memory on turnera is CRITICAL: CRITICAL - 4.1% (347512 kB) free! [01:53:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [01:53:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:58:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:00:29] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [02:02:28] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [02:07:49] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16818 [02:07:59] Free Memory on turnera is WARNING: WARNING - 5.1% (423572 kB) free! [02:08:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16840.000000 [02:21:58] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (392968 kB) free! [02:25:58] Free Memory on turnera is WARNING: WARNING - 5.1% (423760 kB) free! [02:34:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:34:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:34:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45871 MB (7% inode=99%): [02:40:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11547.000000 [02:40:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:43:58] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:43:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 507174 MB (9% inode=43%): [02:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:46:29] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:47:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:53:58] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [02:53:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:56:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:58:29] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:59:59] Free Memory on turnera is WARNING: WARNING - 6.5% (540604 kB) free! [03:00:28] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [03:02:29] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [03:07:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20045 [03:08:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20069.000000 [03:21:00] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417804 kB) free! [03:34:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:34:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:34:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45794 MB (7% inode=99%): [03:40:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11105.000000 [03:40:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:43:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:43:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506214 MB (9% inode=43%): [03:44:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:46:30] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:47:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:53:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:54:00] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [03:56:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:58:31] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:00:30] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [04:02:30] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [04:07:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22545 [04:08:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22561.000000 [04:14:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107525 MB (11% inode=99%): [04:20:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106823 MB (10% inode=99%): [04:21:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.4% (200292 kB) free! [04:28:49] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3409.000000 [04:28:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107485 MB (11% inode=99%): [04:33:49] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1525.000000 [04:34:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45734 MB (7% inode=99%): [04:35:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:35:09] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:36:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106861 MB (10% inode=99%): [04:41:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:44:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:44:58] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:44:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 507004 MB (9% inode=43%): [04:46:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:47:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:54:08] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:54:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [04:56:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:58:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:58:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 104974 MB (10% inode=99%): [05:00:38] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [05:02:41] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [05:08:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22555.000000 [05:08:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22519 [05:17:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107758 MB (11% inode=99%): [05:21:59] Free Memory on turnera is CRITICAL: CRITICAL - 2.3% (191616 kB) free! [05:34:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45669 MB (7% inode=99%): [05:36:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:36:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:41:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:44:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:44:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:44:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506848 MB (9% inode=43%): [05:46:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:47:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:54:08] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [05:56:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:58:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:00:40] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [06:02:39] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [06:06:01] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 3651 MB (0% inode=92%): [06:07:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106833 MB (10% inode=99%): [06:08:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22118.000000 [06:08:49] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22112 [06:08:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107007 MB (11% inode=99%): [06:12:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106920 MB (10% inode=99%): [06:21:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.0% (164252 kB) free! [06:32:25] Samuel Klein * Re: [Toolserver-l] Future of the toolserver [06:35:50] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45541 MB (7% inode=99%): [06:36:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:36:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:42:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:42:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107287 MB (11% inode=99%): [06:44:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:44:58] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:44:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506675 MB (9% inode=43%): [06:46:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:47:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:53:09] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 38801 MB (9% inode=99%): [06:54:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:54:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [06:56:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:58:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:00:38] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [07:02:39] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [07:06:49] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 79736 MB (8% inode=98%): [07:06:49] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 79736 MB (8% inode=98%): [07:08:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20413.000000 [07:08:49] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20363 [07:21:59] Free Memory on turnera is CRITICAL: CRITICAL - 2.2% (182260 kB) free! [07:35:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45375 MB (7% inode=99%): [07:37:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:37:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:42:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:44:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:44:58] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:44:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506570 MB (9% inode=43%): [07:46:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:47:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:54:08] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:55:58] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [07:56:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:56:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106737 MB (10% inode=99%): [07:58:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:00:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107389 MB (11% inode=99%): [08:01:39] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [08:02:38] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [08:04:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106895 MB (10% inode=99%): [08:08:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14166.000000 [08:08:49] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14075 [08:09:25] Andrei Cipu * Re: [Toolserver-l] Future of the toolserver [08:21:59] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (250576 kB) free! [08:24:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107141 MB (11% inode=99%): [08:35:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 45218 MB (7% inode=99%): [08:37:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:38:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:42:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:44:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:44:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506361 MB (9% inode=43%): [08:45:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:47:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:48:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:54:08] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:54:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106921 MB (10% inode=99%): [08:55:58] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [08:56:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:58:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:58:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108093 MB (11% inode=99%): [09:00:48] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:00:48] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:01:08] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:01:39] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [09:02:38] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [09:03:49] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3582 [09:04:18] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3510.000000 [09:06:49] Load avg. on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:06:49] /tmp on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:06:49] / on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:06:49] /tmp on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:06:58] / on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:06:58] Load avg. on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:08] Load avg. on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:08] / on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:08] RAID on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:08] /tmp on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:08] / on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:08] /tmp on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:08] Load avg. on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:09] / on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:18] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:18] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:18] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:18] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:18] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:18] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:07:38] Environment IPMI on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:49] /tmp on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:49] /sql on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:07:49] Load avg. on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:08:07] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:08:17] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:08] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [09:09:08] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:18] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:18] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:28] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [09:09:28] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [09:09:28] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [09:09:48] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 159 MB (99% inode=99%): [09:09:48] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 7804 MB (26% inode=85%): [09:09:48] Load avg. on z-dat-s4-a is OK: OK - load average: 0.00, 0.09, 0.59 [09:09:48] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 159 MB (99% inode=99%): [09:09:48] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 79728 MB (8% inode=98%): [09:09:58] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [09:09:58] / on z-dat-s3-a is OK: DISK OK - free space: / 7804 MB (26% inode=85%): [09:09:58] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 79728 MB (8% inode=98%): [09:09:59] Load avg. on z-dat-s3-a is OK: OK - load average: 0.00, 0.09, 0.58 [09:09:59] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 174 MB (99% inode=99%): [09:09:59] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [09:09:59] / on z-dat-s6-a is OK: DISK OK - free space: / 7804 MB (26% inode=85%): [09:10:00] Load avg. on z-dat-s7-a is OK: OK - load average: 0.04, 0.09, 0.58 [09:10:00] Load avg. on z-dat-s6-a is OK: OK - load average: 0.04, 0.09, 0.58 [09:10:01] / on z-dat-s4-a is OK: DISK OK - free space: / 7804 MB (26% inode=85%): [09:10:01] SMTP on hyacinth is OK: SMTP OK - 0.004 sec. response time [09:10:02] SMTP on z-dat-s7-a is OK: SMTP OK - 0.003 sec. response time [09:10:02] / on hyacinth is OK: DISK OK - free space: / 7804 MB (26% inode=85%): [09:10:03] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 13463 MB (99% inode=99%): [09:10:08] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:10:09] /tmp on hyacinth is OK: DISK OK - free space: /tmp 13450 MB (99% inode=99%): [09:10:09] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [09:10:09] SMTP on z-dat-s4-a is OK: SMTP OK - 0.003 sec. response time [09:10:09] SMTP on z-dat-s3-a is OK: SMTP OK - 0.120 sec. response time [09:10:09] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 40278 MB (9% inode=99%): [09:10:28] MySQL on z-dat-s3-a is OK: Uptime: 224298 Threads: 15 Questions: 165460836 Slow queries: 122550 Opens: 2105887 Flush tables: 1 Open tables: 16384 Queries per second avg: 737.683 [09:10:48] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [09:14:28] MySQL on z-dat-s6-a is OK: Uptime: 253 Threads: 4 Questions: 147 Slow queries: 1 Opens: 50 Flush tables: 1 Open tables: 43 Queries per second avg: 0.581 [09:15:08] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2161 [09:17:08] MySQL slave on z-dat-s6-a is OK: Uptime: 415 Threads: 11 Questions: 26215 Slow queries: 4 Opens: 176 Flush tables: 1 Open tables: 164 Queries per second avg: 63.168 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1797 [09:18:18] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1755.000000 [09:18:48] MySQL slave on rosemary is OK: Uptime: 14752882 Threads: 16 Questions: 6519948766 Slow queries: 2007501 Opens: 273784 Flush tables: 6 Open tables: 3877 Queries per second avg: 441.944 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1622 [09:21:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.9% (245836 kB) free! [09:35:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44864 MB (7% inode=99%): [09:37:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:38:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:42:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:45:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:45:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 506103 MB (9% inode=43%): [09:46:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:47:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:48:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:54:08] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:55:58] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [09:56:39] MySQL slave on z-dat-s3-a is OK: Uptime: 227069 Threads: 20 Questions: 167025059 Slow queries: 123158 Opens: 2133216 Flush tables: 1 Open tables: 16368 Queries per second avg: 735.569 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1738 [09:56:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:58:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:01:39] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [10:02:39] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [10:10:39] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1875 [10:11:38] MySQL slave on z-dat-s7-a is OK: Uptime: 396351 Threads: 12 Questions: 69972281 Slow queries: 80439 Opens: 323455 Flush tables: 1 Open tables: 5703 Queries per second avg: 176.541 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1775 [10:21:58] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (176056 kB) free! [10:35:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44620 MB (7% inode=99%): [10:37:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:38:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:42:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:45:58] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:45:58] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505716 MB (9% inode=43%): [10:46:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:47:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:48:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:55:09] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:55:58] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [10:56:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:58:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:01:38] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [11:02:39] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [11:23:09] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (235304 kB) free! [11:35:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44372 MB (7% inode=99%): [11:37:09] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:38:08] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:42:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:45:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:45:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505527 MB (9% inode=43%): [11:47:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:47:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:49:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:55:08] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:56:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:56:59] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [11:58:06] ^^ [11:58:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:01:40] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [12:02:39] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [12:07:58] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2189.000000 [12:09:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106792 MB (10% inode=99%): [12:10:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107197 MB (11% inode=99%): [12:24:08] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (234368 kB) free! [12:24:25] Platonides * Re: [Toolserver-l] Future of the toolserver [12:36:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 44137 MB (7% inode=99%): [12:38:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:38:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:42:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:45:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:45:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 505201 MB (9% inode=43%): [12:48:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:48:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:49:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:54:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106922 MB (10% inode=99%): [12:55:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:56:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:57:58] APT on mayapple is WARNING: APT WARNING: 25 packages available for upgrade (0 critical updates). [12:58:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:02:39] APT on nightshade is WARNING: APT WARNING: 21 packages available for upgrade (0 critical updates). [13:02:39] APT on yarrow is WARNING: APT WARNING: 23 packages available for upgrade (0 critical updates). [13:04:58] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107860 MB (11% inode=99%): [13:04:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3613.000000 [13:10:41] hello all [13:24:58] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (253972 kB) free! [13:25:58] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:32:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106442 MB (10% inode=99%): [13:33:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107237 MB (11% inode=99%): [13:34:39] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:34:39] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:37:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43860 MB (7% inode=99%): [13:38:08] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:38:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:38:25] DaB. * [Toolserver-l] Reboot of the linux-boxes at Monday 19:05 UTC [13:42:49] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:43:27] DaBPunkt: is mayapple an loginserver? [13:43:51] *a [13:43:59] giftpflanze: no, the apache-test-server [13:44:38] ok [13:45:24] (you can login, but the same rules like for wolfsbane and ortelius occures – and there is no working apache at the moment) [13:45:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:46:08] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 504957 MB (9% inode=43%): [13:48:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:48:40] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:49:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:55:08] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:56:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:59:38] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:04:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5919.000000 [14:25:08] Free Memory on turnera is CRITICAL: CRITICAL - 2.5% (211192 kB) free! [14:37:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43620 MB (7% inode=99%): [14:39:09] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:39:09] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:43:48] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:45:59] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:46:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 504648 MB (9% inode=43%): [14:48:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:49:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:49:39] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:51:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106816 MB (10% inode=99%): [14:55:08] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:56:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:59:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:02:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 106938 MB (11% inode=99%): [15:04:30] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106893 MB (10% inode=99%): [15:04:31] Free Memory on turnera is WARNING: WARNING - 6.4% (533760 kB) free! [15:05:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7079.000000 [15:05:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.7% (229576 kB) free! [15:10:25] Samuel Klein * Re: [Toolserver-l] Future of the toolserver [15:12:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107936 MB (11% inode=99%): [15:34:26] Free Memory on turnera is WARNING: WARNING - 5.8% (489652 kB) free! [15:35:25] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (422680 kB) free! [15:38:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43293 MB (7% inode=99%): [15:39:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:44:47] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:46:04] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:47:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 504029 MB (9% inode=43%): [15:48:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:49:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:50:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:55:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:57:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:59:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:01:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106677 MB (10% inode=99%): [16:05:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8263.000000 [16:06:15] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:06:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:09:15] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:28:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:35:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.2% (269724 kB) free! [16:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:39:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:39:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42951 MB (7% inode=99%): [16:44:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:46:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:47:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 503677 MB (9% inode=43%): [16:48:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:49:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:50:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:55:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:58:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107124 MB (11% inode=99%): [16:59:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106839 MB (10% inode=99%): [16:59:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:05:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8225.000000 [17:25:26] Krinkle * Re: [Toolserver-l] Future of the toolserver [17:28:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:35:26] Free Memory on turnera is CRITICAL: CRITICAL - 3.1% (256356 kB) free! [17:39:25] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:39:25] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:39:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42674 MB (6% inode=99%): [17:44:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:46:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:47:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 503359 MB (9% inode=43%): [17:48:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:49:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:50:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:55:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:00:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:05:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8889.000000 [18:06:04] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 6420 MB (0% inode=92%): [18:28:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:34:26] Erik Moeller * Re: [Toolserver-l] Future of the toolserver [18:34:27] Erik Moeller * Re: [Toolserver-l] Future of the toolserver [18:35:25] Free Memory on turnera is CRITICAL: CRITICAL - 3.5% (294136 kB) free! [18:39:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:39:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42378 MB (6% inode=99%): [18:41:25] Marlen Caemmerer * [Toolserver-announce] Maintenance of s2 and s5 Monday, 1st October [18:45:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:46:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:47:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107729 MB (11% inode=99%): [18:47:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 503087 MB (9% inode=43%): [18:48:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:50:36] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:50:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:54:26] Free Memory on turnera is OK: OK - 9.5% (798000 kB) free. [18:54:27] Thehelpfulone * Re: [Toolserver-l] Future of the toolserver [18:55:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:01:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:05:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9956.000000 [19:07:25] Free Memory on turnera is CRITICAL: CRITICAL - 4.1% (342516 kB) free! [19:21:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106756 MB (10% inode=99%): [19:22:25] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107718 MB (11% inode=99%): [19:28:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:39:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:39:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42067 MB (6% inode=99%): [19:45:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:47:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:48:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 502760 MB (9% inode=43%): [19:49:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:50:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:50:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:55:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:01:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:05:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10348.000000 [20:07:26] Free Memory on turnera is CRITICAL: CRITICAL - 2.6% (218924 kB) free! [20:28:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:39:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:39:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41745 MB (6% inode=99%): [20:46:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:47:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:48:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 489058 MB (9% inode=42%): [20:49:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:50:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:50:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:55:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:01:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:05:25] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9252.000000 [21:06:01] @replag [21:06:02] DaBPunkt: s1-rr-a: 19s [+0.00 s/s]; s1-user: 19s [+0.00 s/s]; s2-user: 3h 47m 8s [-0.24 s/s]; s2-user-c: 2h 33m 40s [-0.03 s/s]; s3-rr-a: 14s [-]; s3-user: 14s [-]; s5-user-c: 2h 33m 40s [-0.03 s/s] [21:06:15] @replag [21:06:15] DaBPunkt: s1-rr-a: 19s [-]; s1-user: 19s [-]; s2-user: 3h 46m 51s [-1.34 s/s]; s2-user-c: 2h 33m 33s [-0.55 s/s]; s3-rr-a: 27s [+1.03 s/s]; s3-user: 27s [+1.03 s/s]; s5-user-c: 2h 33m 33s [-0.55 s/s] [21:07:25] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (394956 kB) free! [21:08:26] Free Memory on turnera is WARNING: WARNING - 5.1% (424808 kB) free! [21:10:25] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 39510 MB (9% inode=99%): [21:10:46] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 79448 MB (8% inode=98%): [21:10:46] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 79448 MB (8% inode=98%): [21:11:26] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (421348 kB) free! [21:25:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106909 MB (10% inode=99%): [21:28:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:39:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:40:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41414 MB (6% inode=99%): [21:46:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:47:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:48:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 498920 MB (9% inode=43%): [21:49:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:50:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:50:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:55:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:02:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:05:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8784.000000 [22:11:25] Free Memory on turnera is CRITICAL: CRITICAL - 2.8% (235216 kB) free! [22:17:06] nacht ts [22:28:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:32:25] Free Memory on turnera is WARNING: WARNING - 5.1% (427048 kB) free! [22:34:25] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (420900 kB) free! [22:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:39:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:40:45] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41138 MB (6% inode=99%): [22:46:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:47:06] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:48:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 498751 MB (9% inode=43%): [22:50:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:50:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:51:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:55:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:02:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:05:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8364.000000 [23:12:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107033 MB (11% inode=99%): [23:13:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106756 MB (10% inode=99%): [23:20:26] Free Memory on turnera is WARNING: WARNING - 5.1% (431108 kB) free! [23:21:25] Free Memory on turnera is CRITICAL: CRITICAL - 4.4% (371632 kB) free! [23:29:25] Carl (CBM) * Re: [Toolserver-l] Future of the toolserver [23:29:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:32:26] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 107394 MB (11% inode=99%): [23:39:26] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:40:26] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:40:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40925 MB (6% inode=99%): [23:43:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106866 MB (10% inode=99%): [23:47:05] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:47:46] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:48:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 497311 MB (9% inode=43%): [23:50:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:50:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:51:25] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 106985 MB (11% inode=99%): [23:51:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:54:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106920 MB (10% inode=99%): [23:55:15] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.