[00:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:10:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:25:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65550 MB (10% inode=99%): [00:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:26:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:36:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 341118 MB (6% inode=62%): [00:41:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98866.000000 [00:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:42:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:43:37] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:51:04] test.m.wikipedia.org appears to be returning 503s [00:52:22] whoops wrong channel [00:52:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:55:25] nacht ts [01:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:10:38] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:25:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65317 MB (10% inode=99%): [01:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:26:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:36:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 343381 MB (6% inode=62%): [01:41:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102466.000000 [01:42:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:42:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:43:37] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:10:46] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:25:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65426 MB (10% inode=99%): [02:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:26:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:36:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 344200 MB (6% inode=62%): [02:41:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106065.000000 [02:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:42:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:43:37] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [02:52:46] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:06:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:10:48] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:25:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64997 MB (10% inode=99%): [03:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:26:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:36:18] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345535 MB (6% inode=62%): [03:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109668.000000 [03:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:42:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:43:37] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:52:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:04:07] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 71161 MB (7% inode=99%): [04:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:10:47] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:25:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64875 MB (10% inode=99%): [04:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 347599 MB (6% inode=62%): [04:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113268.000000 [04:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:42:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:43:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [04:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:10:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:26:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64808 MB (10% inode=99%): [05:26:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:26:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:37:18] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349375 MB (6% inode=63%): [05:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116871.000000 [05:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:42:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:43:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:10:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:26:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64696 MB (10% inode=99%): [06:26:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:26:46] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:31:47] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:31:57] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:31:58] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351748 MB (6% inode=63%): [06:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120470.000000 [06:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [06:42:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:43:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:53:06] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:10:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:26:16] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64519 MB (10% inode=99%): [07:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:31:47] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:31:57] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:31:57] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:37:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353823 MB (6% inode=63%): [07:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124070.000000 [07:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:42:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:43:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:53:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:10:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:22:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:26:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64307 MB (10% inode=99%): [08:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:31:47] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:31:56] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:31:57] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 355780 MB (6% inode=63%): [08:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127670.000000 [08:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:42:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:43:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:53:30] [[Special:Log/newusers]] create 10 * Nancyyom * (New user account) [09:03:33] [[User talk:Nancyyom]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7812&rcid=21549 * Nancyyom * (+3066) (Created page with "Addition Remedy Procedure You Could Not Find out. The catalyst medicine, also known as an excellent 'upper' is really a generally used medication in the united states these da...") [09:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:10:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:23:06] Marlen Caemmerer * Re: [Toolserver-l] Block mail from dysgo.org? [09:24:12] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Nancyyom10]]": spam) [09:24:23] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Nancyyom10]] with an expiry time of infinite (account creation disabled): spam) [09:26:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64098 MB (10% inode=99%): [09:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:31:46] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:31:57] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:31:57] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 356512 MB (6% inode=63%): [09:41:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 131271.000000 [09:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:42:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:43:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:49:57] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:53:57] /sql on cassia is WARNING: DISK WARNING - free space: /sql 97044 MB (8% inode=99%): [10:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:08:57] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:19:57] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:26:18] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63867 MB (10% inode=99%): [10:26:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:31:57] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:31:57] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:31:57] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:37:18] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 358027 MB (6% inode=63%): [10:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134871.000000 [10:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:42:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:43:46] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:50:07] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:08:57] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:19:57] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:22:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:26:18] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63616 MB (10% inode=99%): [11:26:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:31:57] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:31:57] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:31:57] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 359938 MB (6% inode=64%): [11:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138471.000000 [11:42:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:42:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:43:46] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:43:49] hello all [11:43:56] oh what a morning… [11:49:28] @relag [11:49:54] @replag [11:49:55] DaBPunkt: s1-rr-a-wd: 1m 4s [-0.00 s/s]; s1-user-wd: 1m 4s [-0.00 s/s]; s2-rr: 2m 55s [-0.03 s/s]; s2-user: 2m 55s [-0.03 s/s]; s2-user-c: error; s2-user-wd: 1m 8s [-0.00 s/s]; s4-user-wd: 1m 1s [-0.00 s/s]; s5-user-c: 1d 14h 36m 36s [-0.63 s/s] [11:49:56] DaBPunkt: s5-user-wd: 1m 8s [-0.00 s/s]; s6-user-wd: 1m 4s [-0.00 s/s]; s7-user: 10s [-0.01 s/s]; s7-user-wd: 1m 4s [-0.00 s/s] [11:50:07] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:08:57] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:19:57] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:26:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63281 MB (10% inode=99%): [12:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:31:57] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:31:57] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:31:57] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 361582 MB (6% inode=64%): [12:40:57] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:40:57] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:40:57] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 142071.000000 [12:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:42:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:43:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:45:17] Load avg. on ptolemy is WARNING: WARNING - load average: 23.81, 19.00, 12.27 [12:50:06] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:51:17] Load avg. on ptolemy is OK: OK - load average: 9.89, 14.35, 12.62 [12:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:04:06] DaB. * [Toolserver-announce] Short downtime for s2 today [13:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:08:57] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:18:53] [[Maintenance schedule]] 10https://wiki.toolserver.org/w/index.php?diff=7813&oldid=7633&rcid=21552 * Dab * (-672) (updated) [13:19:57] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:25:06] DaB. * Re: [Toolserver-l] Short downtime for s2 today [13:25:45] DaBPunkt: If you ever get a few free minutes, I'd appreciate a brief chat. [13:26:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63006 MB (10% inode=99%): [13:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 363201 MB (6% inode=64%): [13:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 145671.000000 [13:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:42:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:43:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:50:06] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:08:57] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:19:57] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:26:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62795 MB (10% inode=99%): [14:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 364557 MB (6% inode=64%): [14:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 149272.000000 [14:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:42:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:43:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:48:17] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.16, 26.21, 15.38 [14:50:06] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:51:17] Load avg. on ortelius is WARNING: WARNING - load average: 12.00, 22.51, 16.08 [14:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:54:17] Load avg. on ortelius is OK: OK - load average: 2.87, 13.60, 13.67 [15:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:08:56] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:18:46] [[Special:Log/newusers]] create 10 * Tuxnani * (New user account) [15:19:56] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:26:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62519 MB (10% inode=99%): [15:26:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:26:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:31:18] Load avg. on ortelius is WARNING: WARNING - load average: 21.95, 17.59, 12.50 [15:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 366429 MB (6% inode=64%): [15:41:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 152871.000000 [15:41:18] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.27, 21.21, 16.52 [15:42:17] Load avg. on ortelius is WARNING: WARNING - load average: 22.51, 20.81, 16.69 [15:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:42:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:43:57] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:50:07] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:04:18] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70902 MB (7% inode=99%): [16:05:17] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 30.33, 24.61, 20.73 [16:06:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:09:07] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:20:07] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:26:17] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62282 MB (10% inode=99%): [16:26:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:27:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:37:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 368475 MB (6% inode=64%): [16:41:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147694.000000 [16:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:42:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:43:57] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:45:39] scfc_de: what did I do? [16:46:25] scfc_de: I've unassigned myself from https://jira.toolserver.org/browse/TS-1608 - maybe you meant to assign DaBPunkt or nosy? [16:47:31] valhallasw: I guess he choosed you because you are the nearest to Merlissimo ;-) [16:48:17] Load avg. on ortelius is WARNING: WARNING - load average: 10.24, 14.38, 19.91 [16:50:07] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:06] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:01:18] Load avg. on ortelius is OK: OK - load average: 5.71, 9.84, 14.77 [17:01:47] Yes, exactly :-). [17:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:09:06] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:20:07] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:23:05] Load avg. on ortelius is OK: OK - load average: 3.88, 3.66, 7.16 [17:26:31] hi all! [17:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62025 MB (10% inode=99%): [17:27:06] Hi dschwen [17:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:27:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:28:15] / on z-dat-s2-b is CRITICAL: NRPE: Command check_root not defined [17:28:15] /sql on z-dat-s2-b is UNKNOWN: NRPE: Unable to read output [17:28:25] /tmp on z-dat-s2-b is UNKNOWN: NRPE: Unable to read output [17:28:35] Load avg. on z-dat-s2-b is CRITICAL: NRPE: Command check_load not defined [17:31:58] Connecting to willow is still very slow (>30s) [18:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 372313 MB (6% inode=65%): [18:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:22:35] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:35] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:35] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:22:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106737.000000 [18:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [18:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61785 MB (10% inode=99%): [18:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:27:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:28:15] / on z-dat-s2-b is CRITICAL: NRPE: Command check_root not defined [18:28:25] /tmp on z-dat-s2-b is UNKNOWN: NRPE: Unable to read output [18:28:35] Load avg. on z-dat-s2-b is CRITICAL: NRPE: Command check_load not defined [18:53:25] /tmp on z-dat-s2-b is CRITICAL: Connection refused by host [18:54:16] /sql on z-dat-s2-b is CRITICAL: Connection refused by host [18:55:15] / on z-dat-s2-b is OK: DISK OK - free space: / 2747 MB (71% inode=89%): [18:55:15] /sql on z-dat-s2-b is OK: DISK OK - free space: /sql 367521 MB (38% inode=99%): [18:55:25] /tmp on z-dat-s2-b is OK: DISK OK - free space: / 2747 MB (71% inode=89%): [18:55:35] Load avg. on z-dat-s2-b is OK: OK - load average: 5.85, 6.47, 5.85 [19:12:55] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2155 [19:22:24] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 374281 MB (7% inode=65%): [19:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:26] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:22:35] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:36] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:22:36] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:22:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:22:46] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 92434.000000 [19:22:55] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [19:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [19:25:35] APT on yucca is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61476 MB (10% inode=99%): [19:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:27:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:31:06] DaB. * Re: [Toolserver-announce] [Toolserver-l] Short downtime for s2 today [19:31:45] its → it's, sorry [19:33:15] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3211 [19:42:15] MySQL slave on z-dat-s2-b is OK: Uptime: 923 Threads: 6 Questions: 522381 Slow queries: 12 Opens: 74938 Flush tables: 1 Open tables: 64 Queries per second avg: 565.959 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1745 [20:02:09] DaBPunkt: dbreps's cronietabs on clematis and hawthorn are different. Is the sync between them broken? [20:09:54] (And how can I fix this in between - "cronie -r" on both, and then "cronie $FILE" on submit again?) [20:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 375606 MB (7% inode=65%): [20:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:22:35] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:36] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:22:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86696.000000 [20:23:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61195 MB (10% inode=99%): [20:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:27:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:27:40] scfc_de: they are a common file-system [20:27:48] are → share [20:28:02] I can not image how they can differ [20:42:01] If I run "diff -u <(ssh clematis.toolserver.org sudo -u dbreps cronie -l) <(ssh hawthorn.toolserver.org sudo -u dbreps cronie -l)", I get lots of differences. hawthorn's cronietab seems to have been stuck several hours ago. [20:43:10] ahhhhh, so tim does IRC! [20:43:39] how's nosy? i saw she blocked that domain (email). but i don't think i've seen anything else recently [20:44:25] DaBPunkt: what's up with the tampa box? [20:49:02] DaBPunkt: Actually, it looks as if hawthorn didn't pick up updates since some time after approximately Tue Feb 12 16:08:17. [20:49:09] scfc_de: I think I see the problem. [20:49:23] Please story a copy of the "correct" version [20:49:30] Already done. [21:07:33] scfc_de: it should be fixt now. You may want to restore the correct version [21:11:28] DaBPunkt: I just installed it on submit, and there are no differences between clematis and hawthorn. Thanks! [21:15:01] ok, I message the other 2 people which were affected by the problem [21:15:11] Sorry for the trouble [21:18:10] No problem. [21:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 377538 MB (7% inode=65%): [21:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:22:35] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:35] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64031.000000 [21:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:23:05] jeremyb_: running at the moment. We are in contact with the WMF to replace the broken memory [21:24:54] DaBPunkt: i believe they're waiting for you. have you talked to silke recently? [21:25:06] yes and yes [21:25:24] I will mail them later today [21:25:29] cool [21:25:36] have a good run :) [21:25:55] also, 14 20:43:39 < jeremyb_> how's nosy? i saw she blocked that domain (email). but i don't think i've seen anything else recently [21:26:01] still in hospital? [21:26:29] jeremyb_: no idea [21:26:40] :/ [21:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60926 MB (9% inode=99%): [21:27:01] But I hope that the TS is the least of her priorities at the moment [21:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:27:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:54:25] /sql on cassia is WARNING: DISK WARNING - free space: /sql 96475 MB (8% inode=99%): [22:22:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 379541 MB (7% inode=65%): [22:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:22:35] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:35] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49801.000000 [22:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:26:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60716 MB (9% inode=99%): [22:27:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:27:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:49:25] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2219.000000 [23:03:41] @replag [23:03:41] DaBPunkt: s1-rr-a-wd: 1m 7s [+0.00 s/s]; s1-user-wd: 1m 7s [+0.00 s/s]; s2-user-c: error; s2-user-wd: 51m 24s [+0.17 s/s]; s4-user-wd: 1m 4s [+0.00 s/s]; s5-user-c: 12h 29m 31s [-3.80 s/s]; s5-user-wd: 1m 7s [+0.00 s/s]; s6-user-wd: 1m 7s [+0.00 s/s] [23:03:42] DaBPunkt: s7-user-wd: 1m 7s [+0.00 s/s] [23:19:25] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1666.000000 [23:22:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 381861 MB (7% inode=66%): [23:22:25] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:25] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:22:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:22:35] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:35] SMTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38923.000000 [23:23:05] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:23:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:23:05] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:23:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:26:54] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60409 MB (9% inode=99%): [23:27:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:27:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:32:36] / on willow is WARNING: DISK WARNING - free space: / 21943 MB (20% inode=98%): [23:32:36] /tmp on willow is WARNING: DISK WARNING - free space: / 21943 MB (20% inode=98%):