[09:40:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107311 MB (17% inode=99%): [09:51:01] /sql on cassia is WARNING: DISK WARNING - free space: /sql 124698 MB (10% inode=99%): [09:51:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:02:41] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (375860 kB) free! [10:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:05:41] / on willow is WARNING: DISK WARNING - free space: / 20915 MB (19% inode=98%): [10:05:41] /tmp on willow is WARNING: DISK WARNING - free space: / 20915 MB (19% inode=98%): [10:07:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:08:41] Free Memory on damiana is WARNING: WARNING - 5.1% (425384 kB) free! [10:09:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:12:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:13:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.34, 26.25, 26.26 [10:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1163074.000000 [10:18:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 210754 MB (3% inode=52%): [10:28:41] Free Memory on damiana is CRITICAL: CRITICAL - 4.6% (386092 kB) free! [10:29:41] Free Memory on damiana is WARNING: WARNING - 5.6% (469060 kB) free! [10:31:41] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (415696 kB) free! [10:37:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 107021 MB (17% inode=99%): [10:51:41] Free Memory on damiana is WARNING: WARNING - 6.5% (545464 kB) free! [10:51:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:05:42] /tmp on willow is WARNING: DISK WARNING - free space: / 20772 MB (19% inode=98%): [11:05:42] / on willow is WARNING: DISK WARNING - free space: / 20772 MB (19% inode=98%): [11:07:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:09:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:12:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:13:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.78, 27.65, 26.94 [11:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1165543.000000 [11:18:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 210261 MB (3% inode=52%): [11:23:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:24:41] Free Memory on damiana is OK: OK - 7.4% (619124 kB) free. [11:37:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106722 MB (17% inode=99%): [11:47:41] Free Memory on damiana is WARNING: WARNING - 6.5% (544120 kB) free! [11:51:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:55:12] it's that time of year again, I need to renew my toolserver account [11:55:51] of course I don't remember my LDAP password, so I ssh wolfsbane, try resetpass, but it just says: cannot open LDAP secret: No such file or directory [11:56:55] is something broken just today, or has the instructions changed? [12:02:42] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (398132 kB) free! [12:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:05:41] / on willow is WARNING: DISK WARNING - free space: / 20640 MB (19% inode=98%): [12:05:42] /tmp on willow is WARNING: DISK WARNING - free space: / 20640 MB (19% inode=98%): [12:07:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:08:41] Free Memory on damiana is WARNING: WARNING - 5.1% (423756 kB) free! [12:09:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:09:41] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (410496 kB) free! [12:10:00] are we watching the rise and fall of the German toolserver? [12:12:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:13:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.48, 28.30, 27.18 [12:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1168223.000000 [12:18:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 209474 MB (3% inode=52%): [12:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:37:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:38:29] LA2: try run that on willow? [12:39:53] yes, that worked better. Why? the page https://wiki.toolserver.org/view/LDAP says Solaris, and wolfsbane runs solaris, right? [12:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106505 MB (17% inode=99%): [12:40:45] but wolfsbane is not a login server iirc [12:42:53] ? [12:44:03] ah, it's *listed* here, https://wiki.toolserver.org/view/Login_servers [12:44:09] but it says "web server", not login server [12:44:15] it's a web server, you're not supposed to do anything but view logs etc on it [12:44:29] I just went to that page, looked for some familiar name in the list, and picked wolfsbane [12:45:04] note that [[Login servers]] redirects to the "User servers" page [12:45:11] gaaah [12:46:22] now I tried to make that a bit more clear [12:51:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:02:41] Free Memory on damiana is WARNING: WARNING - 5.7% (473860 kB) free! [13:03:20] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:05:41] /tmp on willow is WARNING: DISK WARNING - free space: / 20510 MB (19% inode=98%): [13:05:41] / on willow is WARNING: DISK WARNING - free space: / 20510 MB (19% inode=98%): [13:07:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:09:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:12:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:13:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.98, 25.97, 26.25 [13:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1171401.000000 [13:18:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 208864 MB (3% inode=52%): [13:24:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:28:42] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (371052 kB) free! [13:30:41] Free Memory on damiana is WARNING: WARNING - 5.7% (481508 kB) free! [13:31:41] jimmyxu: thanks! [13:31:47] LA2: np [13:37:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106250 MB (17% inode=99%): [13:43:41] Free Memory on damiana is OK: OK - 7.1% (597736 kB) free. [13:47:03] DaB. * Re: [Toolserver-l] Sick [13:51:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:03:21] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:05:41] /tmp on willow is WARNING: DISK WARNING - free space: / 20376 MB (19% inode=98%): [14:05:42] / on willow is WARNING: DISK WARNING - free space: / 20376 MB (19% inode=98%): [14:07:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:09:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:12:11] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:13:41] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:17:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.86, 26.20, 26.02 [14:18:21] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1174982.000000 [14:18:41] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 208215 MB (3% inode=52%): [14:21:41] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 76481 MB (7% inode=99%): [14:33:41] Free Memory on damiana is WARNING: WARNING - 5.3% (442452 kB) free! [14:37:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:40:01] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105998 MB (17% inode=99%): [14:50:03] DaB. * Re: [Toolserver-l] Adding more roots [14:51:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:53:17] Hello all [14:54:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:54:03] Tim Landscheidt * Re: [Toolserver-l] Sick [15:35:03] Jeremy Baron * Re: [Toolserver-l] Sick [15:40:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105768 MB (17% inode=99%): [15:44:04] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (413216 kB) free! [15:46:04] Free Memory on damiana is WARNING: WARNING - 5.1% (430292 kB) free! [15:52:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:59:05] Free Memory on damiana is OK: OK - 7.4% (618232 kB) free. [16:03:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:03:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 207347 MB (3% inode=52%): [16:03:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:03:29] / on willow is OK: DISK OK - free space: / 60628 MB (57% inode=99%): [16:04:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:04:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 207337 MB (3% inode=52%): [16:04:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:05:04] / on willow is OK: DISK OK - free space: / 60626 MB (57% inode=99%): [16:06:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:06:21] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 207324 MB (3% inode=52%): [16:06:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:06:31] / on willow is OK: DISK OK - free space: / 60622 MB (57% inode=99%): [16:09:43] @replag [16:09:43] DaBPunkt: s1-rr-a-wd: 6m 21s [+0.00 s/s]; s1-user-wd: 6m 21s [+0.00 s/s]; s2-user-c: 1w 6d 16h 14m 30s [+0.94 s/s]; s2-user-wd: 8m 53s [+0.00 s/s]; s4-user-wd: 6m 21s [+0.00 s/s]; s5-user-c: 1w 6d 16h 14m 30s [+0.94 s/s]; s5-user-wd: 8m 53s [+0.00 s/s]; s6-user-wd: 6m 21s [+0.00 s/s] [16:09:44] DaBPunkt: s7-user-wd: 6m 21s [+0.00 s/s] [16:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:58:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:58:27] Free Memory on damiana is WARNING: WARNING - 5.1% (425016 kB) free! [17:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:27] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (341392 kB) free! [17:33:26] Free Memory on damiana is WARNING: WARNING - 5.1% (423956 kB) free! [17:40:27] Free Memory on damiana is OK: OK - 7.1% (596216 kB) free. [17:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 206254 MB (3% inode=52%): [17:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1188150.000000 [17:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105176 MB (17% inode=99%): [17:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.80, 22.23, 22.58 [17:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:58:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:00:04] Thehelpfulone * Re: [Toolserver-l] Adding more roots [18:00:04] Silke Meyer * Re: [Toolserver-l] Adding more roots [18:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:27] Free Memory on damiana is WARNING: WARNING - 5.1% (426852 kB) free! [18:12:27] Free Memory on damiana is OK: OK - 7.1% (597796 kB) free. [18:19:03] MZMcBride * Re: [Toolserver-l] Adding more roots [18:29:27] Free Memory on damiana is WARNING: WARNING - 5.3% (446860 kB) free! [18:40:27] Free Memory on damiana is OK: OK - 7.1% (593388 kB) free. [18:47:27] Free Memory on damiana is WARNING: WARNING - 6.8% (568232 kB) free! [18:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 205657 MB (3% inode=52%): [18:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1191750.000000 [18:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104914 MB (17% inode=99%): [18:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.77, 23.33, 23.36 [18:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:19:27] Free Memory on damiana is WARNING: WARNING - 5.8% (484064 kB) free! [19:27:27] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (372012 kB) free! [19:28:27] Free Memory on damiana is WARNING: WARNING - 5.4% (452060 kB) free! [19:38:03] DaB. * Re: [Toolserver-l] Adding more roots [19:39:27] Free Memory on damiana is OK: OK - 7.1% (594924 kB) free. [19:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 203851 MB (3% inode=52%): [19:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:57:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:57:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1195350.000000 [19:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104602 MB (17% inode=99%): [19:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.41, 22.05, 22.20 [19:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:59:27] Free Memory on damiana is WARNING: WARNING - 6.4% (537312 kB) free! [20:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:27] Free Memory on damiana is OK: OK - 7.2% (607428 kB) free. [20:29:07] Load avg. on ortelius is WARNING: WARNING - load average: 6.64, 16.46, 13.36 [20:30:07] Load avg. on ortelius is OK: OK - load average: 5.87, 14.44, 12.84 [20:38:14] @łreplag [20:38:19] @replag [20:38:19] DaBPunkt: s2-user-c: 1w 6d 20h 43m 7s [+1.00 s/s]; s5-user-c: 1w 6d 20h 43m 7s [+1.00 s/s] [20:54:03] Danny B. * Re: [Toolserver-l] Adding more roots [20:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 203127 MB (3% inode=52%): [20:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:57:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:57:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1198889.000000 [20:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104356 MB (17% inode=99%): [20:57:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.35, 21.79, 22.20 [20:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:58:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:51:27] /sql on cassia is WARNING: DISK WARNING - free space: /sql 125007 MB (10% inode=99%): [21:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 191561 MB (3% inode=52%): [21:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:57:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1202041.000000 [21:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103951 MB (17% inode=99%): [21:57:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.67, 22.85, 22.73 [21:58:06] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:58:27] Free Memory on damiana is WARNING: WARNING - 5.8% (487024 kB) free! [21:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:27] Free Memory on damiana is OK: OK - 7.4% (624144 kB) free. [22:09:03] Tim Landscheidt * Re: [Toolserver-l] Adding more roots [22:57:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 159079 MB (2% inode=52%): [22:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:57:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:57:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:57:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:57:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1205230.000000 [22:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103591 MB (17% inode=99%): [22:57:57] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.21, 21.88, 22.33 [22:58:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:58:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:03:37] SSH on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:28:27] Free Memory on damiana is WARNING: WARNING - 5.3% (441112 kB) free! [23:43:27] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422984 kB) free! [23:49:27] Free Memory on damiana is WARNING: WARNING - 5.1% (425464 kB) free! [23:50:27] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421360 kB) free! [23:57:17] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 188663 MB (3% inode=52%): [23:57:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:57:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:57:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:57:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:57:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:57:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1207596.000000 [23:57:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103250 MB (16% inode=99%): [23:57:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.62, 20.94, 21.32 [23:58:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:58:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:59:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio