[00:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:07:49] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:16:39] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 176471 MB (3% inode=54%): [00:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:27:29] / on ortelius is WARNING: DISK WARNING - free space: / 3947 MB (13% inode=92%): [00:38:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 357928.000000 [00:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:40:39] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:41:44] do I want to know the replag...? [00:41:46] @replag [00:41:47] stwalkerster: s1-rr-a-wd: 59s [-0.01 s/s]; s1-user: 1m 51s [-0.18 s/s]; s1-user-wd: 1m 5s [-0.01 s/s]; s2-user-c: error; s2-user-wd: 58s [-0.01 s/s]; s4-user-wd: 58s [-0.01 s/s]; s5-user-c: 4d 3h 23m 58s [-1.34 s/s]; s5-user-wd: 58s [-0.01 s/s] [00:41:48] stwalkerster: s6-user-wd: 1m 1s [-0.01 s/s]; s7-user-wd: 59s [-0.01 s/s] [00:42:01] oh, not too bad :) [00:42:28] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69587 MB (11% inode=99%): [00:49:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:54:33] Any ts-admins around? [00:55:03] Would like to process a new ssh key for my account, a la carte :) [00:55:24] Filing ticket on jira now [01:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:07:49] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:14:33] night [01:17:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 195412 MB (3% inode=54%): [01:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:23:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:27:29] / on ortelius is WARNING: DISK WARNING - free space: / 3894 MB (12% inode=92%): [01:38:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 348538.000000 [01:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:40:39] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:42:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67658 MB (11% inode=99%): [01:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:07:49] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:17:29] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 209852 MB (3% inode=55%): [02:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:27:29] / on ortelius is WARNING: DISK WARNING - free space: / 3835 MB (12% inode=92%): [02:38:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 322328.000000 [02:39:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:40:39] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [02:42:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69440 MB (11% inode=99%): [02:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:51:09] Load avg. on ortelius is WARNING: WARNING - load average: 22.70, 18.89, 14.81 [02:57:09] Load avg. on ortelius is OK: OK - load average: 9.32, 14.63, 14.58 [03:06:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:07:49] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:13:39] / on ortelius is CRITICAL: DISK CRITICAL - free space: / 3265 MB (10% inode=92%): [03:17:39] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 220742 MB (4% inode=55%): [03:18:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:20:39] / on ortelius is OK: DISK OK - free space: / 9127 MB (30% inode=92%): [03:22:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:23:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:38:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 317475.000000 [03:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:39:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:40:40] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:42:29] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69360 MB (11% inode=99%): [03:49:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:02:36] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 71681 MB (7% inode=99%): [04:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:08:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:18:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 229770 MB (4% inode=56%): [04:19:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:23:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:24:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:30:15] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7811&oldid=7790&rcid=21547 * 91.198.174.202 * (+430) (updated page) [04:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 309669.000000 [04:39:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [04:40:02] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:40:47] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [04:42:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69272 MB (11% inode=99%): [04:50:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:08:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:18:33] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 242727 MB (4% inode=57%): [05:19:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:23:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:24:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 298543.000000 [05:39:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:40:02] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:40:42] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:42:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69137 MB (11% inode=99%): [05:50:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:08:42] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:18:33] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 255784 MB (4% inode=58%): [06:19:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:23:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:24:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 286323.000000 [06:39:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:40:02] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:40:42] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:42:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69027 MB (11% inode=99%): [06:50:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:08:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:18:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 267521 MB (5% inode=58%): [07:19:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:23:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:24:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 272765.000000 [07:39:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:40:02] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:40:42] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:42:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68907 MB (11% inode=99%): [07:50:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:08:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:18:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 277468 MB (5% inode=59%): [08:19:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:23:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:24:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 265465.000000 [08:39:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:40:02] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:40:42] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:42:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68709 MB (11% inode=99%): [08:50:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:08:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:18:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 285800 MB (5% inode=59%): [09:19:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:23:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:24:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 250522.000000 [09:39:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:40:01] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:40:42] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:42:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68547 MB (11% inode=99%): [09:50:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:53:42] /sql on cassia is WARNING: DISK WARNING - free space: /sql 101409 MB (8% inode=99%): [10:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:08:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:18:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 292821 MB (5% inode=59%): [10:19:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:23:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:24:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 246172.000000 [10:39:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:40:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:42] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:42:32] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68334 MB (11% inode=99%): [10:50:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:08:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:18:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 300023 MB (5% inode=60%): [11:19:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:23:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:24:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 238564.000000 [11:39:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:40:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:40:52] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:42:36] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68200 MB (11% inode=99%): [11:50:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:08:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:18:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 306705 MB (5% inode=60%): [12:19:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:23:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:24:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 224987.000000 [12:39:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:40:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:40:52] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:42:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68045 MB (11% inode=99%): [12:50:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:08:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:18:43] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 312551 MB (5% inode=60%): [13:18:43] Load avg. on ptolemy is WARNING: WARNING - load average: 17.49, 17.53, 11.19 [13:19:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:23:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:24:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:33:06] Michael Andersen * Re: [Toolserver-l] Cron on submit [13:36:42] Load avg. on ptolemy is OK: OK - load average: 4.93, 13.96, 14.55 [13:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 201320.000000 [13:39:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:40:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:40:53] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:42:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67869 MB (11% inode=99%): [13:44:06] Marco Fleckinger * Re: [Toolserver-l] Cron on submit [13:51:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:07:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:09:02] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:16:56] hello all [14:16:58] @replag [14:16:59] DaBPunkt: s1-rr-a-wd: 1m 31s [+0.00 s/s]; s1-user-c: 10s [-0.01 s/s]; s1-user-wd: 1m 25s [+0.00 s/s]; s2-user-c: error; s2-user-wd: 1m 25s [+0.00 s/s]; s4-user-wd: 1m 31s [+0.00 s/s]; s5-user-c: 2d 4h 5m 35s [-3.48 s/s]; s5-user-wd: 1m 31s [+0.00 s/s] [14:17:00] DaBPunkt: s6-user-wd: 1m 26s [+0.00 s/s]; s7-user-wd: 1m 26s [+0.00 s/s] [14:18:42] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318273 MB (5% inode=60%): [14:19:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:23:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:24:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:34:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 319636 MB (6% inode=60%): [14:38:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 184164.000000 [14:39:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:40:42] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:40:52] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:42:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67726 MB (11% inode=99%): [14:51:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:09:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:21:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:24:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:24:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:35:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 323944 MB (6% inode=60%): [15:36:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:37:06] DaB. * Re: [Toolserver-announce] [Toolserver-l] New Rule: SGE-constraint for bots [15:39:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 170051.000000 [15:40:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:41:21] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:41:22] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:43:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67558 MB (11% inode=99%): [15:51:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:03:14] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 71702 MB (7% inode=99%): [16:09:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:11:25] aaaaand good morning [16:11:35] hello dschwen [16:11:43] 5pm here :P [16:12:17] so, I'm not sure if I should be proud to have freed 230GB on user-store by deleting cached tiles, or if I should be embarassed to have used up so much space in the first place [16:12:37] the first [16:12:40] 5pm, must be "the old world" [16:12:53] it's better to build soemthing awesome that is not quite as efficient than you would like it [16:12:55] dschwen: that depens if you plan to do it better in future [16:13:01] than to get everything right in the first place [16:13:03] I have switched to caching gzipped tiles now [16:13:18] so, that won't happen so quickly [16:13:24] plus I'll do cache pruning [16:14:23] I'll have to do some measurements though to find a goos balance between cache size and mapnik db load [16:21:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:25:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:25:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:26:05] hi [16:26:09] good afternoon [16:26:13] hi Mathias_WMDE [16:26:20] someone in the right timezone! :) [16:27:21] is there a proper way to access source code of a tool that has been published on the toolserver on someone elses' account? [16:29:44] you can usually browse his files [16:30:25] I get a permission denied when I log in via ssh [16:30:35] what are you trying to view? [16:31:30] hi Mathias_WMDE [16:33:07] http://toolserver.org/~apper/dpa/ [16:33:12] the code is gpl licensed [16:33:18] I wanted to view the source code [16:34:37] ah [16:34:38] got it [16:34:40] sorry, my fault [16:35:34] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 328997 MB (6% inode=60%): [16:37:04] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:39:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 163358.000000 [16:40:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:41:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:41:54] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:43:34] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67430 MB (11% inode=99%): [16:52:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:10:14] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:21:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:25:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:25:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:35:58] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 333990 MB (6% inode=61%): [17:39:57] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 159762.000000 [17:40:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:41:57] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:42:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:43:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67225 MB (11% inode=99%): [17:52:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:06:48] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:10:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:14:51] can I practically run rails on ts? [18:21:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:24:57] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67080 MB (10% inode=99%): [18:25:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:25:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:35:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 335695 MB (6% inode=61%): [18:39:57] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 139612.000000 [18:40:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:41:57] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:42:07] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:52:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:52:57] AzaToth: TS runs ZWS, so if you are able to, it won't be easy :-). [19:00:57] Load avg. on ortelius is WARNING: WARNING - load average: 4.13, 15.96, 15.41 [19:01:58] Load avg. on ortelius is OK: OK - load average: 3.54, 13.66, 14.63 [19:06:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:10:17] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:10:55] scfc_de: heh [19:11:34] scfc_de: though I have no idea at all what ZWS is [19:12:50] AzaToth: zeus web server [19:18:13] Zeus Web Server is a proprietary web server for Unix and Unix-like platforms (currently Solaris, FreeBSD, HP-UX and Linux). Support for AIX, Tru64, and Mac OS X was dropped on 10 June 2008.[1] ZWS seems to be dead since it hasn't been updated since January 2010,[2] and the company no longer offers a similar server solution. [19:18:27] seems like the perfect choise :-P [19:18:34] choice* [19:19:34] Alchimista: any reasons ts runs stuff few people would use? [19:20:52] AzaToth: i have no idea, but i think some parts are because at some time, who managed that part provably where more confortable with zews [19:21:00] k [19:21:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:22:35] Alchimista: are there still solaris experts running ts? [19:24:13] AzaToth: i hope so, we still some boxes running solaris. but peraps DaBPunkt can guide you better, he is a kind of ts God, at least one of them [19:24:35] I thought they all run solaris [19:24:57] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66852 MB (10% inode=99%): [19:25:15] I notice it takes ages to ssh to willow [19:25:27] is it over worked? [19:25:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:25:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:26:09] servers description: https://wiki.toolserver.org/view/Servers [19:27:18] pretty much everything except yarrow and nightshade is solaris [19:27:52] nightshade seems a lot faster [19:28:59] It's due to history. River, one of the first TS admins, was very good with Solaris (which Sun had also donated servers for) and had less knowledge with Linux, so there even was a drive to get rid of Linux altogether. [19:29:22] scfc_de: ah river, I forgot the name [19:29:36] river aint around any more I assume? [19:30:01] Nope. There has been talk to migrate to Apache, but there hasn't been any push for months. [19:30:28] (Ceterum censo: That's why we need more roots :-).) [19:30:38] hehe [19:30:43] well, if ts will be killed, why moving now? :P [19:32:09] it's to be killed? [19:32:14] You're right, ZWS is running pretty robustly. [19:33:11] and if rails doesn't work fine, python works perfectly with fcgi [19:33:21] or php, of course [19:33:46] *facepalm* [19:34:11] It's to be replaced by (Tool) Labs. Might even be a nice idea to set up a project for Ruby on Rails there. [19:34:18] ah [19:34:41] infortunelly that's true [19:35:24] DaBPunkt: Did you ever consider making the commons db a resource in sge? [19:35:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 337296 MB (6% inode=61%): [19:36:35] AzaToth: You might want to join Platonides' Webtools project: https://labsconsole.wikimedia.org/wiki/Nova_Resource:Webtools [19:37:47] multichill: there is already sql-s4-rr as a resource [19:37:59] Oh right [19:38:33] I knew I missed something ;-) [19:40:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122036.000000 [19:40:14] DaBPunkt: did you get money? [19:40:22] pardon? [19:40:39] " If this vote should fail (and we get no money for new hardware), I am going to retire from my job as root at 30. December 2012." [19:41:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:42:07] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:42:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:42:47] I think there was a follow-up e-mail. [19:42:58] WMDE agreed to support the Toolserver through at least 2013, I think. [19:43:16] WMF as well. [19:43:20] oki [19:43:36] I've been away too long [19:44:17] I've been here too long. [19:44:28] hehe [19:48:45] scfc_de: https://labsconsole.wikimedia.org/wiki/Nova_Resource_Talk:Webtools#rails.2Frack done [19:52:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:10:18] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:22:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:25:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66597 MB (10% inode=99%): [20:26:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:26:17] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:35:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 339021 MB (6% inode=61%): [20:41:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100447.000000 [20:42:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:42:17] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:42:17] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:52:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:10:28] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:22:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:25:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66323 MB (10% inode=99%): [21:26:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:26:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:36:08] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340221 MB (6% inode=61%): [21:41:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 92655.000000 [21:42:07] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:42:27] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:42:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:52:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:53:06] Christopher David Howie * Re: [Toolserver-l] Block mail from dysgo.org? [21:53:47] /sql on cassia is WARNING: DISK WARNING - free space: /sql 97598 MB (8% inode=99%): [22:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:10:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:17:00] is the source for add-mmt and add-mmt-maintainer available anywhere? [22:22:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:25:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66062 MB (10% inode=99%): [22:26:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:26:27] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:28:48] furry: On willow, "less $(which add-mmt)" (and ditto for -maintainer). [22:29:50] thak you [22:29:55] *thank you [22:36:07] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 341786 MB (6% inode=61%): [22:41:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91666.000000 [22:42:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:42:27] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:42:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:52:37] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:06:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:10:27] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:22:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:25:18] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65781 MB (10% inode=99%): [23:26:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:26:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:26:44] so, I switched all of my bot tasks over to the qcronsub system today [23:26:49] everything seems to work fine, except [23:27:00] I'm getting strange errors on some python scripts [23:27:16] errors that imply that my scripts are being run on python 2.6 instead of 2.7 [23:27:51] is there something special I have to do to get my python scripts to run on python 2.7 when they are run from SGE? [23:30:30] What is the script's shebang? [23:32:02] the python script or the shell file? [23:33:08] #!/bin/bash [23:33:52] hmm took a look on JIRA, TS-1462 seems to imply that I need to do a "arch='sol'" in order to get python 2.7 [23:33:55] How do you call the script? [23:34:05] python /path/to/script [23:36:04] Yep, totally forgot about that. You probably need to run it on Solaris. [23:36:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340466 MB (6% inode=62%): [23:36:36] Snottywong: the linux hosts run 2.6, not 2.7 [23:36:45] so depending on the execution host, you get a different python version [23:37:15] so you have to restrict the jobs to solaris (arch=sol , I think) [23:37:15] k thanks [23:37:36] oh, you already found that on JIRA [23:41:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95265.000000 [23:42:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:42:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:42:39] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:52:47] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default