[00:02:28] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[00:03:28] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[00:03:37] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[00:07:07] <reba>	 DaB. * Re: [Toolserver-l] rosemary's copy of enwiki_p also corrupt?
[00:09:00] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[00:10:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[00:16:28] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[00:26:40] <mattbuck>	 is there something wrong with ts? flickr2commons (old and new) are painfully slow and buggy
[00:44:13] <mattbuck>	 yeah, definitely fucked
[00:51:30] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[00:53:38] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[00:55:30] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[00:55:30] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[00:55:30] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[00:55:30] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[00:57:38] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[01:02:28] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[01:03:29] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[01:03:47] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[01:08:57] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[01:10:30] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[01:16:28] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[01:30:07] <reba>	 Erik Moeller * Re: [Toolserver-l] rosemary's copy of enwiki_p also corrupt?
[01:51:29] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[01:53:39] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[01:53:57] <tsnag>	 /sql on rosemary is WARNING: DISK WARNING - free space: /sql 127275 MB (13% inode=99%):  
[01:55:28] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[01:55:28] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[01:55:28] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:55:28] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:58:37] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[02:02:32] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[02:04:32] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[02:04:47] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[02:08:57] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[02:10:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[02:16:28] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[02:18:09] <tsnag>	 Load avg. on ortelius is WARNING: WARNING - load average: 13.45, 15.27, 11.00  
[02:19:09] <tsnag>	 Load avg. on ortelius is OK: OK - load average: 13.12, 14.93, 11.16  
[02:51:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[02:54:38] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[02:55:29] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[02:55:30] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[02:55:30] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[02:55:30] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[02:56:57] <tsnag>	 /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 27566 MB (6% inode=99%):  
[02:58:37] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[03:02:28] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[03:05:35] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[03:05:37] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[03:08:57] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[03:11:33] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[03:16:29] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[03:53:46] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:54:27] <tsnag>	 /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:54:36] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[03:54:41] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:55:56] <tsnag>	 /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 28112 MB (6% inode=99%):  
[03:56:11] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59805 MB (6% inode=97%):  
[03:56:16] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59781 MB (6% inode=97%):  
[03:56:27] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[03:56:27] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[03:56:27] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:56:27] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:58:36] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[04:03:28] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[04:05:29] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[04:05:48] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[04:08:58] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[04:11:28] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[04:16:29] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[04:21:33] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[04:54:49] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[04:56:31] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[04:56:31] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[04:56:37] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[04:56:37] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[04:58:48] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[05:03:29] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[05:05:30] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[05:05:51] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[05:10:01] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[05:11:31] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[05:16:29] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[05:21:29] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[05:54:48] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[05:56:39] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[05:56:39] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[05:56:39] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[05:57:30] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[05:58:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[06:03:37] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[06:05:48] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[06:06:28] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[06:09:59] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[06:11:38] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[06:16:41] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[06:51:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[06:54:50] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[06:56:37] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[06:56:37] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[06:56:38] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[06:57:38] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[06:58:09] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 57636 MB (5% inode=97%):  
[06:58:17] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 56618 MB (5% inode=97%):  
[06:58:49] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[06:59:09] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 61810 MB (6% inode=97%):  
[06:59:18] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 61772 MB (6% inode=97%):  
[07:03:42] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[07:05:49] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[07:06:28] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[07:10:09] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[07:11:37] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[07:16:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[07:26:18] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58505 MB (5% inode=97%):  
[07:27:19] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58603 MB (6% inode=97%):  
[07:28:22] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58498 MB (5% inode=97%):  
[07:30:09] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58474 MB (5% inode=97%):  
[07:31:09] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58578 MB (6% inode=97%):  
[07:32:09] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58374 MB (5% inode=97%):  
[07:51:21] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58260 MB (5% inode=97%):  
[07:51:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[07:55:48] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[07:56:39] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[07:56:39] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[07:56:39] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[07:57:40] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[07:58:49] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[08:03:29] <tsnag>	 /sql on thyme is WARNING: DISK WARNING - free space: /sql 197068 MB (20% inode=99%):  
[08:03:43] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[08:05:49] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[08:07:27] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[08:10:09] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[08:11:09] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59679 MB (6% inode=97%):  
[08:11:22] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59679 MB (6% inode=97%):  
[08:11:39] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[08:13:09] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58511 MB (5% inode=97%):  
[08:13:21] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58482 MB (5% inode=97%):  
[08:14:09] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58533 MB (6% inode=97%):  
[08:14:20] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58533 MB (6% inode=97%):  
[08:16:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[08:34:19] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 57341 MB (5% inode=97%):  
[08:44:12] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 57918 MB (5% inode=97%):  
[08:56:38] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[08:56:38] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[08:56:38] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[08:56:52] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[08:57:38] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[08:59:51] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[09:04:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[09:05:49] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[09:07:28] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[09:10:09] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[09:12:38] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[09:16:09] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59278 MB (6% inode=97%):  
[09:16:23] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59255 MB (6% inode=97%):  
[09:16:39] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[09:21:29] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[09:56:39] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:56:39] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:56:40] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[09:56:49] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[09:58:38] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[10:00:48] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[10:04:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[10:06:49] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[10:08:27] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[10:10:11] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[10:13:38] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[10:16:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[10:21:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[10:57:38] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:57:38] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:57:38] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[10:57:50] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[10:58:38] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[11:00:49] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[11:04:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[11:06:48] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[11:08:27] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[11:10:09] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[11:13:42] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[11:16:40] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[11:21:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[11:57:38] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[11:57:39] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[11:57:39] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[11:57:49] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[11:58:38] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[12:01:48] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[12:04:39] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[12:06:48] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[12:08:28] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[12:11:12] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[12:13:42] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[12:16:40] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[12:21:27] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[12:26:48] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:27:17] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58963 MB (6% inode=97%):  
[12:39:18] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58501 MB (5% inode=97%):  
[12:40:18] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58494 MB (5% inode=97%):  
[12:57:38] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:57:38] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:57:39] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[12:57:49] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[12:58:42] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[13:02:51] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[13:04:39] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[13:06:49] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[13:08:29] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[13:11:09] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[13:14:38] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[13:15:13] <DaBPunkt>	 hello all
[13:17:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[13:19:00] <DaBPunkt>	 @replag
[13:19:01] <tsbot>	 DaBPunkt: s1-rr-a-wd: 32s [-0.19 s/s]; s1-user-wd: 32s [-0.23 s/s]; s2-user-wd: 4d 16h 39m 14s [+1.00 s/s]; s3-rr-a: 16s [-0.00 s/s]; s3-user: 16s [-0.00 s/s]; s4-user-wd: 2d 1h 23m 47s [-2.40 s/s]; s5-user-wd: 4d 16h 39m 14s [+1.00 s/s]
[13:19:26] <DaBPunkt>	 ok, look good
[13:21:27] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[13:36:09] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 60109 MB (6% inode=97%):  
[13:36:18] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 60105 MB (6% inode=97%):  
[13:51:29] <DaBPunkt>	 @replag
[13:51:29] <tsbot>	 DaBPunkt: s1-rr-a-wd: 28s [-0.00 s/s]; s1-user-wd: 28s [-0.00 s/s]; s2-user: 15s [+0.00 s/s]; s2-user-wd: 4d 17h 11m 42s [+1.00 s/s]; s3-rr-a: 1m 10s [+0.03 s/s]; s3-user: 1m 10s [+0.03 s/s]; s4-user-wd: 1d 22h 57m 14s [-4.51 s/s]; s5-user-wd: 4d 17h 11m 42s [+1.00 s/s]
[13:53:58] <tsnag>	 /sql on rosemary is WARNING: DISK WARNING - free space: /sql 126808 MB (13% inode=99%):  
[13:57:41] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:57:41] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:57:41] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[13:57:49] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[13:59:41] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[14:02:52] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[14:05:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[14:06:49] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[14:08:31] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[14:11:12] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[14:13:10] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58389 MB (5% inode=97%):  
[14:13:19] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58387 MB (5% inode=97%):  
[14:14:09] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59134 MB (6% inode=97%):  
[14:14:17] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59138 MB (6% inode=97%):  
[14:14:38] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[14:17:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[14:21:29] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[14:42:08] <reba>	 Platonides * Re: [Toolserver-l] rosemary's copy of enwiki_p also corrupt?
[14:45:09] <reba>	 Platonides * Re: [Toolserver-l] rosemary's copy of enwiki_p also corrupt?
[14:57:39] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[14:57:39] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[14:57:39] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[14:58:50] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[14:59:38] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[15:03:49] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[15:06:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[15:06:52] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[15:08:27] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[15:11:09] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[15:14:38] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[15:18:39] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[15:21:29] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[15:35:09] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58500 MB (5% inode=97%):  
[15:35:18] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58512 MB (5% inode=97%):  
[15:36:10] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58564 MB (6% inode=97%):  
[15:36:23] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58565 MB (6% inode=97%):  
[15:37:01] <DaBPunkt>	 @replag
[15:37:01] <tsbot>	 DaBPunkt: s1-rr-a-wd: 54m 54s [+0.52 s/s]; s1-user-wd: 53m 58s [+0.51 s/s]; s2-user-wd: 4d 18h 57m 15s [+1.00 s/s]; s3-rr-a: 14s [-0.01 s/s]; s3-user: 14s [-0.01 s/s]; s4-user-wd: 1d 20h 14m 33s [-1.54 s/s]; s5-user-wd: 4d 18h 57m 15s [+1.00 s/s]
[15:53:18] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58507 MB (5% inode=97%):  
[15:53:41] <DaBPunkt>	 @replag
[15:53:41] <tsbot>	 DaBPunkt: s1-rr-a-wd: 38m 22s [-0.99 s/s]; s1-user-wd: 36m 6s [-1.07 s/s]; s2-user-wd: 3d 1h 11m 19s [-150.48 s/s]; s3-rr-a: 2m 6s [+0.11 s/s]; s3-user: 2m 6s [+0.11 s/s]; s4-user-wd: 1d 20h 1m 21s [-0.79 s/s]; s5-user-wd: 3d 1h 11m 19s [-150.48 s/s]; s6-rr-a: 10s [+0.00 s/s]
[15:53:41] <tsbot>	 DaBPunkt: s6-user: 10s [+0.00 s/s]
[15:53:56] <DaBPunkt>	 @replag
[15:53:56] <tsbot>	 DaBPunkt: s1-rr-a-wd: 36m 42s [-6.41 s/s]; s1-user-wd: 34m 34s [-5.90 s/s]; s2-user-wd: 3d 1h 8m 21s [-11.42 s/s]; s3-rr-a: 2m 21s [+0.96 s/s]; s3-user: 2m 21s [+0.96 s/s]; s4-user-wd: 1d 20h 48s [-2.12 s/s]; s5-user-wd: 3d 1h 8m 21s [-11.42 s/s]
[15:55:59] <tsnag>	 /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 27748 MB (6% inode=99%):  
[15:56:11] <DaBPunkt>	 @replag
[15:56:12] <tsbot>	 DaBPunkt: s1-rr-a-wd: 24m 41s [-5.32 s/s]; s1-user-wd: 22m 42s [-5.25 s/s]; s2-user-wd: 3d 53m 3s [-6.77 s/s]; s3-rr-a: 38s [-0.76 s/s]; s3-user: 38s [-0.76 s/s]; s4-user-wd: 1d 19h 55m 39s [-2.28 s/s]; s5-user-wd: 3d 53m 3s [-6.77 s/s]
[15:58:40] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[15:58:40] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[15:58:40] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[15:59:39] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[15:59:48] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[16:01:09] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58476 MB (5% inode=97%):  
[16:02:09] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58547 MB (6% inode=97%):  
[16:03:09] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58497 MB (5% inode=97%):  
[16:03:48] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[16:06:42] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[16:06:49] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[16:08:28] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[16:11:09] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[16:12:26] <DaBPunkt>	 @replag
[16:12:27] <tsbot>	 DaBPunkt: s1-rr-a-wd: 1m 30s [-1.43 s/s]; s1-user-wd: 1m 30s [-1.30 s/s]; s2-user-c: 6m 43s [+0.01 s/s]; s2-user-wd: 3d 5m 40s [-2.92 s/s]; s3-rr-a: 40s [+0.00 s/s]; s3-user: 40s [+0.00 s/s]; s4-user-wd: 1d 19h 19m 36s [-2.22 s/s]; s5-user-c: 6m 43s [+0.01 s/s]
[16:12:28] <tsbot>	 DaBPunkt: s5-user-wd: 3d 5m 38s [-2.92 s/s]
[16:14:38] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[16:19:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[16:21:27] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[16:23:09] <DaBPunkt>	 @replag
[16:23:10] <tsbot>	 DaBPunkt: s1-rr-a-wd: 53s [-0.06 s/s]; s1-user-wd: 53s [-0.06 s/s]; s2-user: 13s [-0.00 s/s]; s2-user-wd: 2d 23h 21m 46s [-4.10 s/s]; s4-user-wd: 1d 18h 44m 44s [-3.25 s/s]; s5-user-wd: 2d 23h 21m 46s [-4.09 s/s]
[16:32:53] <DaBPunkt>	 @replag
[16:32:53] <tsbot>	 DaBPunkt: s1-rr-a-wd: 48s [-0.01 s/s]; s1-user-wd: 48s [-0.01 s/s]; s2-user-wd: 2d 22h 29m 31s [-5.50 s/s]; s3-rr-a: 1m 48s [+0.06 s/s]; s3-user: 1m 48s [+0.06 s/s]; s4-user-wd: 1d 18h 7m 46s [-3.89 s/s]; s5-user-wd: 2d 22h 29m 31s [-5.50 s/s]
[16:33:09] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58469 MB (5% inode=97%):  
[16:33:22] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58467 MB (5% inode=97%):  
[16:36:08] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59301 MB (6% inode=97%):  
[16:36:19] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59301 MB (6% inode=97%):  
[16:37:34] <DaBPunkt>	 brb
[16:38:25] <DaBPunkt>	 re
[16:58:39] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[16:58:39] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[16:58:39] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[16:59:09] <reba>	 DaB. * Re: [Toolserver-l] rosemary's copy of enwiki_p also corrupt?
[16:59:38] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[16:59:53] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[17:00:07] <reba>	 DaB. * Re: [Toolserver-l] rosemary's copy of enwiki_p also corrupt?
[17:02:56] <DaBPunkt>	 @replag
[17:02:57] <tsbot>	 DaBPunkt: s1-rr-a-wd: 57s [+0.00 s/s]; s1-user: 19s [+0.00 s/s]; s1-user-wd: 57s [+0.00 s/s]; s2-user-wd: 2d 21h 2m 46s [-2.86 s/s]; s3-rr-a: 1m 11s [-0.02 s/s]; s3-user: 1m 11s [-0.02 s/s]; s4-user-wd: 1d 16h 15m 49s [-3.70 s/s]; s5-user-wd: 2d 21h 2m 47s [-2.86 s/s]
[17:04:49] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[17:06:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[17:06:48] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[17:08:28] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[17:11:10] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[17:13:07] <reba>	 DaB. * Re: [Toolserver-l] Welcome: wikidata
[17:15:42] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[17:18:25] <jeremyb>	 DaBPunkt: how about nagios/ @replag / etc. for the trainwreck replacement?
[17:18:30] <jeremyb>	 does it have a name? where's the source code?
[17:19:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[17:21:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[17:30:46] <DaBPunkt>	 jeremyb: it is already in @replag (the "-wd" values). Nagios and source comes later. It is called expresstrain
[17:58:39] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:58:39] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:58:39] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[17:59:39] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[17:59:49] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[18:05:50] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[18:06:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[18:07:49] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[18:08:31] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[18:11:12] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[18:16:38] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[18:19:41] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[18:21:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[18:58:38] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[18:59:38] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[18:59:38] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[18:59:51] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[19:00:38] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[19:06:48] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[19:07:41] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[19:08:28] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[19:08:48] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[19:11:09] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[19:16:56] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[19:19:40] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[19:21:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[19:32:59] <DaBPunkt>	 @replag
[19:33:00] <tsbot>	 DaBPunkt: s1-rr-a-wd: 1h 12m 54s [+0.48 s/s]; s1-user-wd: 1h 14m 9s [+0.49 s/s]; s2-user-wd: 2d 17h 52m 47s [-1.27 s/s]; s3-rr-a: 24s [-0.01 s/s]; s3-user: 24s [-0.01 s/s]; s4-user-wd: 1d 14h 41m 52s [-0.63 s/s]; s5-user-wd: 2d 17h 52m 47s [-1.27 s/s]
[19:44:40] <DaBPunkt>	 @replag
[19:44:40] <tsbot>	 DaBPunkt: s1-rr-a-wd: 19m 15s [-4.59 s/s]; s1-user-wd: 21m 29s [-4.51 s/s]; s2-user-wd: 2d 17h 4s [-4.51 s/s]; s3-rr-a: 15s [-0.01 s/s]; s3-user: 15s [-0.01 s/s]; s4-user-wd: 1d 13h 56m 14s [-3.91 s/s]; s5-user-wd: 2d 17h 4s [-4.51 s/s]
[19:58:40] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[19:59:38] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[19:59:38] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[19:59:52] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[20:00:39] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[20:04:31] <tsnag>	 /sql on thyme is WARNING: DISK WARNING - free space: /sql 196834 MB (20% inode=99%):  
[20:05:45] <DaBPunkt>	 @replag
[20:05:45] <tsbot>	 DaBPunkt: s1-rr-a-wd: 55s [-0.87 s/s]; s1-user-wd: 55s [-0.98 s/s]; s2-user: 16s [+0.00 s/s]; s2-user-wd: 2d 14h 16m 27s [-7.77 s/s]; s4-user-wd: 1d 11h 23m 56s [-7.23 s/s]; s5-user-wd: 2d 14h 16m 27s [-7.77 s/s]
[20:05:49] <jongleur>	 DaBPunkt: did you remove the postgresql-jars already as announced? (if not, don't hurry, but it's not marked in TS-1500 yet, that's why  I ask)
[20:06:06] <DaBPunkt>	 jongleur: no, but I can now. Wait a moment
[20:06:30] <jongleur>	 don't hurry, I won't have the time to work on it today, I fear
[20:06:39] <jongleur>	 but would be great to know that it's done ;)
[20:06:47] <jongleur>	 ...when it's done
[20:06:49] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[20:07:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[20:08:30] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[20:09:48] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[20:11:12] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[20:12:35] <jeremyb>	 DaBPunkt: hah, keeping the transit names...
[20:13:43] <DaBPunkt>	 jongleur: done
[20:13:55] <jongleur>	 thanks
[20:13:59] <DaBPunkt>	 np
[20:16:24] <DaBPunkt>	 "java.sql.SQLException: No suitable driver found for jdbc:postgresql://sql-mapnik/osm_mapnik" oh oh
[20:16:39] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[20:19:08] <jongleur>	 DaBPunkt: at the lalm page? yes, I'm not prepared for the removal, yet ;)
[20:19:27] <DaBPunkt>	 jongleur: oh, that yours? OK :)
[20:19:37] <jongleur>	 yes
[20:19:38] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[20:19:52] <DaBPunkt>	 puh
[20:20:51] <jongleur>	 I have no postgres-stuff in the war, yet as I suggested to add postgis, no as they are removed I have to re-add them to the war, but that has to wait until tomorrow, as I have other more urgent work to do today ,)
[20:21:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[20:29:46] <Bryan>	 test
[20:33:45] <DaBPunkt>	 @replag
[20:33:45] <tsbot>	 DaBPunkt: s1-rr-a-wd: 31s [-0.01 s/s]; s1-user-wd: 31s [-0.01 s/s]; s2-user-wd: 2d 11h 30m 35s [-5.92 s/s]; s4-user-wd: 1d 9h 44m 17s [-3.56 s/s]; s5-user-wd: 2d 11h 30m 35s [-5.92 s/s]
[20:43:57] <DaBPunkt>	 @replag
[20:43:58] <tsbot>	 DaBPunkt: s1-rr-a-wd: 28s [-0.00 s/s]; s1-user-wd: 28s [-0.00 s/s]; s2-user-wd: 2d 10h 36m 43s [-5.27 s/s]; s3-rr-a: 1m 56s [+0.03 s/s]; s3-user: 1m 56s [+0.03 s/s]; s4-user-wd: 1d 9h 9m 46s [-3.38 s/s]; s5-user-wd: 2d 10h 36m 43s [-5.27 s/s]
[20:45:09] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58328 MB (5% inode=97%):  
[20:45:19] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58317 MB (5% inode=97%):  
[20:46:09] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 59466 MB (6% inode=97%):  
[20:46:18] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 59466 MB (6% inode=97%):  
[20:59:41] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[20:59:48] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[21:00:39] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[21:00:39] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[21:00:39] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[21:06:48] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[21:07:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[21:08:29] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[21:09:49] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[21:10:41] <tsnag>	 Load avg. on adenia is WARNING: WARNING - load average: 15.19, 15.15, 12.44  
[21:11:10] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[21:12:39] <tsnag>	 Load avg. on adenia is OK: OK - load average: 14.59, 14.91, 12.68  
[21:17:37] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[21:18:24] <DaBPunkt>	 @replag
[21:18:25] <tsbot>	 DaBPunkt: s1-rr-a-wd: 28s [-]; s1-user-wd: 28s [-]; s2-user-wd: 2d 7h 34m 24s [-5.29 s/s]; s4-user-wd: 1d 8h 1m 56s [-1.97 s/s]; s5-user-wd: 2d 7h 34m 24s [-5.29 s/s]
[21:20:42] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[21:21:29] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[21:21:41] <tsnag>	 Load avg. on adenia is WARNING: WARNING - load average: 14.02, 15.11, 13.90  
[21:59:43] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[22:00:40] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[22:00:40] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[22:00:40] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[22:00:48] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[22:05:08] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58423 MB (5% inode=97%):  
[22:05:18] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58394 MB (5% inode=97%):  
[22:06:48] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[22:07:38] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[22:08:29] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[22:09:48] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[22:10:11] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58867 MB (6% inode=97%):  
[22:10:18] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58867 MB (6% inode=97%):  
[22:12:10] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[22:17:39] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[22:20:41] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[22:41:11] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58510 MB (5% inode=97%):  
[22:41:18] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58510 MB (5% inode=97%):  
[22:42:23] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58593 MB (6% inode=97%):  
[22:43:10] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58700 MB (6% inode=97%):  
[22:50:32] <DaBPunkt>	 @replag
[22:50:32] <tsbot>	 DaBPunkt: s1-rr-a-wd: 1m 34s [+0.01 s/s]; s1-user: 2m 47s [+0.01 s/s]; s1-user-wd: 1m 35s [+0.01 s/s]; s2-user-wd: 2d 1h 39m 35s [-3.85 s/s]; s3-rr-a: 24s [-0.01 s/s]; s3-user: 24s [-0.01 s/s]; s4-user-wd: 1d 3h 55m 29s [-2.68 s/s]; s5-user-wd: 2d 1h 39m 35s [-3.85 s/s]
[22:51:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[22:59:38] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[23:00:39] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[23:00:39] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 21:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 21:  
[23:00:39] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[23:00:49] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[23:06:48] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[23:07:39] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[23:08:27] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[23:09:50] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[23:12:09] <tsnag>	 RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1  
[23:15:13] <DaBPunkt>	 @replag
[23:15:17] <tsbot>	 DaBPunkt: s1-rr-a-wd: 56s [-0.03 s/s]; s1-user-wd: 56s [-0.03 s/s]; s2-user-wd: 2d 19m 50s [-3.23 s/s]; s3-rr-a: 1m 38s [+0.05 s/s]; s3-user: 1m 38s [+0.05 s/s]; s4-user-wd: 1d 2h 29m 52s [-3.47 s/s]; s5-user-wd: 2d 19m 50s [-3.23 s/s]
[23:17:38] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[23:20:41] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[23:35:08] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: DISK CRITICAL - free space: /sql 58517 MB (5% inode=97%):  
[23:35:17] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: DISK CRITICAL - free space: /sql 58517 MB (5% inode=97%):  
[23:45:12] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 58634 MB (6% inode=97%):  
[23:45:17] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 58634 MB (6% inode=97%):  
[23:46:04] <DaBPunkt>	 @replag
[23:46:05] <tsbot>	 DaBPunkt: s1-rr-a-wd: 1m 10s [+0.01 s/s]; s1-user-wd: 1m 10s [+0.01 s/s]; s2-user-wd: 1d 22h 25m 4s [-3.72 s/s]; s3-rr-a: 46s [-0.03 s/s]; s3-user: 46s [-0.03 s/s]; s4-user-wd: 1d 59m 21s [-2.93 s/s]; s5-user-wd: 1d 22h 25m 4s [-3.72 s/s]
[23:51:28] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[23:59:38] <tsnag>	 Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.