[00:17:15] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 288890 MB (5% inode=67%):  
[00:21:45] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4765.000000  
[00:22:05] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10992.000000  
[00:24:15] <tsnag>	 wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: Unknown database wikidatawiki_p  
[00:24:35] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[00:25:15] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[00:27:05] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[00:27:45] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[00:28:55] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[00:29:45] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86492.000000  
[00:29:45] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[00:30:45] <tsnag>	 wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3525.000000  
[00:31:45] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[00:33:45] <tsnag>	 /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20221 MB (3% inode=99%):  
[00:42:45] <tsnag>	 wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1696.000000  
[00:53:05] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[01:17:15] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 291143 MB (5% inode=67%):  
[01:22:05] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9124.000000  
[01:24:15] <tsnag>	 wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: Unknown database wikidatawiki_p  
[01:24:35] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:25:15] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[01:27:05] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[01:27:46] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[01:29:05] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[01:29:45] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90093.000000  
[01:29:45] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[01:31:45] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[01:33:45] <tsnag>	 /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20089 MB (3% inode=99%):  
[01:36:39] <DaBPunkt>	 @replag
[01:36:39] <tsbot>	 DaBPunkt: s1-rr-a-wd: 50s [-0.00 s/s]; s1-user: 2m 41s [-0.02 s/s]; s1-user-wd: 50s [-0.00 s/s]; s2-rr: 29m 52s [+0.65 s/s]; s2-user: 29m 52s [+0.65 s/s]; s2-user-c: error; s2-user-wd: 2h 41m 53s [+0.63 s/s]; s3-user: 20s [-0.02 s/s]
[01:36:40] <tsbot>	 DaBPunkt: s3-user-wd: error; s4-user-wd: 50s [-0.00 s/s]; s5-user-c: 1m 1s [-0.02 s/s]; s6-user-wd: error; s7-user-wd: 50s [-0.00 s/s]
[01:36:55] <DaBPunkt>	 @replag
[01:36:56] <tsbot>	 DaBPunkt: s1-rr-a-wd: 1m 1s [+0.66 s/s]; s1-user: 2m 28s [-0.78 s/s]; s1-user-wd: 60s [+0.60 s/s]; s2-rr: 29m 45s [-0.42 s/s]; s2-user: 29m 45s [-0.42 s/s]; s2-user-c: error; s2-user-wd: 2h 41m 49s [-0.24 s/s]; s3-user-wd: error
[01:36:57] <tsbot>	 DaBPunkt: s4-user-wd: 60s [+0.60 s/s]; s5-user-c: 58s [-0.18 s/s]; s6-user-wd: error; s7-user-wd: 60s [+0.60 s/s]
[01:37:36] <DaBPunkt>	 acht ts
[01:37:38] <DaBPunkt>	 nacht ts
[01:40:15] <tsnag>	 /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70801 MB (7% inode=99%):  
[01:53:05] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[02:07:05] <tsnag>	 MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1808  
[02:17:14] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 290795 MB (5% inode=67%):  
[02:18:47] <Jasper_Deng>	 Does anyone know why the Wikidata copy db is apparently missing?
[02:21:36] <Susan>	 It got moved.
[02:21:46] <Susan>	 Read toolserver-l and/or toolserver-announce.
[02:22:05] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10413.000000  
[02:23:05] <tsnag>	 MySQL slave on z-dat-s2-b is OK: Uptime: 629773  Threads: 10  Questions: 542156653  Slow queries: 10912  Opens: 21336930  Flush tables: 1  Open tables: 64  Queries per second avg: 860.876 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795  
[02:24:15] <tsnag>	 wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42422.000000  
[02:24:34] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[02:25:15] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[02:27:05] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[02:27:45] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[02:29:44] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93692.000000  
[02:29:45] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[02:29:55] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[02:31:45] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[02:33:45] <tsnag>	 /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20207 MB (3% inode=99%):  
[02:52:05] <tsnag>	 wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3450.000000  
[02:53:04] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[03:02:52] <tsnag>	 wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1536.000000  
[03:17:15] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 290263 MB (5% inode=67%):  
[03:24:15] <tsnag>	 wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31928.000000  
[03:24:35] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:25:15] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[03:27:05] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[03:27:45] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[03:29:45] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 97292.000000  
[03:29:45] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[03:29:55] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[03:31:45] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[03:33:45] <tsnag>	 /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 19863 MB (3% inode=99%):  
[03:53:05] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[04:14:03] <tsnag>	 MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2128  
[04:16:03] <tsnag>	 wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2110.000000  
[04:18:12] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289182 MB (5% inode=67%):  
[04:24:23] <tsnag>	 wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18558.000000  
[04:25:33] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[04:26:13] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[04:27:06] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[04:28:03] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[04:30:04] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100912.000000  
[04:30:04] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[04:30:05] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[04:32:03] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[04:34:03] <tsnag>	 /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 19724 MB (3% inode=99%):  
[05:03:13] <tsnag>	 MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3622  
[05:06:23] <tsnag>	 wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3280.000000  
[05:07:13] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3630.000000  
[05:09:23] <tsnag>	 wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1431.000000  
[05:17:13] <tsnag>	 MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3379  
[05:17:13] <tsnag>	 wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3560.000000  
[05:18:13] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 290066 MB (5% inode=67%):  
[05:23:03] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[05:23:14] <tsnag>	 MySQL slave on z-dat-s2-b is OK: Uptime: 640585  Threads: 12  Questions: 561912112  Slow queries: 11098  Opens: 21730003  Flush tables: 1  Open tables: 64  Queries per second avg: 877.185 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1703  
[05:25:43] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[05:26:13] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[05:27:12] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[05:28:09] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[05:28:13] <tsnag>	 wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1653.000000  
[05:30:14] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[05:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104571.000000  
[05:31:03] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[05:32:03] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[05:34:02] <tsnag>	 /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 19602 MB (3% inode=99%):  
[06:18:14] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289941 MB (5% inode=67%):  
[06:23:03] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[06:25:53] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[06:26:13] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[06:27:13] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[06:28:03] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[06:30:13] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[06:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108171.000000  
[06:31:03] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[06:32:03] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[06:34:04] <tsnag>	 /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 19505 MB (3% inode=99%):  
[07:17:11] <Susan>	 @replag
[07:17:11] <tsbot>	 Susan: s1-rr-a-wd: 1m 11s [+0.00 s/s]; s1-user-wd: 1m 11s [+0.00 s/s]; s2-user-c: error; s2-user-wd: 1m 14s [-0.47 s/s]; s3-user: 1m 6s [+0.00 s/s]; s3-user-wd: error; s4-user-wd: 1m 12s [+0.00 s/s]; s5-user-c: 1m 8s [+0.00 s/s]
[07:17:12] <tsbot>	 Susan: s6-user-wd: 1m 11s [-]; s7-user: 14s [+0.00 s/s]; s7-user-wd: 1m 11s [+0.00 s/s]
[07:18:13] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289665 MB (5% inode=67%):  
[07:23:03] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[07:25:53] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[07:26:13] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[07:27:13] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[07:28:03] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[07:30:13] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[07:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 111772.000000  
[07:31:03] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[07:32:03] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[07:34:04] <tsnag>	 /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 19327 MB (3% inode=99%):  
[07:36:03] <tsnag>	 /sql on ptolemy is OK: DISK OK - free space: /sql 158692 MB (26% inode=99%):  
[08:18:13] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 289112 MB (5% inode=67%):  
[08:23:03] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[08:25:53] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[08:26:13] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri
[08:27:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[08:28:03] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[08:30:13] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[08:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 115371.000000  
[08:31:03] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[08:32:03] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[09:13:49] <Merlissimo>	 @replag
[09:13:50] <tsbot>	 Merlissimo: s1-rr-a-wd: 1m 9s [-0.00 s/s]; s1-user-wd: 1m 8s [-0.00 s/s]; s2-rr: 7m 10s [-0.05 s/s]; s2-user: 7m 10s [-0.05 s/s]; s2-user-c: error; s2-user-wd: 5m 14s [+0.03 s/s]; s3-user: 23s [-0.01 s/s]; s3-user-wd: error
[09:13:50] <tsbot>	 Merlissimo: s4-user-wd: 1m 8s [-0.00 s/s]; s5-user-c: 1m 26s [+0.00 s/s]; s6-user-wd: 1m 9s [-0.00 s/s]; s7-user-wd: 1m 8s [-0.00 s/s]
[09:18:13] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 288761 MB (5% inode=67%):  
[09:23:03] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[09:25:53] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:26:13] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[09:27:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[09:28:03] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[09:30:13] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[09:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118971.000000  
[09:31:04] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[09:32:03] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[10:18:13] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 288365 MB (5% inode=67%):  
[10:23:04] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[10:25:53] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:26:13] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[10:27:23] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[10:28:03] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[10:30:12] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[10:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122571.000000  
[10:31:03] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[10:32:02] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[11:18:13] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 286310 MB (5% inode=67%):  
[11:23:03] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[11:25:53] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[11:26:13] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[11:27:23] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[11:28:03] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[11:30:13] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[11:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126171.000000  
[11:31:03] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[11:32:03] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[12:18:13] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 285620 MB (5% inode=67%):  
[12:23:06] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[12:25:53] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:26:23] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[12:27:23] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[12:28:04] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[12:30:12] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[12:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 129771.000000  
[12:31:03] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[12:32:13] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[12:39:27] <Danny_B>	 @replag
[12:39:27] <tsbot>	 Danny_B: s1-rr-a-wd: 1m 7s [-0.00 s/s]; s1-user-wd: 1m 6s [-0.00 s/s]; s2-rr: 2m 6s [-0.02 s/s]; s2-user: 2m 6s [-0.02 s/s]; s2-user-c: error; s2-user-wd: 2m 29s [-0.01 s/s]; s3-user-wd: error; s4-user-wd: 1m 11s [+0.00 s/s]
[12:39:28] <tsbot>	 Danny_B: s5-user-c: 1m 29s [+0.00 s/s]; s6-user-wd: 1m 6s [-0.00 s/s]; s7-user-wd: 1m 6s [-0.00 s/s]
[13:18:13] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 284500 MB (5% inode=67%):  
[13:23:04] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[13:26:03] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:26:23] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[13:27:23] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[13:28:14] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[13:30:14] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[13:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133371.000000  
[13:31:03] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[13:32:13] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[13:41:13] <tsnag>	 /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70300 MB (7% inode=99%):  
[13:43:17] <DaBPunkt>	 Hello all
[14:18:13] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 283460 MB (5% inode=67%):  
[14:23:03] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[14:26:03] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[14:26:23] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[14:27:23] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[14:28:13] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[14:30:13] <tsnag>	 Load avg. on ptolemy is WARNING: WARNING - load average: 10.84, 15.48, 13.20  
[14:30:14] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[14:31:03] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136972.000000  
[14:31:13] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[14:31:14] <tsnag>	 Load avg. on ptolemy is OK: OK - load average: 10.97, 14.59, 13.02  
[14:32:13] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[14:54:06] <reba>	 Maciej Jaros * Re: [Toolserver-l] Short downtime for s2 today
[15:18:22] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 281723 MB (5% inode=67%):  
[15:26:12] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[15:27:22] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[15:28:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[15:29:12] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[15:30:22] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[15:31:12] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 140581.000000  
[15:32:12] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[15:33:12] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[15:53:13] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[16:18:22] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 279021 MB (5% inode=67%):  
[16:25:32] <DaBPunkt>	 I extract now the fesrh s5-dump
[16:26:08] <DanielK_WMDE_>	 DaBPunkt: hi! 
[16:26:12] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[16:26:16] <DanielK_WMDE_>	 does s5 include wikidata?
[16:26:22] <DaBPunkt>	 yes
[16:26:37] <DaBPunkt>	 since 2 days
[16:26:51] <DanielK_WMDE_>	 ah, yea, right!
[16:27:15] <DanielK_WMDE_>	 we had a schema change there too. two columns to be added to the view.
[16:27:22] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[16:27:23] <DanielK_WMDE_>	 ...of wb_terms
[16:27:28] <DanielK_WMDE_>	 is there already a ticket for that?
[16:28:13] <DaBPunkt>	 no, but I know about the shema-change. But if you have details (like visibility) it would be great if you could open one
[16:28:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[16:29:16] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[16:29:17] <DanielK_WMDE__>	 DaBPunkt: everything that got added should be visible
[16:29:24] <DanielK_WMDE__>	 maybe i'll look into it later tonight
[16:29:32] <DaBPunkt>	 ok
[16:30:22] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[16:31:12] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 144181.000000  
[16:32:12] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[16:33:13] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[17:18:24] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 279452 MB (5% inode=67%):  
[17:23:02] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[17:26:12] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:27:22] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[17:28:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[17:29:12] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[17:30:22] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[17:31:12] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147781.000000  
[17:32:12] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[17:33:13] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[18:18:23] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 281149 MB (5% inode=67%):  
[18:23:02] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[18:26:13] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[18:27:22] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[18:28:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[18:29:12] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[18:30:22] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[18:31:12] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 151382.000000  
[18:32:12] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[18:33:13] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[19:18:22] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 280411 MB (5% inode=67%):  
[19:23:02] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[19:26:12] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[19:27:22] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[19:28:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[19:29:12] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[19:30:22] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[19:31:12] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 154982.000000  
[19:32:13] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[19:33:12] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[19:36:26] <duh>	 DaBPunkt: i might have missed it in your email, but how long is the s5 import expected to take?
[19:37:00] <DaBPunkt>	 The import of s5 itself will be finished within 1h
[19:38:15] <DaBPunkt>	 the user-databases will be moved tomorrow
[19:38:30] <DaBPunkt>	 (I have to announce that  ~24h in adventage)
[19:45:23] <duh>	 ok
[19:45:26] <duh>	 thanks :)
[20:18:22] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 279759 MB (5% inode=67%):  
[20:23:01] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[20:26:12] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[20:27:22] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[20:28:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[20:29:12] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[20:30:22] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[20:31:06] <reba>	 DaB. * [Toolserver-l] 2. Try: sql-s5 (dewiki) will be read-only tomorrow
[20:31:12] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 158582.000000  
[20:32:13] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[20:33:12] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[20:36:23] <DaBPunkt>	 @replag
[20:36:23] <tsbot>	 DaBPunkt: (z-dat-s4-a): error; (z-dat-s5-b): error; s1-rr-a: 10m 20s [-]; s1-rr-a-wd: 1m 10s [-]; s1-user: 5m 21s [-]; s1-user-wd: 1m 20s [-]; s2-rr: 18m 46s [-]; s2-user: 18m 46s [-]
[20:36:24] <tsbot>	 DaBPunkt: s2-user-c: error; s2-user-wd: 12m 2s [-]; s3-user: 46s [-]; s3-user-wd: error; s4-user-wd: 1m 10s [-]; s5-user-c: 45s [-]; s6-user-wd: 1m 17s [-]; s7-user-wd: 1m 17s [-]
[20:37:32] <DaBPunkt>	 @replag
[20:37:33] <tsbot>	 DaBPunkt: (z-dat-s4-a): error; (z-dat-s5-b): error; s1-rr-a: 10m 14s [-0.09 s/s]; s1-rr-a-wd: 1m 13s [+0.04 s/s]; s1-user: 5m 3s [-0.26 s/s]; s1-user-wd: 1m 13s [-0.10 s/s]; s2-rr: 19m 48s [+0.90 s/s]; s2-user: 19m 48s [+0.90 s/s]
[20:37:34] <tsbot>	 DaBPunkt: s2-user-c: error; s2-user-wd: 12m 55s [+0.77 s/s]; s3-user: 28s [-0.26 s/s]; s3-user-wd: error; s4-user-wd: 1m 9s [-0.01 s/s]; s5-user-c: 43s [-0.03 s/s]; s6-user-wd: 1m 12s [-0.07 s/s]; s7-user-wd: 1m 14s [-0.04 s/s]
[20:41:06] <reba>	 John * [Toolserver-l] enwiki_p
[20:41:52] <DaBPunkt>	 @replag
[20:41:54] <tsbot>	 DaBPunkt: (z-dat-s4-a): error; (z-dat-s5-b): error; s1-rr-a: 9m 1s [-0.28 s/s]; s1-rr-a-wd: 53s [-0.08 s/s]; s1-user: 5m 18s [+0.06 s/s]; s1-user-wd: 56s [-0.07 s/s]; s2-rr: 23m 32s [+0.86 s/s]; s2-user: 23m 32s [+0.86 s/s]
[20:41:55] <tsbot>	 DaBPunkt: s2-user-c: error; s2-user-wd: 16m 7s [+0.74 s/s]; s3-user-wd: error; s4-user-wd: 56s [-0.05 s/s]; s5-user-c: 40s [-0.01 s/s]; s6-user-wd: 50s [-0.08 s/s]; s7-user-wd: 51s [-0.09 s/s]
[20:43:07] <DaBPunkt>	 @replag
[20:43:07] <tsbot>	 DaBPunkt: (z-dat-s4-a): error; (z-dat-s5-b): error; s1-rr-a: 9m 1s [-]; s1-rr-a-wd: 57s [-]; s1-user: 5m 35s [-]; s1-user-wd: 56s [-]; s2-rr: 24m 32s [-]; s2-user: 24m 32s [-]
[20:43:08] <tsbot>	 DaBPunkt: s2-user-c: error; s2-user-wd: 16m 58s [-]; s3-user: 21s [-]; s3-user-wd: error; s4-user-wd: 56s [-]; s5-user-c: 33s [-]; s6-user-wd: 56s [-]; s7-user-wd: 57s [-]
[20:53:13] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds)  
[20:56:13] <tsnag>	 MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2132  
[21:08:21] <Merlissimo>	 @replag
[21:08:21] <tsbot>	 Merlissimo: s1-rr-a: 6m 23s [-]; s1-rr-a-wd: 1m 9s [-]; s1-user: 7m 12s [-]; s1-user-wd: 1m 9s [-]; s2-rr: 30m 59s [-]; s2-user: 30m 59s [-]; s2-user-c: error; s2-user-wd: 17m 6s [-]
[21:08:22] <tsbot>	 Merlissimo: s3-user: 14s [-]; s3-user-wd: error; s4-user-wd: 1m 7s [-]; s5-user-c: 52s [-]; s6-user-wd: 1m 9s [-]; s7-user-wd: 1m 13s [-]
[21:09:13] <tsnag>	 MySQL slave on z-dat-s2-b is OK: Uptime: 697346  Threads: 6  Questions: 630761889  Slow queries: 12125  Opens: 23888534  Flush tables: 1  Open tables: 64  Queries per second avg: 904.517 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1779  
[21:18:22] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 279065 MB (5% inode=67%):  
[21:23:01] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[21:23:16] <DaBPunkt>	 the new s5-host's replag is down to 10h
[21:26:06] <reba>	 DaB. * Re: [Toolserver-l] enwiki_p
[21:26:12] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[21:27:22] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[21:28:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[21:29:12] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[21:30:22] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[21:31:13] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 162182.000000  
[21:32:13] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[21:33:12] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[21:39:06] <reba>	 DaB. * [Toolserver-announce] Short downtime of s2 tomorrow
[21:46:22] <DaBPunkt>	 the new s5-host's replag is down to 5h
[21:53:12] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds)  
[22:04:41] <DaBPunkt>	 the new s5-host's replag is down to 1h
[22:08:15] <DaBPunkt>	 ok, it's live
[22:08:23] <Susan>	 \o/
[22:08:24] <DaBPunkt>	 stopping it now for wikidata-dump
[22:18:22] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 278663 MB (5% inode=67%):  
[22:23:01] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[22:23:01] <DaBPunkt>	 ok, dump is done. Stopping now wikidata-replication everywhere and re-import the dump
[22:26:12] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[22:27:22] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio
[22:28:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[22:29:13] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[22:30:22] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[22:31:12] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 165782.000000  
[22:32:12] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[22:33:12] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[22:51:42] <tsnag>	 wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2636.000000  
[22:53:12] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds)  
[22:54:42] <tsnag>	 wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2817.000000  
[22:57:32] <tsnag>	 wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2986.000000  
[22:58:12] <tsnag>	 wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3027.000000  
[23:00:32] <tsnag>	 wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3167.000000  
[23:00:52] <tsnag>	 wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3187.000000  
[23:01:12] <tsnag>	 wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3206.000000  
[23:03:51] <jonas_agx>	 Hi! I received my TS account that week, but I can't access the ptwiki_p database, how can I request it. My username is jonas_agx
[23:05:20] <DaBPunkt>	 where is the problem?
[23:07:52] <tsnag>	 wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000  
[23:08:13] <tsnag>	 wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000  
[23:08:13] <tsnag>	 wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000  
[23:08:32] <tsnag>	 wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3647.000000  
[23:08:32] <tsnag>	 wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3647.000000  
[23:08:42] <tsnag>	 wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3657.000000  
[23:08:42] <tsnag>	 wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3657.000000  
[23:10:42] <jonas_agx>	 DaBPunkt, my access to ptwiki_p is not allowed
[23:11:09] <DaBPunkt>	 jonas_agx: which command did you try?
[23:13:12] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1990  
[23:13:12] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1990.000000  
[23:17:31] <jonas_agx>	 DaBPunkt, sql ptwiki_p
[23:18:04] <DaBPunkt>	 and how is your login-name?
[23:18:12] <jonas_agx>	 jonas_agx
[23:18:22] <tsnag>	 /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 276422 MB (5% inode=67%):  
[23:19:42] <DaBPunkt>	 I see the problem. Wait a moment
[23:19:55] <jonas_agx>	 Okay, Dank!
[23:20:37] <DaBPunkt>	 ok, try now
[23:21:09] <jonas_agx>	 It's fine DaBPunkt ! Thank you
[23:21:15] <DaBPunkt>	 np
[23:23:02] <tsnag>	 Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL  
[23:26:13] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[23:27:22] <tsnag>	 CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:  
[23:28:22] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[23:29:12] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[23:30:22] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[23:30:32] <tsnag>	 s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2001.000000  
[23:30:42] <tsnag>	 MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2007  
[23:32:12] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[23:33:13] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[23:53:12] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds)  
[23:59:42] <tsnag>	 wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3519.000000