[00:15:42] Load avg. on nightshade is WARNING: WARNING - load average: 15.78, 18.58, 17.13 [00:23:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:30:43] Load avg. on nightshade is OK: OK - load average: 12.74, 13.50, 14.95 [00:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 102496 MB (1% inode=66%): [00:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:45:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62265.000000 [00:46:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58693 MB (9% inode=99%): [00:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:46:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20328.000000 [00:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20314 [00:51:13] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [00:51:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [00:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [00:51:47] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1400699.000000 [00:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [00:51:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [00:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [01:02:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50276 MB (5% inode=99%): [01:22:42] Load avg. on nightshade is WARNING: WARNING - load average: 12.78, 16.86, 16.00 [01:23:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:34:42] Load avg. on nightshade is OK: OK - load average: 12.95, 14.20, 14.97 [01:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 102368 MB (1% inode=66%): [01:45:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63643.000000 [01:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57497 MB (9% inode=99%): [01:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:46:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20464.000000 [01:49:03] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20473 [01:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [01:51:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [01:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [01:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1384339.000000 [01:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [01:51:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [01:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [02:23:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 100138 MB (1% inode=66%): [02:45:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62014.000000 [02:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58257 MB (9% inode=99%): [02:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:46:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20060.000000 [02:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20042 [02:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [02:51:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [02:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [02:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1383592.000000 [02:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [02:51:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:52:03] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [02:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [03:23:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 99318 MB (1% inode=66%): [03:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62628.000000 [03:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58112 MB (9% inode=99%): [03:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:46:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19762.000000 [03:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19744 [03:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [03:51:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [03:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [03:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1381682.000000 [03:51:53] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [03:51:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:03] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [03:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [04:23:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 99925 MB (1% inode=66%): [04:45:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61260.000000 [04:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57963 MB (9% inode=99%): [04:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:46:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21281.000000 [04:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21290 [04:51:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [04:51:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [04:51:21] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [04:51:43] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1357846.000000 [04:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [04:51:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [04:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [05:15:42] Load avg. on nightshade is WARNING: WARNING - load average: 15.53, 15.04, 14.01 [05:20:43] Load avg. on nightshade is OK: OK - load average: 14.27, 14.88, 14.28 [05:23:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 99869 MB (1% inode=66%): [05:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:45:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61147.000000 [05:46:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57823 MB (9% inode=99%): [05:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:46:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21319.000000 [05:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21306 [05:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [05:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1345433.000000 [05:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [05:51:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:53] /sql on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [05:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [05:52:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [05:52:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [05:52:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [06:23:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:45:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:45:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 99775 MB (1% inode=66%): [06:45:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60430.000000 [06:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57660 MB (9% inode=99%): [06:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:46:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:48:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23260.000000 [06:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23261 [06:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [06:51:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1344276.000000 [06:51:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [06:51:52] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:52:01] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [06:52:13] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [06:52:13] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [06:52:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [07:13:43] Load avg. on nightshade is WARNING: WARNING - load average: 16.65, 16.57, 15.40 [07:23:01] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:27:53] Load avg. on nightshade is OK: OK - load average: 12.32, 14.11, 14.84 [07:44:53] Load avg. on nightshade is WARNING: WARNING - load average: 16.42, 15.51, 15.15 [07:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 99543 MB (1% inode=66%): [07:45:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:45:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57245.000000 [07:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57495 MB (9% inode=99%): [07:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:46:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:47:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:48:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23096.000000 [07:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23088 [07:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [07:51:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1345120.000000 [07:51:53] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [07:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [07:53:11] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [07:53:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [07:53:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [08:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:35:53] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2639 MB (20% inode=99%): [08:37:53] /tmp on ortelius is OK: DISK OK - free space: /tmp 3095 MB (22% inode=99%): [08:44:53] Load avg. on nightshade is WARNING: WARNING - load average: 17.67, 17.98, 18.53 [08:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:45:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 99382 MB (1% inode=66%): [08:45:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:45:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54475.000000 [08:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57298 MB (9% inode=99%): [08:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:46:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:48:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23170.000000 [08:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23162 [08:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [08:51:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1347507.000000 [08:51:53] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [08:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [08:53:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [08:53:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [08:53:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [09:02:53] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 23.86, 22.15, 20.01 [09:10:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2522 MB (19% inode=99%): [09:15:52] Load avg. on nightshade is WARNING: WARNING - load average: 14.58, 18.69, 19.95 [09:20:52] /tmp on ortelius is OK: DISK OK - free space: /tmp 2871 MB (21% inode=99%): [09:21:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 366464 MB (6% inode=66%): [09:21:53] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 22.94, 20.15, 20.08 [09:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:25:12] /mnt user-store on rosemary is OK: DISK OK - free space: /mnt 595916 MB (11% inode=71%): [09:26:02] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:52] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.099 second response time [09:33:53] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2528 MB (19% inode=99%): [09:44:52] /tmp on ortelius is OK: DISK OK - free space: /tmp 2850 MB (21% inode=99%): [09:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:45:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54310.000000 [09:45:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57085 MB (9% inode=99%): [09:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:46:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23035.000000 [09:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23014 [09:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [09:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:03] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [09:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1345679.000000 [09:52:53] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [09:53:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [09:53:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [09:53:12] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [09:55:53] Load avg. on nightshade is WARNING: WARNING - load average: 18.04, 16.55, 19.98 [10:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:23:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2299 MB (18% inode=99%): [10:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53800.000000 [10:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56835 MB (9% inode=99%): [10:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:46:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:46:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24563.000000 [10:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24570 [10:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [10:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:52:03] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [10:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1345952.000000 [10:52:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [10:53:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [10:53:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [10:53:13] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [10:55:52] Load avg. on nightshade is WARNING: WARNING - load average: 12.85, 14.21, 15.30 [11:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:23:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2400 MB (18% inode=99%): [11:45:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54190.000000 [11:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56508 MB (9% inode=99%): [11:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:47:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:47:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:48:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25636.000000 [11:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25616 [11:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [11:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [11:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1339177.000000 [11:52:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [11:53:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [11:53:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [11:53:13] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [11:55:52] Load avg. on nightshade is WARNING: WARNING - load average: 20.00, 17.93, 16.74 [12:15:53] /tmp on ortelius is OK: DISK OK - free space: /tmp 2859 MB (21% inode=99%): [12:22:52] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1292 MB (10% inode=99%): [12:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:23:53] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1459 MB (12% inode=99%): [12:35:52] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1269 MB (10% inode=99%): [12:45:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:45:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54146.000000 [12:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56285 MB (9% inode=99%): [12:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:48:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23580.000000 [12:48:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:48:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23588 [12:51:21] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [12:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [12:52:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1336595.000000 [12:52:53] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [12:53:13] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [12:53:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [12:53:23] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [12:55:52] Load avg. on nightshade is WARNING: WARNING - load average: 16.35, 16.63, 16.25 [13:02:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49901 MB (5% inode=99%): [13:02:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2474 MB (18% inode=99%): [13:03:23] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1812 [13:03:56] Load avg. on nightshade is OK: OK - load average: 12.76, 13.80, 15.00 [13:04:22] MySQL slave on z-dat-s5-b is OK: Uptime: 219102 Threads: 5 Questions: 435562487 Slow queries: 2903 Opens: 192733 Flush tables: 1 Open tables: 256 Queries per second avg: 1987.943 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1792 [13:13:53] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1309 MB (10% inode=99%): [13:17:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1863 MB (14% inode=99%): [13:20:53] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 936 MB (8% inode=99%): [13:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55999 MB (9% inode=99%): [13:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:46:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:46:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49941.000000 [13:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:48:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22659.000000 [13:48:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:48:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22662 [13:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [13:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [13:52:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1331383.000000 [13:52:53] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2206 MB (17% inode=99%): [13:52:53] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [13:53:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [13:53:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [13:53:22] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [14:04:53] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1171 MB (9% inode=99%): [14:05:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1345 MB (11% inode=99%): [14:18:38] uhm... why */10 on crontab (editor) doesn't work? :( [14:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:23:52] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1186 MB (10% inode=99%): [14:24:53] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1470 MB (12% inode=99%): [14:25:33] that's strange: ERROR 1345 (HY000): EXPLAIN/SHOW can not be issued; lacking privileges for underlying table [14:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55733 MB (9% inode=99%): [14:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:46:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:46:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45327.000000 [14:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21743.000000 [14:48:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:48:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21758 [14:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [14:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:03] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [14:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1332567.000000 [14:53:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [14:53:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [14:53:22] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [14:53:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [15:03:11] Gnumarcoo: crontab or cronietab? IIRC Solaris' crontab doesn't support this syntax. [15:04:22] fale: Not strange, cf. https://jira.toolserver.org/browse/TS-1585. [15:17:53] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1280 MB (10% inode=99%): [15:18:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1521 MB (12% inode=99%): [15:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:29:00] [[Database access]] ! 10https://wiki.toolserver.org/w/index.php?diff=7860&oldid=7659&rcid=21673 * Tim.Landscheidt * (+14) (/* Database schema */ Update link.) [15:31:53] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1289 MB (10% inode=99%): [15:34:04] Tim Landscheidt * Re: [Toolserver-l] No keys/indexes on wiki databases? [15:39:25] @replag [15:39:26] Betacommand: s1-user: 6h 50m 7s [+0.83 s/s]; s2-rr: 2m 22s [-0.02 s/s]; s2-user: 2m 22s [-0.02 s/s]; s2-user-c: error; s2-user-wd: 10h 50m 37s [-1.78 s/s]; s5-user-c: 2w 1d 10h 3m 52s [+0.00 s/s] [15:39:53] Load avg. on nightshade is WARNING: WARNING - load average: 19.76, 17.54, 15.50 [15:45:57] [[WikiProxy]] ! 10https://wiki.toolserver.org/w/index.php?diff=7861&oldid=4435&rcid=21674 * Tim.Landscheidt * (-540) (Set [[Template:Soft redirect]].) [15:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55518 MB (9% inode=99%): [15:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:46:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:46:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38937.000000 [15:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25150.000000 [15:48:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:48:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25166 [15:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [15:51:53] Load avg. on nightshade is OK: OK - load average: 13.67, 14.35, 15.00 [15:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [15:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1326530.000000 [15:53:13] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [15:53:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [15:53:29] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [15:53:52] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [15:54:47] [[Database access]] ! 10https://wiki.toolserver.org/w/index.php?diff=7862&oldid=7860&rcid=21675 * Tim.Landscheidt * (+187) (/* Page text access */ Add information on how to access page content.) [16:09:54] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2455 MB (18% inode=99%): [16:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:27:53] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1300 MB (10% inode=99%): [16:28:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1509 MB (12% inode=99%): [16:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 55262 MB (9% inode=99%): [16:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:46:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:46:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29985.000000 [16:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:48:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28645.000000 [16:48:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:48:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28663 [16:51:23] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [16:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:52:02] /tmp on z-dat-s1-b is UNKNOWN: NRPE: Unable to read output [16:52:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1313729.000000 [16:53:12] Load avg. on z-dat-s1-b is CRITICAL: NRPE: Command check_load not defined [16:53:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [16:53:22] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [16:53:53] / on z-dat-s1-b is CRITICAL: NRPE: Command check_root not defined [16:54:54] hello all [16:59:02] /tmp on z-dat-s1-b is OK: DISK OK - free space: / 2755 MB (71% inode=89%): [16:59:12] Load avg. on z-dat-s1-b is OK: OK - load average: 1.29, 1.26, 1.20 [16:59:53] / on z-dat-s1-b is OK: DISK OK - free space: / 2755 MB (71% inode=89%): [16:59:53] /sql on z-dat-s1-b is OK: DISK OK - free space: /sql 110263 MB (11% inode=99%): [17:15:33] @replag [17:15:36] Betacommand: s1-user: 8h 23m 47s [+0.97 s/s]; s1-user-wd: 32s [-0.01 s/s]; s2-rr: 4m 50s [+0.03 s/s]; s2-user: 4m 50s [+0.03 s/s]; s2-user-c: error; s2-user-wd: 7h 57m 47s [-1.80 s/s]; s3-user: 3m 15s [+0.00 s/s]; s5-user-c: 2w 1d 4h 26s [-3.78 s/s] [17:19:25] [[Special:Log/newusers]] create 10 * Jmaahir * (New user account) [17:20:11] the new enwiki-server has 75k seconds (~21h) replag at the moment [17:20:32] should be near null somewhen tonight or tomorrow morning [17:21:04] DaBPunkt: thanks [17:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:29:52] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2063 MB (16% inode=99%): [17:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54974 MB (9% inode=99%): [17:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:46:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:46:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28284.000000 [17:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:47:53] /tmp on ortelius is OK: DISK OK - free space: /tmp 10863 MB (87% inode=99%): [17:48:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31244.000000 [17:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31236 [17:49:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:49:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [17:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1303010.000000 [17:53:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [17:53:22] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [18:04:04] DaB. * [Toolserver-announce] /tmp is not a waste dump [18:10:21] Is there a resource for SGE jobs that forces a job to be executed on a specific host ("-l host=yarrow")? [18:15:04] Tim Landscheidt * Re: [Toolserver-l] /tmp is not a waste dump [18:20:04] Tim Landscheidt * Re: [Toolserver-l] /tmp is not a waste dump [18:20:17] hah, scfc_de (your last mail) [18:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:27:04] Andre Koopal * Re: [Toolserver-l] [Toolserver-announce] /tmp is not a waste dump [18:31:01] jeremyb_: My personal favourite in the last few weeks was when I stared for probably half an hour at inexplicable differences in a file in my editor and in a ssh session on Labs. I made several tests and thought of a glusterfs split brain, until I finally discovered that my editor was connected to webtools-login and my ssh session was to tools-login. *Argl*. [18:32:09] what's webtools-login? [18:32:21] i haven't tried the tools project at all yet [18:40:37] jeremyb_: Webtools was (is) Platonides' Tools variant (https://wikitech.wikimedia.org/wiki/Nova_Resource:Webtools). I had started to do some stuff there as he was quicker on setup :-). [18:41:02] oh [18:41:07] * jeremyb_ heads off for a while [18:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54749 MB (8% inode=99%): [18:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [18:46:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:46:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28305.000000 [18:47:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:48:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30137.000000 [18:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30130 [18:49:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:49:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [18:52:03] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1295428.000000 [18:53:12] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [18:53:22] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [19:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54510 MB (8% inode=99%): [19:46:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:46:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:47:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27580.000000 [19:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29788 [19:49:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29804.000000 [19:49:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:49:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:51:23] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [19:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:03] DaBPunkt, munin seems to not cleanup for itself, either [19:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1295013.000000 [19:53:13] MySQL on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [19:53:17] Platonides: example? [19:53:22] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [19:56:09] ortelius tmp $ ls -l | grep -c munin [19:56:10] 8886 [19:56:34] most of them 0-byte files, but I don't think they should be left there by the program [19:59:31] Platonides: I will look where the porblem is, thanks [20:18:04] DaB. * Re: [Toolserver-l] /tmp is not a waste dump [20:18:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1963.000000 [20:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:45:31] I just made a bash-scrip for personal use, that fetches the json data from stats.grok.se and produces a pdf, containing graphs for a number of pages that I specified… I Guess, I could produce a service, that allows users to subscribe with a mail-address and a list of pages that mails them how "their" articles are doing once a month or so. Here’s how the result looks for me. https://dl.dropbox.com/u/6443497/final.pdf [20:45:32] I’m not sure, if I’d like to apply for an own account though… It sounds like a lot of work and I’m an amateur at best, when it comes to coding. So I reckoned it’d be easiest if somebody just incorporated it to his_her own account. [20:46:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:46:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 54158 MB (8% inode=99%): [20:46:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:47:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:47:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:47:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23116.000000 [20:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29737 [20:49:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29741.000000 [20:49:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:50:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:51:22] MySQL slave on z-dat-s1-b is CRITICAL: Access denied for user tsnagios7643@turnera-bge0.esi.toolserver.org (using password: YES) [20:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1295293.000000 [20:53:12] MySQL on z-dat-s1-b is OK: Uptime: 99224 Threads: 1 Questions: 77468899 Slow queries: 1 Opens: 982 Flush tables: 1 Open tables: 204 Queries per second avg: 780.747 [20:53:22] APT on z-dat-s1-b is WARNING: APT WARNING: 2 packages available for upgrade (0 critical updates). [20:58:23] APT on z-dat-s1-b is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:18:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3013.000000 [21:21:04] Platonides * Re: [Toolserver-l] /tmp is not a waste dump [21:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:35:34] Platonides: Okay, okay, okay :-). [21:35:53] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2166.000000 [21:45:21] @replag [21:45:22] Susan: s1-user: 8h 9m 27s [-0.09 s/s]; s1-user-c: 58m 44s [+0.03 s/s]; s1-user-wd: 45m 37s [+0.27 s/s]; s2-rr: 6m 24s [-0.05 s/s]; s2-user: 6m 24s [-0.05 s/s]; s2-user-c: error; s2-user-wd: 4h 52m 19s [-1.07 s/s]; s3-user: 14s [+0.00 s/s] [21:45:22] Susan: s3-user-wd: 13s [-0.00 s/s]; s5-user-c: 2w 1d 29m 14s [+0.24 s/s] [21:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53916 MB (8% inode=99%): [21:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:47:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:47:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17520.000000 [21:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29287 [21:49:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29271.000000 [21:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:50:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:51:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31082 [21:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1297734.000000 [21:59:52] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [22:02:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3631.000000 [22:16:53] / on ortelius is WARNING: DISK WARNING - free space: / 6248 MB (20% inode=92%): [22:23:13] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53555 MB (8% inode=99%): [22:46:12] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:47:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:47:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15525.000000 [22:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27287 [22:49:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27235.000000 [22:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:51:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23453 [22:51:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1298699.000000 [22:59:52] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7018.000000 [23:02:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4323.000000 [23:16:52] / on ortelius is WARNING: DISK WARNING - free space: / 5712 MB (19% inode=92%): [23:23:12] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:24:52] / on ortelius is OK: DISK OK - free space: / 12063 MB (40% inode=92%): [23:24:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3452.000000 [23:30:52] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1650.000000 [23:46:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:46:02] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 53207 MB (8% inode=99%): [23:46:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:47:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:47:12] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:47:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13942.000000 [23:49:02] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27066 [23:49:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27047.000000 [23:50:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:51:22] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18614 [23:51:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:52:02] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:52] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1298249.000000 [23:59:52] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6229.000000