[00:01:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:09:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:10:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 432435 MB (8% inode=64%): [00:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:13:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:13:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:14:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15176.000000 [00:15:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100448.000000 [00:15:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14327.000000 [00:16:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113646 [00:19:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 167921.000000 [00:22:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113606.000000 [00:23:43] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [00:23:43] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [00:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:24:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [00:25:25] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [00:25:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:26:34] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:26:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:27:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:27:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:28:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:28:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:28:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:28:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:30:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:30:43] / on thyme is UNKNOWN: NRPE: Unable to read output [00:31:22] aliasd on yarrow is CRITICAL: Connection refused [00:33:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7223.000000 [00:43:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88267 MB (14% inode=99%): [00:50:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:01:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:09:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:09:46] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31494 MB (3% inode=99%): [01:11:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 435001 MB (8% inode=64%): [01:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:13:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:13:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:55] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15320.000000 [01:15:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14617.000000 [01:15:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102466.000000 [01:16:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113284 [01:19:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 169709.000000 [01:22:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113323.000000 [01:23:44] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [01:23:44] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [01:24:12] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [01:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [01:25:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:26:23] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [01:26:34] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [01:27:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:27:44] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:28:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:28:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:28:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:29:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:30:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:30:45] / on thyme is UNKNOWN: NRPE: Unable to read output [01:31:24] aliasd on yarrow is CRITICAL: Connection refused [01:34:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7780.000000 [01:43:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 88139 MB (14% inode=99%): [01:51:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:10:43] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:10:43] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:12:12] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 434808 MB (8% inode=64%): [02:14:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:14:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:15:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15451.000000 [02:16:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14830.000000 [02:16:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102737.000000 [02:17:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 112707 [02:20:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 169532.000000 [02:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112679.000000 [02:24:12] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [02:24:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [02:24:45] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [02:24:45] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [02:26:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [02:26:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:27:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [02:27:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:28:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:28:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:28:06] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:28:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:29:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:30:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:30:45] / on thyme is UNKNOWN: NRPE: Unable to read output [02:31:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:32:23] aliasd on yarrow is CRITICAL: Connection refused [02:34:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7984.000000 [02:43:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87970 MB (14% inode=99%): [02:52:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:10:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:46] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:12:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 434225 MB (8% inode=64%): [03:14:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:14:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15928.000000 [03:16:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100762.000000 [03:16:55] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15280.000000 [03:17:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113284 [03:20:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 169818.000000 [03:23:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113398.000000 [03:24:12] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [03:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [03:25:45] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [03:25:45] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [03:26:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:27:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [03:27:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [03:27:44] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:27:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:28:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:29:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:29:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:29:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:30:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:30:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:30:44] / on thyme is UNKNOWN: NRPE: Unable to read output [03:32:23] aliasd on yarrow is CRITICAL: Connection refused [03:34:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8126.000000 [03:43:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87751 MB (14% inode=99%): [03:53:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:10:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:10:45] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:13:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 432189 MB (8% inode=64%): [04:15:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:15:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16413.000000 [04:16:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15739.000000 [04:16:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103191.000000 [04:18:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 114524 [04:21:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 172212.000000 [04:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [04:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [04:24:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114890.000000 [04:25:44] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [04:25:44] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [04:26:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:27:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [04:27:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [04:27:44] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:28:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:29:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:29:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:29:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:29:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:30:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:31:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:31:45] / on thyme is UNKNOWN: NRPE: Unable to read output [04:33:27] aliasd on yarrow is CRITICAL: Connection refused [04:34:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8233.000000 [04:44:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87565 MB (14% inode=99%): [04:53:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:10:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:10:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:14:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 432110 MB (8% inode=64%): [05:15:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:15:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:16:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16447.000000 [05:17:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15820.000000 [05:17:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101602.000000 [05:19:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 116210 [05:22:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 174695.000000 [05:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [05:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [05:25:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116377.000000 [05:26:43] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [05:26:43] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [05:27:34] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [05:27:44] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:27:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [05:28:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [05:28:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:29:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:29:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:29:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:29:26] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:30:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:31:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:31:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:31:45] / on thyme is UNKNOWN: NRPE: Unable to read output [05:34:25] aliasd on yarrow is CRITICAL: Connection refused [05:35:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7988.000000 [05:45:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87461 MB (14% inode=99%): [05:53:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:10:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:10:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:11:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:12:45] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 40425 MB (7% inode=99%): [06:14:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 431921 MB (8% inode=64%): [06:15:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:16:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:16:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16618.000000 [06:17:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104070.000000 [06:17:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16251.000000 [06:19:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 118241 [06:22:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 176863.000000 [06:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [06:25:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118232.000000 [06:26:43] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [06:26:43] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [06:27:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:27:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:28:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [06:28:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:28:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:29:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:29:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:29:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:29:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:30:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:30:23] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2071 [06:31:23] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:31:23] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:31:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:31:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:31:45] / on thyme is UNKNOWN: NRPE: Unable to read output [06:31:53] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:35:24] aliasd on yarrow is CRITICAL: Connection refused [06:36:43] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9619.000000 [06:38:23] MySQL slave on z-dat-s2-b is OK: Uptime: 458115 Threads: 14 Questions: 721820080 Slow queries: 7190 Opens: 4124772 Flush tables: 1 Open tables: 256 Queries per second avg: 1575.630 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1793 [06:46:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87300 MB (14% inode=99%): [06:54:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:11:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:11:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:11:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:14:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 431675 MB (8% inode=64%): [07:15:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:17:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:17:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105132.000000 [07:17:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16487.000000 [07:17:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16680.000000 [07:19:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119869 [07:22:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 177776.000000 [07:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [07:25:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119802.000000 [07:26:44] MySQL on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [07:26:45] MySQL slave on daphne is CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [07:27:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:27:54] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:28:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:29:24] s4 replag on daphne is CRITICAL: QUERY CRITICAL: Access denied for user tsnagios7643@185.15.59.197 (using password: YES) [07:29:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:29:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:30:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:30:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:30:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:30:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:31:24] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:31:24] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:31:54] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:32:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:32:44] / on thyme is UNKNOWN: NRPE: Unable to read output [07:36:24] aliasd on yarrow is CRITICAL: Connection refused [07:36:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11585.000000 [07:44:23] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 46.000000 [07:44:23] aliasd on yarrow is OK: TCP OK - 0.007 second response time on port 984 [500 Not found.] [07:44:44] MySQL on daphne is OK: Uptime: 63 Threads: 6 Questions: 8627 Slow queries: 2 Opens: 72 Flush tables: 1 Open tables: 63 Queries per second avg: 136.936 [07:44:45] MySQL slave on daphne is OK: Uptime: 63 Threads: 6 Questions: 8628 Slow queries: 2 Opens: 72 Flush tables: 1 Open tables: 63 Queries per second avg: 136.952 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6 [07:46:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 87106 MB (14% inode=99%): [07:49:23] MySQL slave on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [07:49:33] Load avg. on nightshade is WARNING: WARNING - load average: 15.22, 15.54, 13.08 [07:50:04] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [07:50:14] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [07:50:24] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [07:50:33] Load avg. on nightshade is OK: OK - load average: 13.15, 14.90, 13.01 [07:51:25] aliasd on yarrow is CRITICAL: Connection refused [07:54:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:01:04] MySQL on z-dat-s7-a is OK: Uptime: 1056 Threads: 3 Questions: 5490 Slow queries: 1 Opens: 164 Flush tables: 1 Open tables: 148 Queries per second avg: 5.198 [08:01:24] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2169 [08:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:04:23] MySQL slave on z-dat-s7-a is OK: Uptime: 1249 Threads: 2 Questions: 109011 Slow queries: 2 Opens: 450 Flush tables: 1 Open tables: 434 Queries per second avg: 87.278 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1728 [08:05:14] MySQL on z-dat-s3-a is OK: Uptime: 1299 Threads: 1 Questions: 16 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 15 Queries per second avg: 0.12 [08:05:24] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2889 [08:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:11:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:11:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:14:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 431413 MB (8% inode=64%): [08:15:12] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:17:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:17:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104352.000000 [08:17:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19928.000000 [08:17:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17734.000000 [08:18:23] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3616 [08:19:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119207 [08:23:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 179423.000000 [08:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26075.000000 [08:25:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118972.000000 [08:28:44] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:28:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:28:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:29:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:29:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:30:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:31:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:31:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:31:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:31:24] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:31:24] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:31:54] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:32:44] / on thyme is UNKNOWN: NRPE: Unable to read output [08:32:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:34:25] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3488 [08:37:43] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13627.000000 [08:41:33] Load avg. on nightshade is WARNING: WARNING - load average: 15.40, 17.52, 15.72 [08:45:24] MySQL slave on z-dat-s3-a is OK: Uptime: 3706 Threads: 18 Questions: 2703155 Slow queries: 176 Opens: 24603 Flush tables: 1 Open tables: 16384 Queries per second avg: 729.399 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1736 [08:46:44] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86940 MB (14% inode=99%): [08:49:33] Load avg. on nightshade is OK: OK - load average: 10.66, 13.55, 14.82 [08:52:24] aliasd on yarrow is CRITICAL: Connection refused [08:55:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:01:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:11:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:11:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:12:33] Load avg. on nightshade is WARNING: WARNING - load average: 13.02, 15.08, 14.43 [09:13:32] Load avg. on nightshade is OK: OK - load average: 12.27, 14.48, 14.26 [09:14:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 449862 MB (8% inode=64%): [09:15:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:17:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:17:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103637.000000 [09:17:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23527.000000 [09:17:54] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19555.000000 [09:19:44] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119167 [09:23:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180573.000000 [09:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:24:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26434.000000 [09:25:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119284.000000 [09:29:39] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:29:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:29:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:29:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:30:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:31:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:32:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:32:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:32:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:32:23] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:32:24] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:32:53] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [09:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:37:43] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15615.000000 [09:40:04] Artem Korzhimanov * Re: [Toolserver-l] IP Renumbering of the complete Toolserver cluster [09:47:43] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86725 MB (14% inode=99%): [09:53:23] aliasd on yarrow is CRITICAL: Connection refused [09:56:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:11:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:12:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:12:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 449514 MB (8% inode=64%): [10:16:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:17:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:17:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27133.000000 [10:17:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102513.000000 [10:18:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21500.000000 [10:19:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119716 [10:23:53] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 182184.000000 [10:24:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [10:25:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26717.000000 [10:26:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119766.000000 [10:27:04] Platonides * Re: [Toolserver-l] IP Renumbering of the complete Toolserver cluster [10:29:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [10:29:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:30:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:30:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:30:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:31:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:32:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:32:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:32:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:32:23] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:32:23] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:33:03] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [10:37:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17666.000000 [10:47:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86465 MB (14% inode=99%): [10:53:24] aliasd on yarrow is CRITICAL: Connection refused [10:56:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:12:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:12:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 449007 MB (8% inode=64%): [11:17:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:17:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101004.000000 [11:17:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30733.000000 [11:18:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24006.000000 [11:19:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 116359 [11:23:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 182292.000000 [11:24:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:25:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27884.000000 [11:27:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116549.000000 [11:29:32] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:29:44] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:30:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:31:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:31:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:32:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:32:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:32:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:32:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:32:23] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2173 [11:32:23] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:32:23] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:33:03] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:33:24] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2168 [11:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [11:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:37:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19707.000000 [11:41:23] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2185 [11:46:23] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [11:47:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86230 MB (14% inode=99%): [11:51:24] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2686 [11:53:24] aliasd on yarrow is CRITICAL: Connection refused [11:57:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:58:23] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3639 [11:58:23] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3630 [12:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:12:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:47] hello all - What's with the " [12:14:49] Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) [12:14:50] SELECT domain FROM toolserver.wiki WHERE dbname="commonswiki_p" [12:14:52] errors [12:15:04] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [12:15:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 448317 MB (8% inode=64%): [12:15:16] commons transfer and a few other tools are not working... probably most tools [12:17:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:17:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:17:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99716.000000 [12:18:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [12:18:05] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27521.000000 [12:19:05] MySQL on z-dat-s7-a is OK: Uptime: 625 Threads: 2 Questions: 2166 Slow queries: 2 Opens: 189 Flush tables: 1 Open tables: 180 Queries per second avg: 3.465 [12:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 114960 [12:23:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 183343.000000 [12:25:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [12:25:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30353.000000 [12:27:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 115001.000000 [12:28:24] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3481 [12:29:23] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3597 [12:29:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [12:30:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:30:43] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:31:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:31:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:31:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:32:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:32:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:32:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:32:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:32:23] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:32:23] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:33:05] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [12:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:37:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21781.000000 [12:39:23] MySQL slave on z-dat-s7-a is OK: Uptime: 1840 Threads: 2 Questions: 924641 Slow queries: 23 Opens: 658 Flush tables: 1 Open tables: 610 Queries per second avg: 502.522 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1791 [12:40:05] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2174.000000 [12:40:05] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2180 [12:43:24] MySQL slave on z-dat-s3-a is OK: Uptime: 17987 Threads: 18 Questions: 16120371 Slow queries: 4761 Opens: 96506 Flush tables: 1 Open tables: 16384 Queries per second avg: 896.223 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1646 [12:47:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85960 MB (14% inode=99%): [12:51:54] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2126.000000 [12:53:24] aliasd on yarrow is CRITICAL: Connection refused [12:57:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:02:06] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [13:02:55] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3453.000000 [13:05:04] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3628 [13:05:24] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3633.000000 [13:07:03] s4 replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [13:07:24] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2128 [13:07:33] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2953.000000 [13:10:04] [[Special:Log/newusers]] create 10 * WikiUserFS * (New user account) [13:10:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31472 MB (3% inode=99%): [13:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:12:06] Load avg. on z-dat-s5-b is WARNING: WARNING - load average: 18.61, 16.85, 13.45 [13:13:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:15:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 448092 MB (8% inode=64%): [13:17:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102357.000000 [13:18:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:18:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:19:05] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31181.000000 [13:19:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37997.000000 [13:19:09] [[Special:Log/newusers]] create 10 * Crea * (New user account) [13:19:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 112937 [13:21:20] [[Stable server]] ! 10https://wiki.toolserver.org/w/index.php?diff=7895&oldid=5512&rcid=21739 * Crea * (+138) () [13:23:06] Load avg. on z-dat-s5-b is OK: OK - load average: 13.48, 14.94, 14.35 [13:23:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 185121.000000 [13:25:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [13:25:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33024.000000 [13:28:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112762.000000 [13:29:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [13:30:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:30:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:31:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:31:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:31:13] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4384.000000 [13:32:23] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:32:23] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:33:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:33:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:33:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:33:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [13:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:34:03] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:38:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23850.000000 [13:41:13] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3474.000000 [13:48:41] [[Stable server]] M 10https://wiki.toolserver.org/w/index.php?diff=7896&oldid=7895&rcid=21740 * Betacommand * (-138) (Reverted edits by [[Special:Contributions/Crea|Crea]] ([[User talk:Crea|talk]]) to last revision by [[User:Dschwen|Dschwen]]) [13:48:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85742 MB (14% inode=99%): [13:49:09] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:Crea10]] with an expiry time of infinite (account creation disabled): Spam) [13:54:24] aliasd on yarrow is CRITICAL: Connection refused [13:58:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:01:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:05:04] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6107 [14:05:43] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6131.000000 [14:07:23] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2077 [14:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:13:04] Lydia Pintscher * [Toolserver-l] Hi! [14:13:15] reba: hi! :) [14:13:17] you're quick! [14:13:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:39] hello all [14:14:53] hi DaBPunkt [14:14:59] DaBPunkt: got a minute for a query? [14:15:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 447939 MB (8% inode=64%): [14:15:16] Lydia_WMDE: sure [14:17:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103969.000000 [14:18:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:18:25] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41656.000000 [14:20:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34841.000000 [14:20:24] MySQL slave on z-dat-s2-b is OK: Uptime: 485836 Threads: 9 Questions: 763730721 Slow queries: 7569 Opens: 4335708 Flush tables: 1 Open tables: 256 Queries per second avg: 1571.992 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1666 [14:20:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110195 [14:23:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 187335.000000 [14:25:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35228.000000 [14:26:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [14:28:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109685.000000 [14:30:04] Marlen Caemmerer * Re: [Toolserver-l] IP Renumbering of the complete Toolserver cluster [14:30:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:30:36] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [14:30:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:31:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:31:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:32:24] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:32:25] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:33:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:33:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:33:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:33:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [14:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:34:05] APT on mayapple is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:38:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25713.000000 [14:39:04] Artem Korzhimanov * Re: [Toolserver-l] IP Renumbering of the complete Toolserver cluster [14:41:24] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3049.000000 [14:48:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85451 MB (14% inode=99%): [14:49:14] s4 replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [14:54:24] aliasd on yarrow is CRITICAL: Connection refused [14:55:26] [[Special:Log/newusers]] create 10 * Lydia Pintscher (WMDE) * (New user account) [14:55:40] i feel stalked by bots :P [14:58:01] [[System administrators]] ! 10https://wiki.toolserver.org/w/index.php?diff=7897&oldid=7760&rcid=21743 * Lydia Pintscher (WMDE) * (+67) (/* staff */ + Lydia as asked for by Silke) [14:58:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:05:04] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9190 [15:06:14] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9225.000000 [15:08:54] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3229.000000 [15:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:14:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:14:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:15:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 447609 MB (8% inode=64%): [15:17:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105287.000000 [15:18:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:18:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:22] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2112 [15:20:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45256.000000 [15:20:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38442.000000 [15:20:54] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1641.000000 [15:20:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108882 [15:23:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 188342.000000 [15:25:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38117.000000 [15:26:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [15:27:23] MySQL slave on z-dat-s2-b is OK: Uptime: 489856 Threads: 12 Questions: 769348189 Slow queries: 7641 Opens: 4373026 Flush tables: 1 Open tables: 257 Queries per second avg: 1570.559 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1730 [15:29:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108599.000000 [15:30:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:30:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [15:30:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:31:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:31:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:31:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:32:33] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:33:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:33:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:33:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:33:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:33:24] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [15:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:35:04] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:39:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27543.000000 [15:48:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85192 MB (13% inode=99%): [15:54:24] aliasd on yarrow is CRITICAL: Connection refused [15:58:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:05:04] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5770 [16:06:15] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5815.000000 [16:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:14:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 447147 MB (8% inode=64%): [16:18:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:18:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103569.000000 [16:19:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:20:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48857.000000 [16:20:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42041.000000 [16:20:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 107302 [16:25:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 190419.000000 [16:25:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40559.000000 [16:27:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [16:30:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106868.000000 [16:30:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:30:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [16:31:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:31:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:31:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:32:33] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:33:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:33:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:33:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [16:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:34:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:34:23] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:36:03] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:39:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28942.000000 [16:48:04] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3578 [16:48:13] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3558.000000 [16:48:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84925 MB (13% inode=99%): [16:55:04] MySQL slave on z-dat-s5-b is OK: Uptime: 497521 Threads: 3 Questions: 814907131 Slow queries: 5703 Opens: 409701 Flush tables: 1 Open tables: 255 Queries per second avg: 1637.935 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1659 [16:55:14] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1630.000000 [16:55:24] aliasd on yarrow is CRITICAL: Connection refused [16:58:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:15:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:15:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 446883 MB (8% inode=64%): [17:19:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102359.000000 [17:20:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:21:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52517.000000 [17:21:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45701.000000 [17:21:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 105390 [17:25:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42675.000000 [17:25:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 192296.000000 [17:27:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [17:30:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:30:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [17:31:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104923.000000 [17:31:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:31:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:31:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:33:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:33:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:33:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:33:32] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [17:33:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:34:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:34:25] APT on nightshade is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:36:05] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:39:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29999.000000 [17:48:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84687 MB (13% inode=99%): [17:55:24] aliasd on yarrow is CRITICAL: Connection refused [17:59:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:12:53] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 41989 MB (7% inode=99%): [18:15:52] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 446531 MB (8% inode=64%): [18:19:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101811.000000 [18:20:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:21:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56118.000000 [18:21:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49303.000000 [18:21:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103469 [18:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45045.000000 [18:26:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 193326.000000 [18:27:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [18:30:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:30:34] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [18:31:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103110.000000 [18:31:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:32:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:32:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [18:33:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:34:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:34:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:34:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:34:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:34:33] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:35:24] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:37:04] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:39:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30880.000000 [18:48:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84419 MB (13% inode=99%): [18:55:23] aliasd on yarrow is CRITICAL: Connection refused [18:59:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:09:23] aliasd on yarrow is OK: TCP OK - 0.009 second response time on port 984 [500 Not found.] [19:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:15:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:15:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:17:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 446195 MB (8% inode=64%): [19:19:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:19:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101989.000000 [19:20:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:21:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59717.000000 [19:21:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52902.000000 [19:21:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 101764 [19:26:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47077.000000 [19:26:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 193879.000000 [19:28:12] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [19:28:20] DaBPunkt: hi [19:31:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:31:34] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [19:31:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:32:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101148.000000 [19:32:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:32:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [19:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [19:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:33:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:34:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:34:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:34:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:34:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:35:23] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:36:23] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:37:06] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:39:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31997.000000 [19:48:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84178 MB (13% inode=99%): [19:59:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:11:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:16:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:18:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 445821 MB (8% inode=64%): [20:20:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:20:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102836.000000 [20:21:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:22:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63378.000000 [20:22:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56562.000000 [20:22:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 97688 [20:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49244.000000 [20:26:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 193660.000000 [20:28:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:31:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:31:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:31:48] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:33:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 97229.000000 [20:33:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:33:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [20:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:34:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:34:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:34:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:34:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:35:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:35:33] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:36:32] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:38:03] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:40:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33871.000000 [20:49:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83862 MB (13% inode=99%): [20:55:05] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:59:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:07:35] @replag [21:11:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:17:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:18:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 445584 MB (8% inode=64%): [21:20:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:20:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103834.000000 [21:21:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:22:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66978.000000 [21:22:05] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60162.000000 [21:22:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 95400 [21:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51826.000000 [21:27:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 193812.000000 [21:28:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [21:31:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:31:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [21:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:33:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95063.000000 [21:33:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:33:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [21:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:34:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:35:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:35:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:35:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:35:33] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:36:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:37:23] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:38:06] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:40:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35779.000000 [21:42:24] aliasd on yarrow is CRITICAL: Connection refused [21:49:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83628 MB (13% inode=99%): [21:55:04] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:00:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:11:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:17:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:17:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 445351 MB (8% inode=64%): [22:20:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:20:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104923.000000 [22:21:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:22:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 93989 [22:23:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70638.000000 [22:23:06] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63822.000000 [22:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54569.000000 [22:27:13] aliasd on nightshade is CRITICAL: Connection refused [22:28:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 194295.000000 [22:29:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:31:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:31:34] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:31:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:33:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93492.000000 [22:33:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:33:06] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [22:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:34:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:35:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:35:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:35:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:35:33] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:36:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:37:24] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:38:05] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:40:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37696.000000 [22:43:24] aliasd on yarrow is CRITICAL: Connection refused [22:50:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83330 MB (13% inode=99%): [22:55:04] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:00:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:11:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:18:52] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:18:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 444392 MB (8% inode=64%): [23:20:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:21:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:21:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106228.000000 [23:22:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 92187 [23:24:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74298.000000 [23:24:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67483.000000 [23:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57169.000000 [23:27:13] aliasd on nightshade is CRITICAL: Connection refused [23:29:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 194389.000000 [23:30:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:31:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:31:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:33:42] / on thyme is UNKNOWN: NRPE: Unable to read output [23:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:34:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91966.000000 [23:34:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:34:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:34:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:35:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:35:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:35:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:35:32] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:36:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:37:24] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:39:14] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:41:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39688.000000 [23:43:24] aliasd on yarrow is CRITICAL: Connection refused [23:50:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83104 MB (13% inode=99%): [23:55:04] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates).