[00:02:07] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2429.000000 [00:03:06] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1111.000000 [00:12:47] aliasd on yarrow is CRITICAL: Connection refused [00:28:16] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:29:56] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:29:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:29:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80875 MB (13% inode=99%): [00:30:57] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:31:16] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58709 [00:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [00:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 350675 MB (6% inode=63%): [00:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 297216.000000 [00:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [00:32:58] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:32:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58610.000000 [00:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:33:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 438592.000000 [00:34:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 509059.000000 [00:34:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:34:36] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 421825.000000 [00:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 446431.000000 [00:35:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 442649.000000 [00:40:57] Free Memory on damiana is CRITICAL: CRITICAL - 1.9% (160320 kB) free! [00:48:57] Free Memory on damiana is WARNING: WARNING - 5.2% (437808 kB) free! [00:51:57] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (369880 kB) free! [00:59:57] Free Memory on damiana is WARNING: WARNING - 5.7% (474588 kB) free! [01:12:46] aliasd on yarrow is CRITICAL: Connection refused [01:28:17] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:28:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:29:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:29:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:29:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80565 MB (13% inode=99%): [01:30:57] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:31:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58286 [01:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [01:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353481 MB (6% inode=63%): [01:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 296121.000000 [01:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [01:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58191.000000 [01:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:33:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 440557.000000 [01:34:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 512333.000000 [01:34:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:34:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 419530.000000 [01:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 446363.000000 [01:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:36:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 443858.000000 [02:12:47] aliasd on yarrow is CRITICAL: Connection refused [02:19:57] Free Memory on damiana is CRITICAL: CRITICAL - 3.4% (283428 kB) free! [02:28:17] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:28:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81256 MB (13% inode=99%): [02:30:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:30:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:30:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:30:58] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:31:16] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57450 [02:31:46] / on thyme is UNKNOWN: NRPE: Unable to read output [02:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353050 MB (6% inode=63%): [02:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 296511.000000 [02:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [02:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57483.000000 [02:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:33:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 442120.000000 [02:34:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 514686.000000 [02:34:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:34:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 418753.000000 [02:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 445953.000000 [02:35:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:35:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:35:16] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 443574.000000 [02:41:45] [[Special:Log/newusers]] create 10 * Naruto123 * (New user account) [02:45:42] [[User talk:Naruto123]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7904&rcid=21754 * Naruto123 * (+317) (Created page with "Kubarz Beverage Catering locates in Sydney, Australia. We are one of the biggest mobile beverages catering company in Sydney and New South Wales. Come to us to enjoy your parties...") [02:47:08] Susan: ^ [02:48:58] closedmouth: v [02:49:11] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Naruto12310]]": spam) [02:49:16] eat it [02:49:25] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Naruto12310]] with an expiry time of infinite (account creation disabled): spam) [02:49:34] closedmouth: Make me. [02:49:47] done [03:12:47] aliasd on yarrow is CRITICAL: Connection refused [03:19:57] Free Memory on damiana is CRITICAL: CRITICAL - 2.3% (193516 kB) free! [03:28:17] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:28:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:30:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80101 MB (13% inode=99%): [03:30:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:30:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:30:58] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:30:58] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:31:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57267 [03:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [03:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 352908 MB (6% inode=63%): [03:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 297195.000000 [03:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [03:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57283.000000 [03:33:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:33:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 443918.000000 [03:34:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 517084.000000 [03:34:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:34:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 417262.000000 [03:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 445972.000000 [03:35:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 445001.000000 [03:43:57] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 32016 MB (6% inode=99%): [03:44:57] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 30787 MB (5% inode=99%): [04:12:48] aliasd on yarrow is CRITICAL: Connection refused [04:19:57] Free Memory on damiana is CRITICAL: CRITICAL - 2.2% (182048 kB) free! [04:28:17] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:28:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:29:57] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:30:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79877 MB (13% inode=99%): [04:30:57] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:30:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:30:59] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:30:59] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:31:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:31:17] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:31:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58274 [04:31:47] / on thyme is UNKNOWN: NRPE: Unable to read output [04:31:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:31:57] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 352150 MB (6% inode=63%): [04:32:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 298281.000000 [04:32:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:32:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [04:32:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:32:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:33:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58352.000000 [04:33:08] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:33:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 445699.000000 [04:34:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 518358.000000 [04:34:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:34:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:34:37] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 415808.000000 [04:34:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 447126.000000 [04:35:16] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:35:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:35:17] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:36:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:38:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 446437.000000 [06:20:28] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 303509.000000 [06:20:28] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79721 MB (13% inode=99%): [06:20:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:20:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:20:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [06:20:29] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58187 [06:20:29] / on thyme is UNKNOWN: NRPE: Unable to read output [06:20:35] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [06:20:35] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:44] Free Memory on damiana is OK: OK - 91.8% (7694100 kB) free. [06:25:33] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:25:43] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [06:25:43] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [06:25:43] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [06:25:44] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2739 [06:26:04] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4998.000000 [06:26:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4999.000000 [06:26:14] Free Memory on turnera is UNKNOWN: You cant report on Solaris caches without Solaris::Kstat available! [06:26:14] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5002.000000 [06:26:44] MySQL slave on z-dat-s3-a is OK: Uptime: 130099 Threads: 25 Questions: 153946612 Slow queries: 9676 Opens: 1358443 Flush tables: 1 Open tables: 16384 Queries per second avg: 1183.303 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1388 [06:30:43] MySQL slave on z-dat-s7-a is OK: Uptime: 130344 Threads: 9 Questions: 90178904 Slow queries: 5285 Opens: 195523 Flush tables: 1 Open tables: 3353 Queries per second avg: 691.853 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1631 [06:52:44] SSH on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:53:04] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [06:59:14] Free Memory on turnera is OK: OK - 65.8% (5516116 kB) free. [07:07:23] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:43] Sun Grid Engine execd on mayapple is UNKNOWN: Execution timeout exceeded [07:17:34] SSH on yarrow is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [07:17:53] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [07:19:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:19:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:19:44] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:19:44] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:19:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 454603.000000 [07:19:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:19:53] aliasd on yarrow is CRITICAL: Connection refused [07:19:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:19:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:19:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:20:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:20:03] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58149.000000 [07:20:04] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 454544.000000 [07:20:13] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 455350.000000 [07:20:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 423985.000000 [07:20:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 526958.000000 [07:20:14] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:20:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 307109.000000 [07:20:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:20:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79446 MB (13% inode=99%): [07:20:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:20:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [07:20:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58174 [07:20:24] / on thyme is UNKNOWN: NRPE: Unable to read output [07:20:34] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [07:20:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:20:34] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:34] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351811 MB (6% inode=63%): [07:20:34] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:25:33] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [07:25:44] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [07:25:44] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:26:04] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8598.000000 [07:26:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8599.000000 [07:26:14] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8602.000000 [07:52:24] Sun Grid Engine execd on mayapple is OK: Host and Queues Ok [07:53:13] SSH on mayapple is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [08:17:04] Marlen Caemmerer * Re: [Toolserver-l] srwiki_p database incosistency [08:19:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:19:33] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:19:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:19:43] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:19:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 458203.000000 [08:19:56] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:19:56] aliasd on yarrow is CRITICAL: Connection refused [08:19:56] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:19:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:19:56] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:20:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:20:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56214.000000 [08:20:04] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:14] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 458953.000000 [08:20:14] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 427588.000000 [08:20:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 458148.000000 [08:20:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 530557.000000 [08:20:14] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:20:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 310710.000000 [08:20:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79238 MB (13% inode=99%): [08:20:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:20:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:20:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:20:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56241 [08:20:25] / on thyme is UNKNOWN: NRPE: Unable to read output [08:20:34] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [08:20:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:20:34] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:34] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351495 MB (6% inode=63%): [08:20:34] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:24:04] K. Peachey * Re: [Toolserver-l] srwiki_p database incosistency [08:24:34] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [08:25:44] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9877 [08:25:44] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [08:25:44] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10665 [08:26:04] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12199.000000 [08:26:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12200.000000 [08:26:13] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9875.000000 [08:52:44] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3555 [08:53:13] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3338.000000 [08:56:44] MySQL slave on z-dat-s5-b is OK: Uptime: 2352 Threads: 2 Questions: 1437232 Slow queries: 21 Opens: 252 Flush tables: 1 Open tables: 195 Queries per second avg: 611.68 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1795 [08:57:14] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1477.000000 [09:10:44] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3592 [09:19:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:19:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:19:45] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:19:45] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:19:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 461805.000000 [09:20:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:20:04] aliasd on yarrow is CRITICAL: Connection refused [09:20:04] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:20:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:20:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54669.000000 [09:20:05] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:20:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:14] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 462554.000000 [09:20:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 461748.000000 [09:20:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 431189.000000 [09:20:15] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 534157.000000 [09:20:16] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:20:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 314310.000000 [09:20:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79004 MB (12% inode=99%): [09:20:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:20:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:20:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:20:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54628 [09:20:34] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:20:34] / on thyme is UNKNOWN: NRPE: Unable to read output [09:20:34] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:20:34] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351117 MB (6% inode=63%): [09:20:34] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [09:20:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:24:33] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [09:25:43] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [09:26:04] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15799.000000 [09:26:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15800.000000 [09:30:50] MySQL slave on z-dat-s2-b is OK: Uptime: 4338 Threads: 14 Questions: 2814146 Slow queries: 52 Opens: 45321 Flush tables: 1 Open tables: 256 Queries per second avg: 648.719 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1680 [10:06:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 24106 MB (2% inode=99%): [10:17:11] @replag [10:17:13] Alchimista: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 14h 30m 13s [-]; s1-user-wd: 6d 5h 19m 41s [-]; s2-user-c: error; s2-user-wd: 5d 9h 12m 52s [-]; s3-user: 2m 4s [-]; s3-user-wd: 5d 54m 56s [-] [10:17:14] Alchimista: s4-user-wd: 5d 9h 14m 9s [-]; s5-rr-a-wd: 3d 16h 15m 28s [-]; s5-user: 40s [-]; s5-user-c: 5h 14m 32s [-]; s6-user-wd: 5d 43m 33s [-]; s7-user: 10s [-]; s7-user-wd: 5d 9h 26m 18s [-] [10:19:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:19:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:19:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 465404.000000 [10:19:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:20:04] aliasd on yarrow is CRITICAL: Connection refused [10:20:04] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:20:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51967.000000 [10:20:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:20:05] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:20:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:13] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 465348.000000 [10:20:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 537757.000000 [10:20:14] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:20:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 317910.000000 [10:20:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:20:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78755 MB (12% inode=99%): [10:20:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:20:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51924 [10:20:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:20:34] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:20:34] / on thyme is UNKNOWN: NRPE: Unable to read output [10:20:34] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:20:34] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 350652 MB (6% inode=63%): [10:20:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:20:35] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:20:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 434814.000000 [10:20:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 466179.000000 [10:20:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:20:44] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [10:24:34] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:25:43] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:26:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19399.000000 [10:26:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19399.000000 [11:05:04] Marlen Caemmerer * Re: [Toolserver-l] srwiki_p database incosistency [11:06:22] @replag [11:06:22] Betacommand: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 14h 1m 52s [-0.58 s/s]; s1-user-wd: 6d 6h 8m 50s [+1.00 s/s]; s2-rr: 21s [-]; s2-user: 22s [-]; s2-user-c: error; s2-user-wd: 5d 10h 2m 2s [+1.00 s/s] [11:06:23] Betacommand: s3-user: 17s [-0.04 s/s]; s3-user-wd: 5d 1h 44m 6s [+1.00 s/s]; s4-user-wd: 5d 10h 3m 19s [+1.00 s/s]; s5-rr-a-wd: 3d 17h 4m 38s [+1.00 s/s]; s5-user-c: 6h 3m 42s [+1.00 s/s]; s6-user-wd: 5d 1h 32m 43s [+1.00 s/s]; s7-user-wd: 5d 10h 15m 28s [+1.00 s/s] [11:19:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:19:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:19:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:19:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 469005.000000 [11:20:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:20:04] aliasd on yarrow is CRITICAL: Connection refused [11:20:04] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:20:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:20:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50259.000000 [11:20:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:20:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 468948.000000 [11:20:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 541358.000000 [11:20:15] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:20:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 321509.000000 [11:20:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:20:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78564 MB (12% inode=99%): [11:20:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:20:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50276 [11:20:34] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:20:34] / on thyme is UNKNOWN: NRPE: Unable to read output [11:20:34] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:20:34] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 350349 MB (6% inode=63%): [11:20:34] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:20:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:20:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 438414.000000 [11:20:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 469779.000000 [11:20:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:20:44] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [11:21:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [11:24:33] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:25:44] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:26:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22999.000000 [11:26:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23000.000000 [11:35:22] hello all [12:01:04] Platonides * Re: [Toolserver-l] srwiki_p database incosistency [12:19:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:19:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:19:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:19:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 472604.000000 [12:20:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:20:06] aliasd on yarrow is CRITICAL: Connection refused [12:20:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:20:06] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:20:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50388.000000 [12:20:06] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:20:14] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:14] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 472548.000000 [12:20:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 544958.000000 [12:20:15] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:20:25] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 325110.000000 [12:20:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:20:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78344 MB (12% inode=99%): [12:20:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:20:25] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50395 [12:20:34] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:20:34] / on thyme is UNKNOWN: NRPE: Unable to read output [12:20:34] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:20:34] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 350031 MB (6% inode=63%): [12:20:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:20:35] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:20:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 442015.000000 [12:20:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 473381.000000 [12:20:43] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:20:44] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [12:21:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:24:33] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:25:43] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:26:04] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26599.000000 [12:26:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26599.000000 [13:10:04] DaB. * [Toolserver-l] Restart of z-dat-s1-b (sql-s1-rr), z-dat-s2-b (sql-s2) and z-dat-s5-b (sql-5-user) TODAY [13:19:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:19:34] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:19:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 476204.000000 [13:19:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:20:04] aliasd on yarrow is CRITICAL: Connection refused [13:20:04] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:20:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:20:04] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51113.000000 [13:20:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:20:19] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:19] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:19] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 476148.000000 [13:20:19] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:19] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 548559.000000 [13:20:19] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:20:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 328710.000000 [13:20:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:20:24] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78086 MB (12% inode=99%): [13:20:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:20:24] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51114 [13:20:33] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:20:33] / on thyme is UNKNOWN: NRPE: Unable to read output [13:20:33] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:20:33] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349614 MB (6% inode=63%): [13:20:34] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:20:34] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:20:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 476981.000000 [13:20:43] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 445615.000000 [13:20:44] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:20:44] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [13:21:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:23:17] @rep;ag [13:23:20] @replag [13:23:21] Betacommand: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 14h 11m 47s [+0.07 s/s]; s1-user-wd: 6d 8h 25m 49s [+1.00 s/s]; s2-user-c: error; s2-user-wd: 5d 12h 19m 0s [+1.00 s/s]; s3-user: 1m 48s [+0.01 s/s]; s3-user-wd: 5d 4h 1m 4s [+1.00 s/s] [13:23:21] Betacommand: s4-user-wd: 5d 12h 20m 17s [+1.00 s/s]; s5-rr-a-wd: 3d 19h 21m 36s [+1.00 s/s]; s5-user-c: 8h 20m 40s [+1.00 s/s]; s6-user-wd: 5d 3h 49m 41s [+1.00 s/s]; s7-user-wd: 5d 12h 32m 26s [+1.00 s/s] [13:23:40] DaBPunkt: know why we are getting an error for s1-rr ?? [13:24:34] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [13:25:47] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [13:26:03] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30199.000000 [13:26:04] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30199.000000 [13:28:14] Free Memory on turnera is WARNING: WARNING - 5.6% (472288 kB) free! [13:33:13] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (404232 kB) free! [13:34:14] Free Memory on turnera is WARNING: WARNING - 5.2% (433728 kB) free! [13:35:02] Betacommand: does recovery at the moment [13:55:15] @replag [13:55:15] Betacommand: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 14h 22m 24s [-]; s1-user-wd: 6d 8h 57m 44s [-]; s2-rr: 18s [-]; s2-user: 18s [-]; s2-user-c: error; s2-user-wd: 5d 12h 50m 55s [-] [13:55:17] Betacommand: s3-user: 38s [-]; s3-user-wd: 5d 4h 32m 59s [-]; s4-user-wd: 5d 12h 52m 12s [-]; s5-rr-a-wd: 3d 19h 53m 31s [-]; s5-user-c: 8h 52m 35s [-]; s6-user-wd: 5d 4h 21m 36s [-]; s7-user-wd: 5d 13h 4m 21s [-] [14:19:58] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 479805.000000 [14:20:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:20:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52430.000000 [14:20:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:20:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:20:09] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:20:18] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52434 [14:20:37] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:20:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77823 MB (12% inode=99%): [14:20:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:20:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 479780.000000 [14:20:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 332340.000000 [14:20:47] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349271 MB (6% inode=63%): [14:20:58] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 480594.000000 [14:20:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:20:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:20:58] aliasd on yarrow is CRITICAL: Connection refused [14:20:59] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 449235.000000 [14:20:59] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:20:59] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:21:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 552208.000000 [14:21:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:08] / on thyme is UNKNOWN: NRPE: Unable to read output [14:21:08] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:18] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:21:28] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:21:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [14:21:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:24:38] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [14:25:48] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [14:26:07] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33799.000000 [14:26:07] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33800.000000 [14:50:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:19:58] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:19:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 483406.000000 [15:20:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53343.000000 [15:20:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:20:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:20:08] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:20:18] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:28] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 53361 [15:20:38] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:20:48] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77599 MB (12% inode=99%): [15:20:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:20:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 483380.000000 [15:20:48] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 335940.000000 [15:20:48] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348999 MB (6% inode=63%): [15:20:58] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 484195.000000 [15:20:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:20:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:20:58] aliasd on yarrow is CRITICAL: Connection refused [15:20:58] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 452835.000000 [15:20:59] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:20:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:21:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:21:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555807.000000 [15:21:08] / on thyme is UNKNOWN: NRPE: Unable to read output [15:21:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:08] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:09] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:17] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:21:27] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:21:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [15:21:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:24:37] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [15:25:48] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [15:26:08] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37399.000000 [15:26:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37399.000000 [15:50:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:19:58] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:19:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 487006.000000 [16:20:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54219.000000 [16:20:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:20:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:20:08] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:20:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:20:31] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54222 [16:20:38] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:20:48] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77337 MB (12% inode=99%): [16:20:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:20:48] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 486980.000000 [16:20:48] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 339540.000000 [16:20:48] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348635 MB (6% inode=63%): [16:20:58] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 487794.000000 [16:20:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:20:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:20:58] aliasd on yarrow is CRITICAL: Connection refused [16:20:58] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 456436.000000 [16:20:59] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:20:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:21:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:21:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 559407.000000 [16:21:08] / on thyme is UNKNOWN: NRPE: Unable to read output [16:21:18] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:21:28] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:21:37] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [16:21:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:22:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:08] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:18] /sql on z-dat-s7-a is CRITICAL: DISK CRITICAL - free space: /sql 30590 MB (5% inode=99%): [16:24:38] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [16:25:48] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [16:26:08] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40999.000000 [16:26:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40999.000000 [16:42:37] MySQL on z-dat-s1-b is OK: Uptime: 19834 Threads: 6 Questions: 2174 Slow queries: 0 Opens: 72 Flush tables: 1 Open tables: 65 Queries per second avg: 0.109 [16:50:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:19:58] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 490606.000000 [17:20:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53877.000000 [17:20:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:20:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:20:08] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:20:17] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:27] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 53851 [17:20:38] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:20:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77004 MB (12% inode=99%): [17:20:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:20:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 490580.000000 [17:20:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 343140.000000 [17:20:48] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348289 MB (6% inode=63%): [17:20:57] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 491394.000000 [17:20:57] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:20:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:20:58] aliasd on yarrow is CRITICAL: Connection refused [17:20:58] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 460035.000000 [17:20:59] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:20:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:21:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:21:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 563008.000000 [17:21:08] / on thyme is UNKNOWN: NRPE: Unable to read output [17:21:18] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:21:28] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:21:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [17:21:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:22:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:08] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:48] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31531 [17:26:08] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44599.000000 [17:26:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44600.000000 [17:50:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:01:29] hello [18:01:36] maintenance tonight again [18:01:42] i am going to prepare [18:07:35] nosy1: good luck :-) [18:09:02] valhallasw: thanks [18:19:57] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 494205.000000 [18:20:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45607.000000 [18:20:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:20:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:20:09] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:20:18] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:20:28] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45634 [18:20:38] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:20:51] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76731 MB (12% inode=99%): [18:20:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:20:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 494180.000000 [18:20:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 346739.000000 [18:20:51] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 347844 MB (6% inode=63%): [18:20:58] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 494994.000000 [18:20:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:20:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:20:58] aliasd on yarrow is CRITICAL: Connection refused [18:20:58] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 463636.000000 [18:20:59] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:20:59] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:21:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:21:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 566608.000000 [18:21:08] / on thyme is UNKNOWN: NRPE: Unable to read output [18:21:17] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:21:27] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:21:37] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [18:21:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:22:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:08] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:08] SMTP on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:48] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17291 [18:26:08] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48199.000000 [18:26:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48199.000000 [18:29:48] Authoritative DNS on ha-dns-auth is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:47:40] going to reboot damiana and turnera [19:20:24] Should I not be able to access nightshade/willow right now? [19:23:37] Looks like toolserver is down again [19:23:41] legoktm: nosy1 is rebooting damiana and turnera, so everything is affected, I think [19:23:50] ok [19:26:31] Shouldn't that be announced or did something else go crazy that required the restart [19:26:38] Oh, I see it was mentioned in a reply to an announcement [19:28:38] DNS for toolserver.org fails? [19:29:17] yeah [19:29:40] anybody remembers IP? [19:33:05] i remember ips [19:33:18] its not working anyway though ;) [19:33:39] i am currently bringing everything back as good as possible [19:33:43] :( [19:33:53] after that www will not work i will have to renumber that too [19:41:58] dns itself is back [19:42:10] querying the dns host directly works [19:46:04] Magnus Manske * [Toolserver-l] Toolserver down? [20:06:40] NFS server ha-nfs.esi not responding still trying [20:17:05] Krinkle * Re: [Toolserver-l] Toolserver down? [20:32:10] toolserver is now the history? [20:36:02] ok seems willow is accessible again [20:36:21] can anyone confirm please? [20:38:12] marc@willow:~$ [20:38:28] nosy1: ^^ [20:38:45] Coren: good :) [20:39:48] had a little hard time with dns [20:40:04] DaBPunkt found it [20:40:17] all screens lost :-/ [20:40:33] Danny_B: I did not reboot willow though [20:40:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:40:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355126.000000 [20:40:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:40:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76625 MB (12% inode=99%): [20:40:41] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:40:41] what about screen program itself? [20:40:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [20:40:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14326 [20:40:42] / on thyme is UNKNOWN: NRPE: Unable to read output [20:40:50] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:40:50] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:40:52] mooharharhar [20:41:32] screen keeps running, but it wrote "resource lost" [20:41:42] and whatever was running in screen no longer runs [20:44:16] Danny_B: oh ok [20:44:26] well it regarded dns, nfs, ldap... [20:45:41] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 502866.000000 [20:45:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:45:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 575278.000000 [20:45:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:45:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:45:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:45:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13890 [20:45:46] / on thyme is UNKNOWN: NRPE: Unable to read output [20:45:57] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [20:55:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:55:56] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 575894.000000 [20:55:56] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:56:00] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:56:00] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:56:11] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). warnings detected. run with -v for information. [21:01:13] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:01:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 576212.000000 [21:01:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:01:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:01:19] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 356365.000000 [21:01:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76556 MB (12% inode=99%): [21:01:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:01:30] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:01:30] SMTP on thyme is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:14] everything is back apart from web service [21:02:24] going to prepare to reboot ortelius and wolfsbane [21:05:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:10] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:00] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:06:01] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:10] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:30] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:06:30] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:06:39] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:06:40] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:06:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:40] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:40] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:40] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:06:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:06:50] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:50] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:06:50] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:00] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [21:07:00] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:01] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:01] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:01] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:08] Load avg. on yarrow is WARNING: WARNING - load average: 1.01, 1.26, 19.27 [21:07:09] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:11] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:11] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:11] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:19] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:07:20] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:20] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:30] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:10:29] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [21:12:09] Load avg. on yarrow is OK: OK - load average: 1.26, 1.35, 14.33 [21:19:50] SMTP on z-dat-s2-b is OK: SMTP OK - 0.111 sec. response time [21:25:30] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:26:10] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9045 [21:26:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 505300.000000 [21:32:49] SMTP on z-dat-s5-b is OK: SMTP OK - 0.149 sec. response time [21:37:42] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8446 [21:37:42] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [21:38:49] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59760.000000 [21:38:59] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7985.000000 [21:40:09] SSH on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:06] ok web service is back again [21:48:08] it seems [21:48:10] please test [21:50:00] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3517.000000 [21:50:00] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3510 [21:50:34] nosy1, sulinfo is working for me, thanks [21:51:21] 404 not found at http://toolserver.org/~acc/acc.php [21:51:34] Error connecting to database. [21:51:38] it says for me [21:52:59] SSH on z-dat-s1-b is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [21:53:45] hmm... [21:54:19] getting lots of "found" from both hosts [21:54:19] Load avg. on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:54:24] do you still have this issue? [21:54:29] / on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:55:56] I cleared my local browser cache, and the DB error is: Access denied for user 'acc'@'185.15.59.211' (using password: YES) (p_acc_live on sql-user-a): [21:56:18] Cannot run without connection to 'sql': Access denied for user 'vvv'@'185.15.59.211' (using password: YES) [21:57:19] SMTP on thyme is OK: SMTP OK - 0.216 sec. response time [22:00:00] retsreklawts: i hope the dns stuff there expires soone [22:00:49] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 507458.000000 [22:00:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:51] fair enough [22:00:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:00:59] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:01:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 579812.000000 [22:01:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:01:19] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 359966.000000 [22:01:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 76210 MB (12% inode=99%): [22:01:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:01:29] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 347621 MB (6% inode=63%): [22:01:49] aliasd on yarrow is CRITICAL: Connection refused [22:01:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61153.000000 [22:02:08] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3225 [22:03:09] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:03:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:03:28] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:03:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:03:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:03:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:03:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:03:58] / on thyme is UNKNOWN: NRPE: Unable to read output [22:04:17] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10767 [22:04:38] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3503 [22:05:00] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3450.000000 [22:05:59] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1735.000000 [22:06:00] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:06:00] MySQL slave on rosemary is OK: Uptime: 1406497 Threads: 14 Questions: 1238413821 Slow queries: 430172 Opens: 65878 Flush tables: 1 Open tables: 3440 Queries per second avg: 880.495 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1730 [22:06:09] MySQL slave on z-dat-s2-b is OK: Uptime: 2791 Threads: 6 Questions: 3442956 Slow queries: 12 Opens: 39631 Flush tables: 1 Open tables: 256 Queries per second avg: 1233.592 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1395 [22:06:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:10] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:10] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:10] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:52] who removed ~acc/acc.php? [22:06:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:07:27] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:07:28] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:07:39] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:07:39] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:07:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 25251 MB (2% inode=99%): [22:07:39] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:39] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:40] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:07:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:07:50] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:50] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:07:50] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:59] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [22:07:59] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:59] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:59] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:08:08] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:08] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:08] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:19] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:08:19] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:19] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:30] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:10:38] MySQL slave on z-dat-s5-b is OK: Uptime: 2320 Threads: 3 Questions: 1079184 Slow queries: 11 Opens: 294 Flush tables: 1 Open tables: 188 Queries per second avg: 465.165 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1750 [22:10:58] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1662.000000 [22:10:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:11:13] DaBPunkt repaired the sql error :) [22:11:31] probel: accc returns now anothe error… [22:11:53] "DB access control error" is not very helpful… [22:12:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:13:01] retsreklawts: ^^ [22:13:37] DaBPunkt, I hid the error so it showed a nice "sorry, down at the moment" message for newbies [22:13:59] seems to work now anyway [22:14:04] retsreklawts: and waht is the real error? [22:14:07] ah, ok [22:14:18] the real error was: Access denied for user 'acc'@'185.15.59.211' (using password: YES) (p_acc_live on sql-user-a) [22:14:43] yes, that was the first one [22:15:12] I'm still getting the occasional 404 though [22:15:46] appears to be somewhat random though [22:17:04] nosy1: ^^ [22:17:23] i look [22:17:27] log file [22:18:21] nosy1: /var/log/http/ [22:18:44] nginx [22:19:27] nosy1: /var/log/nginx/ [22:19:33] yes [22:20:43] upstream timed out (145: Connection timed out) for wolfsbane [22:24:46] sometimes [22:26:29] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:26:42] looks more like a problem at wolfsbane [22:26:45] for me [22:27:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 508881.000000 [22:31:39] DaBPunkt: seems to happen on ortelius too afais [22:32:47] The osm tiles are also failing atm [22:34:20] apmon: all of them? [22:34:39] I think so [22:34:45] do you have an url? [22:36:19] Ah, seems to work now again [22:36:54] It looks like it might be an intermittend issue though [22:37:15] sorry i have to admit i would like to sleep [22:37:24] ill look tomorrow what is left here [22:38:15] good night then [22:38:30] good night [22:38:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [22:39:48] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55338.000000 [22:42:20] Welp, TS just broke... [22:42:32] At least, my tools did [22:47:03] Working now [22:51:54] Broke again... Is this the IP thing [22:51:56] ? [22:55:55] Matthew_: URL? [22:57:03] DaBPunkt: http://toolserver.org/~matthewrbowker/ is 404ing, but not the customized 404 message. [22:57:36] Now it works... [22:58:03] return a blank page here [22:58:51] Huh, wierd [22:59:02] 404ed for me [23:00:01] Can't get to an MMT either [23:00:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:00:59] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:01:49] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 507959.000000 [23:01:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:01:49] aliasd on yarrow is CRITICAL: Connection refused [23:01:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40385.000000 [23:02:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:02:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 579442.000000 [23:02:18] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 75982 MB (12% inode=99%): [23:02:18] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 361520.000000 [23:02:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:02:29] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 347514 MB (6% inode=63%): [23:03:10] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:03:30] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:03:30] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:03:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:03:39] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:03:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:03:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:03:59] / on thyme is UNKNOWN: NRPE: Unable to read output [23:04:19] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5481 [23:06:59] Sensors on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:06:59] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:09] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:09] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:10] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:10] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:29] / on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:07:29] Sun Grid Engine execd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:07:38] APT on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:07:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:07:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:49] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:58] Load avg. on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:07:58] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:59] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:59] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:18] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:08:18] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:19] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:39] /tmp on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:08:39] aliasd on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:08:39] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:39] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:49] Environment IPMI on nightshade is UNKNOWN: CHECK_NRPE: Error receiving data from daemon. [23:08:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:09:29] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:09:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:09:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:10:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:13:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:14:51] DaBPunkt: Broke again [23:16:40] Matthew_: are you sure that your page worked before the maintenance? [23:17:15] DaBPunkt: Yes, I'm sure. It just keeps intermittently disappearing. After all, it's raw HTML with a couple of PHP methods that print more raw HTML [23:27:09] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 510522.000000 [23:27:28] APT on yucca is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:28:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:31:15] Matthew_: and you changed nothing? [23:31:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:31:24] DaBPunkt: Not since it was working. [23:31:45] I'm also getting intermittent 404's in the ACC multi-maintainer project. http://toolserver.org/~acc [23:31:54] The problem is, they're intermittent :/ [23:38:06] I think those intermittent issues might be general or with the load balancers [23:38:38] APT on sage is WARNING: APT WARNING: 1 packages available for upgrade (0 critical updates). [23:39:55] http://toolserver.org/~apmon/OSM_healthoutput.txt logs the health status of a number of OSM related pages, including the Toolserver tileserver [23:40:26] and one can see that every couple of requests it returns a 404 for the same URL on the toolserver [23:40:48] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26082.000000 [23:40:50] e.g. like currently [23:43:42] Matthew_: /home/matthewrbowker/public_html/index.php also returns a 500 if I run it on the shell [23:44:54] Huh, that's wierd. [23:47:52] DaBPunkt: I have to leave now. You are welcome to continue using my tools for testing, but other people are reporting errors on catscan, acc, and OSM (see above) [23:48:19] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3581 [23:49:44] @replag [23:49:48] DaBPunkt: s1-rr-a: 58m 23s [-]; s1-rr-a-wd: 5d 2h 5m 54s [-]; s1-user-wd: 6d 16h 38m 23s [-]; s2-rr: 2m 34s [-]; s2-user: 2m 37s [-]; s2-user-c: error; s2-user-wd: 5d 21h 51m 23s [-]; s3-user: 21s [-] [23:49:49] DaBPunkt: s3-user-wd: 5d 12h 37m 12s [-]; s4-user-wd: 5d 20h 50m 23s [-]; s5-rr-a: 11s [-]; s5-rr-a-wd: 4d 4h 26m 58s [-]; s5-user: 11s [-]; s5-user-c: 6h 4m 17s [-]; s6-user-wd: 5d 12h 32m 50s [-]; s7-user-wd: 5d 21h 2m 6s [-] [23:50:51] DaBPunkt: one doubt. ptwiki_p is in s2, which lag referes to it? s2-rr? [23:51:29] yes [23:51:42] ok, thanks