[23:59:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:04:36] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 76528 MB (18% inode=99%): [00:15:56] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [00:15:56] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [00:18:16] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [00:19:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:19:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [00:27:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450426 MB (8% inode=42%): [00:45:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:45:58] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 96116 [00:46:37] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:53:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:55:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:59:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:15:58] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [01:15:58] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [01:19:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:19:47] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [01:27:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450122 MB (8% inode=42%): [01:45:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:45:57] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 98960 [01:47:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:51:56] /sql on thyme is WARNING: DISK WARNING - free space: /sql 197367 MB (20% inode=99%): [01:53:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:55:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:00:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:15:57] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [02:15:57] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [02:19:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:20:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [02:27:47] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450063 MB (8% inode=42%): [02:36:36] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:45:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:45:58] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 101707 [02:47:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:51:37] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:54:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:55:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:00:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:15:26] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:15:57] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [03:15:57] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [03:16:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:21:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [03:27:47] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 456349 MB (8% inode=42%): [03:46:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:46:58] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 104577 [03:47:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:54:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:55:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:00:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:15:57] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [04:15:57] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [04:16:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:21:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [04:23:36] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106592 MB (10% inode=98%): [04:23:47] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106587 MB (10% inode=98%): [04:25:37] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107883 MB (11% inode=98%): [04:25:47] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107870 MB (11% inode=98%): [04:28:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 450033 MB (8% inode=42%): [04:46:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:46:57] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108059 [04:47:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:55:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:55:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:00:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:06:36] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:15:57] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [05:16:57] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [05:17:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:21:16] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:21:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [05:28:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449843 MB (8% inode=42%): [05:30:16] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:31:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:45:15] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [05:46:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:46:58] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111659 [05:47:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:55:16] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:16] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:17] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:17] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:17] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:17] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:26] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:26] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:27] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:27] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:27] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:27] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:27] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:36] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:36] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:46] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:47] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:47] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:47] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:55:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:55:48] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:57] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:07] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:07] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:07] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 1783 MB (99% inode=99%): [05:56:07] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1781 MB (99% inode=99%): [05:56:07] Load avg. on z-dat-s4-a is OK: OK - load average: 0.41, 1.38, 2.07 [05:56:08] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 1785 MB (99% inode=99%): [05:56:08] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 78500 MB (19% inode=99%): [05:56:08] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 1781 MB (99% inode=99%): [05:56:08] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 108401 MB (11% inode=98%): [05:56:09] Load avg. on z-dat-s3-a is OK: OK - load average: 0.41, 1.38, 2.07 [05:56:09] Load avg. on z-dat-s7-a is OK: OK - load average: 0.41, 1.35, 2.05 [05:56:10] Load avg. on z-dat-s6-a is OK: OK - load average: 0.41, 1.35, 2.05 [05:56:26] SMF on z-dat-s7-a is OK: OK - all services online [05:56:26] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:56:26] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:56:36] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:56:37] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:56:37] / on z-dat-s3-a is OK: DISK OK - free space: / 8230 MB (27% inode=85%): [05:56:37] / on z-dat-s4-a is OK: DISK OK - free space: / 8230 MB (27% inode=85%): [05:56:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:00:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:05:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:15:57] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [06:17:07] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [06:20:16] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:21:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:22:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [06:29:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449669 MB (8% inode=42%): [06:40:16] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:41:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:47:16] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 115274 [06:47:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:47:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:55:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:56:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:00:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:15:56] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [07:18:06] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [07:22:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [07:29:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449553 MB (8% inode=42%): [07:30:26] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:31:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:46:37] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:47:16] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 118814 [07:47:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:47:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:54:16] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [07:55:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:55:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:56:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:00:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:15:57] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [08:18:07] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41984 [08:22:47] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [08:26:17] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:30:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449472 MB (8% inode=42%): [08:47:16] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120540 [08:47:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:47:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:55:17] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [08:56:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:56:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:57:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:01:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:13:16] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:47] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:15:57] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [09:18:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34794 [09:22:47] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [09:30:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449357 MB (8% inode=42%): [09:35:17] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [09:36:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:47:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121888 [09:47:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:47:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:53:44] 3(commented) [ACCAPP-524] Evaluation tool for wikiversity pages, rating the articles with a 5 start rating system <10https://jira.toolserver.org/browse/ACCAPP-524> (Daniel Reich) [09:56:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:57:27] /sql on cassia is WARNING: DISK WARNING - free space: /sql 98800 MB (8% inode=99%): [09:57:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:01:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:02:44] 3(updated) [ACCAPP-524] Evaluation tool for wikiversity pages, rating the articles with a 5 start rating system <10https://jira.toolserver.org/browse/ACCAPP-524> (Daniel Reich) [10:15:17] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:15:56] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [10:16:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:18:16] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5513 [10:18:17] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [10:23:15] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3273 [10:23:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [10:26:16] MySQL slave on z-dat-s7-a is OK: Uptime: 1578650 Threads: 8 Questions: 518801193 Slow queries: 94733 Opens: 3476455 Flush tables: 1 Open tables: 5879 Queries per second avg: 328.635 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1709 [10:30:47] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 449242 MB (8% inode=42%): [10:38:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:45:46] MySQL on ha-sql.esi is OK: Uptime: 48995 Threads: 6 Questions: 563403 Slow queries: 1 Opens: 664 Flush tables: 1 Open tables: 15 Queries per second avg: 11.499 [10:47:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123091 [10:47:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:49:56] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1.000000 [10:57:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:57:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:59:56] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [11:00:17] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [11:01:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:07:56] MySQL slave on daphne is OK: Uptime: 8759559 Threads: 18 Questions: 597293884 Slow queries: 333751 Opens: 1144 Flush tables: 2 Open tables: 323 Queries per second avg: 68.187 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [11:08:16] MySQL slave on z-dat-s4-a is OK: Uptime: 1602804 Threads: 8 Questions: 103330400 Slow queries: 17148 Opens: 31908 Flush tables: 1 Open tables: 953 Queries per second avg: 64.468 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 545 [11:30:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448991 MB (8% inode=42%): [11:38:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:48:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 124217 [11:48:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:57:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:57:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:02:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:05:26] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:06:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:29:46] 3(assigned) [TS-1376] become not working on yarrow <10https://jira.toolserver.org/browse/TS-1376> (DaB.) [12:29:47] 3(work started) [TS-1376] become not working on yarrow <10https://jira.toolserver.org/browse/TS-1376> (DaB.) [12:30:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448886 MB (8% inode=42%): [12:42:16] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:43:17] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:45:17] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [12:48:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:49:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 125090 [12:57:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:57:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:02:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:13:36] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41885 MB (10% inode=99%): [13:16:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:30:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448686 MB (8% inode=42%): [13:38:46] 3(resolved) [TS-1376] become not working on yarrow <10https://jira.toolserver.org/browse/TS-1376> (DaB.) [13:44:35] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24266 MB (5% inode=99%): [13:48:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:49:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 126119 [13:52:56] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196407 MB (20% inode=99%): [13:54:36] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 86059 MB (21% inode=99%): [13:56:16] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1812 [13:57:16] MySQL slave on z-dat-s7-a is OK: Uptime: 1591310 Threads: 8 Questions: 520361646 Slow queries: 95119 Opens: 3476528 Flush tables: 1 Open tables: 5952 Queries per second avg: 327.2 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1752 [13:57:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:57:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:00:57] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:01:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:17:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:29:46] Load avg. on adenia is WARNING: WARNING - load average: 18.60, 11.46, 6.61 [14:30:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 454677 MB (8% inode=42%): [14:31:58] [[Special:Log/newusers]] create 10 * Sea0q0uype * (New user account) [14:48:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:49:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 127007 [14:51:46] Load avg. on adenia is OK: OK - load average: 11.98, 14.93, 14.39 [14:57:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:57:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:01:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:17:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:30:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448430 MB (8% inode=42%): [15:48:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:49:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 127999 [15:56:36] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:58:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:58:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:00:15] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [16:01:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:01:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:21:36] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:30:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448292 MB (8% inode=42%): [16:36:36] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:50:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 128591 [16:58:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:58:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:01:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:02:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:13:41] [[Job scheduling]] 10https://wiki.toolserver.org/w/index.php?diff=7379&oldid=7308&rcid=9950 * Dispenser * (-45) (/* Submitting jobs */ grammar) [17:21:20] Job scheduling documentation is shit [17:23:01] /properly estimate a job's resource usage use / Catch-22, to run a job I need memory stats and max runtime [17:29:46] Any SGE/Job scheduling experts here? [17:31:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448187 MB (8% inode=42%): [17:36:14] * Dispenser pokes Brooke to fix SGE's documentation [17:48:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:50:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 129451 [17:58:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:58:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:59:18] [[Job scheduling]] 10https://wiki.toolserver.org/w/index.php?diff=7380&oldid=7379&rcid=9951 * Dispenser * (+303) (Replaced deprecated with and with bolding; job id and jobid -> job_id; spellcheck) [18:01:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:02:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:07:47] Merlissimo: You're an SGE guguru? How do I make SGE work like cron (i.e. stdout and stderr is emailed to me if not blank and not tucked away in my home directory) [18:23:49] Dispenser: Ive gotten to the point that I consider sge a lost cause [18:24:36] Hmm maybe we should slap something across the top of the page [18:25:11] Dispenser: its still encouraged but Ive found way too many headaches with it [18:31:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 448098 MB (8% inode=42%): [18:45:15] /tmp on willow is WARNING: DISK WARNING - free space: / 21998 MB (20% inode=99%): [18:45:57] / on willow is WARNING: DISK WARNING - free space: / 21998 MB (20% inode=99%): [18:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:49:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:50:25] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 129061 [18:51:59] [[Job scheduling]] M 10https://wiki.toolserver.org/w/index.php?diff=7381&oldid=7380&rcid=9952 * Dispenser * (+17) (/* Submitting jobs */ needs quotes if tool arguments are specified) [18:55:16] /tmp on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:55:16] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:56:16] /tmp on willow is WARNING: DISK WARNING - free space: / 21989 MB (20% inode=99%): [18:56:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:58:03] hello all [18:58:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:58:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:02:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:12:04] @replag [19:12:05] Akoopal: s2-user: 1w 4d 10h 8m 31s [-2.33 s/s]; s3-rr-a: 53s [-0.01 s/s]; s3-user: 53s [-0.01 s/s]; s6-rr-a: 1d 11h 50m 48s [+0.06 s/s]; s6-user: 1d 11h 50m 48s [+0.06 s/s]; s7-rr-a: 6m 45s [-0.01 s/s]; s7-user: 6m 45s [-0.01 s/s] [19:14:56] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [19:15:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [19:31:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447971 MB (8% inode=42%): [19:32:03] Dr. Trigon * [Toolserver-l] Write attempts to wiki API from TS timeout [19:32:03] Dr. Trigon * [Toolserver-l] Install OpenCV libraries and python bindings [19:32:03] Dr. Trigon * [Toolserver-l] SGE qcronsub "auto-flush"? [19:32:16] /tmp on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:32:16] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:33:16] /tmp on willow is WARNING: DISK WARNING - free space: / 21952 MB (20% inode=99%): [19:33:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:38:16] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1963 [19:42:10] /tmp on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:42:10] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:45:16] MySQL slave on z-dat-s7-a is OK: Uptime: 1612190 Threads: 16 Questions: 524563018 Slow queries: 95605 Opens: 3535322 Flush tables: 1 Open tables: 6003 Queries per second avg: 325.372 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1702 [19:46:56] / on willow is WARNING: DISK WARNING - free space: / 21920 MB (20% inode=99%): [19:49:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:49:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:50:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 129420 [19:57:07] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:57:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:58:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:58:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:05:40] DaBPunkt: My crontab doesn't seem to be running [20:06:00] which host? [20:12:54] willow [20:14:18] The script just worked fine from the command line (sending error emails) [20:14:57] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [20:15:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [20:16:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:18:54] Dispenser: there are several cron-entries of you AFAIS – are all not running or just 1? [20:19:37] $HOME/scripts/mailerrors.sh is the one I noticed [20:20:17] I missed about 1,200 emails today about a typo [20:21:13] looks like its working again [20:22:16] /tmp on willow is WARNING: DISK WARNING - free space: / 21899 MB (20% inode=99%): [20:22:35] are there any news about getting back from zws to apache? [20:22:58] I see no executiions of crons by you at all [20:30:30] Danny_B|backup: I will convert the webhosts to linux after the userlnad-servers [20:30:51] so somewhen in the sommer [20:31:00] ouch... [20:31:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447849 MB (8% inode=42%): [20:36:36] `0,10,20,30,40,50 * * * * $HOME/scripts/mailerrors.sh` didn't run at 20:30 UTC [20:38:11] Dispenser: do you know when it was executed sucessful last time? (my log says taht it was in february) [20:40:35] The last email I got was on June 10 [20:41:08] which time? [20:41:39] Sun Jun 10 17:10 UTC [20:43:02] Platonides * Re: [Toolserver-l] Write attempts to wiki API from TS timeout [20:44:15] Ok I just got my test email [20:46:56] / on willow is WARNING: DISK WARNING - free space: / 21876 MB (20% inode=99%): [20:49:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:49:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:50:02] Platonides * Re: [Toolserver-l] SGE qcronsub "auto-flush"? [20:50:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 131375 [20:52:56] / on willow is OK: DISK OK - free space: / 22071 MB (21% inode=99%): [20:53:16] /tmp on willow is OK: DISK OK - free space: / 22070 MB (21% inode=99%): [20:56:37] Load avg. on willow is WARNING: WARNING - load average: 18.75, 14.66, 9.24 [20:56:56] / on willow is WARNING: DISK WARNING - free space: / 21756 MB (20% inode=99%): [20:57:36] Load avg. on willow is OK: OK - load average: 13.45, 14.00, 9.36 [20:57:56] / on willow is OK: DISK OK - free space: / 25659 MB (24% inode=99%): [20:57:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:58:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:59:55] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:02:36] Load avg. on willow is WARNING: WARNING - load average: 15.42, 18.59, 12.64 [21:03:56] Dispenser: I converted your crontab-cron to a cronie-tab now to make it easier to follow the processes in the log [21:07:02] Ok [21:11:12] Jun 12 21:10:04 willow /OPT/TS/SBIN/CROND[3997]: [ID 529850 cron.info] (dispenser) CMD ($HOME/scripts/mailerrors.sh) [21:11:46] Thanks. [21:11:49] let us look what happens at :20 [21:15:56] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [21:16:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:16:46] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [21:18:56] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 2.000000 [21:19:56] @replag [21:19:56] DaBPunkt: s2-user: 1w 4d 8h 54m 48s [-0.16 s/s]; s3-rr-a: 22s [+0.01 s/s]; s3-user: 22s [+0.01 s/s]; s6-rr-a: 1d 12h 38m 33s [+0.31 s/s]; s6-user: 1d 12h 38m 33s [+0.31 s/s]; s7-rr-a: 2m 52s [-0.35 s/s]; s7-user: 2m 52s [-0.35 s/s] [21:20:18] Dispenser: it runs [21:32:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 451860 MB (8% inode=42%): [21:50:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:50:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:51:21] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 132864 [21:57:31] /sql on cassia is WARNING: DISK WARNING - free space: /sql 96791 MB (8% inode=99%): [21:58:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:59:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:00:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:13:45] 3(commented) [DRTRIGON-118] Enable lua support to enable even better template programming inclusion (subster) <10https://jira.toolserver.org/browse/DRTRIGON-118> (drtrigon) [22:16:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:17:01] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@damiana (using password: YES) [22:21:00] MySQL on ha-sql.esi is OK: Uptime: 90706 Threads: 11 Questions: 1327267 Slow queries: 1 Opens: 1171 Flush tables: 1 Open tables: 13 Queries per second avg: 14.632 [22:23:11] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Error with query - FUNCTION dewiki_p.ts_rc_age does not exist [22:24:11] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1.000000 [22:25:36] Was cron broken or something? [22:25:46] I had a few jobs that didn't run for a few hours as well. [22:26:22] @replag [22:26:22] Brooke: s2-user: 1w 4d 8h 8m 31s [-0.81 s/s]; s3-rr-a: 39s [+0.01 s/s]; s3-user: 39s [+0.01 s/s]; s6-rr-a: 1d 13h 11m 31s [+0.50 s/s]; s6-user: 1d 13h 11m 31s [+0.50 s/s]; s7-rr-a: 16m 33s [+0.10 s/s]; s7-user: 16m 33s [+0.10 s/s] [22:29:00] Brooke: I can't say for sure. The problem is that the old solaris crontab-system does nearly no logging [22:29:40] Is nightshade up again? [22:31:19] no [22:31:56] :-( [22:32:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447741 MB (8% inode=42%): [22:35:15] @replag [22:35:16] DaBPunkt: s2-user: 1w 4d 7h 48m 27s [-2.26 s/s]; s6-rr-a: 1d 13h 16m 26s [+0.55 s/s]; s6-user: 1d 13h 16m 26s [+0.55 s/s] [22:35:52] looks like s2 is finaly done [22:50:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:50:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:51:21] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134612 [22:57:51] nacht ts [22:58:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:59:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:00:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:17:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:32:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 443448 MB (8% inode=42%): [23:50:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:50:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:51:22] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 136309 [23:58:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:59:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default