[00:00:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:16:41] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:17:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:26:12] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:28:01] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:28:41] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:32:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 424931 MB (8% inode=41%): [00:42:50] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:51] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:02] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:43:21] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:21] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:43:42] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:01] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [00:44:01] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [00:44:11] MySQL slave on z-dat-s3-a is OK: Uptime: 3990202 Threads: 22 Questions: 4176806267 Slow queries: 274414 Opens: 41325068 Flush tables: 1 Open tables: 16384 Queries per second avg: 1046.765 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 236 [00:44:11] MySQL on z-dat-s3-a is OK: Uptime: 3990202 Threads: 22 Questions: 4176806272 Slow queries: 274414 Opens: 41325069 Flush tables: 1 Open tables: 16384 Queries per second avg: 1046.765 [00:44:11] SMTP on hyacinth is OK: SMTP OK - 0.022 sec. response time [00:44:11] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:44:32] SMTP on z-dat-s6-a is OK: SMTP OK - 0.081 sec. response time [00:44:41] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:44:42] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:50:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:50:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:52:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 138604 [00:53:32] Free Memory on damiana is WARNING: WARNING - 6.3% (263340 kB) free! [00:55:22] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:33] Free Memory on damiana is OK: OK - 7.5% (313816 kB) free. [00:55:42] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:42] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:43] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:43] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:43] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:43] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:43] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:43] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:43] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:44] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:44] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:55:45] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:01] Load avg. on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:01] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:01] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:01] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:13] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:13] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:13] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:13] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:41] SMF on z-dat-s7-a is OK: OK - all services online [00:56:41] SMF on z-dat-s6-a is OK: OK - all services online [00:56:41] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:56:41] / on z-dat-s3-a is OK: DISK OK - free space: / 8224 MB (27% inode=85%): [00:56:41] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [00:56:42] / on z-dat-s6-a is OK: DISK OK - free space: / 8224 MB (27% inode=85%): [00:56:42] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 68016 MB (16% inode=99%): [00:56:43] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 115942 MB (11% inode=98%): [00:56:43] / on z-dat-s7-a is OK: DISK OK - free space: / 8224 MB (27% inode=85%): [00:56:44] / on z-dat-s4-a is OK: DISK OK - free space: / 8224 MB (27% inode=85%): [00:56:44] /tmp on hyacinth is OK: DISK OK - free space: /tmp 1762 MB (99% inode=99%): [00:56:45] SMF on z-dat-s4-a is OK: OK - all services online [00:56:45] SMF on z-dat-s3-a is OK: OK - all services online [00:56:46] Load avg. on hyacinth is OK: OK - load average: 0.71, 2.19, 2.52 [00:56:50] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [00:56:51] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [00:57:01] MySQL slave on z-dat-s7-a is OK: Uptime: 1630894 Threads: 9 Questions: 531269512 Slow queries: 96496 Opens: 3563508 Flush tables: 1 Open tables: 6017 Queries per second avg: 325.753 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 102 [00:57:13] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 1808 MB (99% inode=99%): [00:57:13] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 115934 MB (11% inode=98%): [00:57:13] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 78147 MB (19% inode=99%): [00:57:13] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 1900 MB (99% inode=99%): [00:57:13] Load avg. on z-dat-s3-a is OK: OK - load average: 2.38, 2.46, 2.61 [00:57:13] / on hyacinth is OK: DISK OK - free space: / 8224 MB (27% inode=85%): [00:57:13] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1900 MB (99% inode=99%): [00:57:14] Load avg. on z-dat-s4-a is OK: OK - load average: 2.42, 2.47, 2.61 [00:57:14] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 1896 MB (99% inode=99%): [00:57:15] SMF on hyacinth is OK: OK - all services online [00:57:15] Load avg. on z-dat-s7-a is OK: OK - load average: 2.42, 2.47, 2.61 [00:57:16] Load avg. on z-dat-s6-a is OK: OK - load average: 2.46, 2.48, 2.61 [00:59:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:59:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:00:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:09:41] [[Special:Log/newusers]] create 10 * Goutsolution * (New user account) [01:18:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:28:13] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:28:51] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:28:51] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:28:52] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:29:42] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:29:42] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:29:42] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:29:51] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:30:31] Free Memory on damiana is WARNING: WARNING - 6.5% (270756 kB) free! [01:32:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 424932 MB (8% inode=41%): [01:50:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:50:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:52:32] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 141555 [01:53:13] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196016 MB (20% inode=99%): [01:59:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:00:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:00:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:01:59] [[Job scheduling]] !M 10https://wiki.toolserver.org/w/index.php?diff=7382&oldid=7381&rcid=9954 * Jason * (-3) (/* Submitting jobs */ fix typo) [02:09:23] [[Job scheduling]] !M 10https://wiki.toolserver.org/w/index.php?diff=7383&oldid=7382&rcid=9955 * Jason * (+0) (/* Best practices */ another typo) [02:14:41] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:51] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:51] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:51] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:52] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:15:12] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:12] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:12] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:12] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:13] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:13] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:13] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:14] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:14] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:21] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:41] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:41] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:41] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:41] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:41] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:15:41] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:41] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:42] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:42] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:43] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:43] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:15:44] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 115739 MB (11% inode=98%): [02:16:11] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 78075 MB (19% inode=99%): [02:16:12] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 1930 MB (99% inode=99%): [02:16:12] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 115731 MB (11% inode=98%): [02:16:12] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 1931 MB (99% inode=99%): [02:16:12] Load avg. on z-dat-s6-a is OK: OK - load average: 2.00, 1.85, 2.36 [02:16:12] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1930 MB (99% inode=99%): [02:16:12] Load avg. on z-dat-s7-a is OK: OK - load average: 2.05, 1.86, 2.36 [02:16:13] Load avg. on z-dat-s4-a is OK: OK - load average: 2.05, 1.86, 2.36 [02:16:13] Load avg. on z-dat-s3-a is OK: OK - load average: 2.05, 1.86, 2.36 [02:16:14] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 1930 MB (99% inode=99%): [02:16:31] SMTP on z-dat-s6-a is OK: SMTP OK - 0.122 sec. response time [02:16:31] SMTP on z-dat-s7-a is OK: SMTP OK - 0.078 sec. response time [02:19:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:30:32] Free Memory on damiana is OK: OK - 7.1% (295684 kB) free. [02:32:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 442189 MB (8% inode=42%): [02:33:31] Free Memory on damiana is WARNING: WARNING - 6.9% (290176 kB) free! [02:36:31] Free Memory on damiana is OK: OK - 7.1% (296720 kB) free. [02:50:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:51:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:52:51] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 143917 [02:59:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:00:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:01:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:16:42] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:19:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:32:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 441454 MB (8% inode=42%): [03:50:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:52:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:52:52] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145488 [03:56:21] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:00:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:00:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:01:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:05:51] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43485 MB (10% inode=99%): [04:11:42] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:16:52] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 67926 MB (16% inode=99%): [04:19:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:21:21] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:21:21] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:21:21] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:21:31] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:21:41] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:21:41] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:21:41] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:21:53] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:32] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [04:22:32] SMTP on z-dat-s3-a is OK: SMTP OK - 3.291 sec. response time [04:22:32] SMTP on z-dat-s6-a is OK: SMTP OK - 0.002 sec. response time [04:22:32] SMTP on z-dat-s7-a is OK: SMTP OK - 0.080 sec. response time [04:22:32] SMTP on z-dat-s4-a is OK: SMTP OK - 0.078 sec. response time [04:22:41] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 2.000000 [04:22:41] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:23:01] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:23:01] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [04:23:12] SMTP on hyacinth is OK: SMTP OK - 0.005 sec. response time [04:30:11] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7384&oldid=7364&rcid=9956 * 91.198.174.202 * (+3) (updated page) [04:32:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 447844 MB (8% inode=42%): [04:50:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:52:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:53:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 146232 [05:00:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:01:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:01:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:20:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:32:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 441399 MB (8% inode=42%): [05:50:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:53:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145706 [06:00:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:01:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:02:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:20:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:32:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 441280 MB (8% inode=42%): [06:50:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:53:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145667 [06:55:00] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:00] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:01] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:01] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:22] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:22] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:22] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:22] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:22] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:22] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:22] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:23] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:23] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:32] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:32] Free Memory on damiana is WARNING: WARNING - 7.0% (294140 kB) free! [06:55:41] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:41] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:41] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:41] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:41] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:41] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:41] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:42] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:42] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:43] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:43] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:52] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:55:52] SMF on z-dat-s3-a is OK: OK - all services online [06:55:52] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:55:52] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:55:52] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:55:52] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:55:52] / on z-dat-s3-a is OK: DISK OK - free space: / 8226 MB (27% inode=85%): [06:55:53] / on z-dat-s6-a is OK: DISK OK - free space: / 8226 MB (27% inode=85%): [06:55:53] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 116631 MB (12% inode=98%): [06:55:54] / on z-dat-s7-a is OK: DISK OK - free space: / 8226 MB (27% inode=85%): [06:55:54] SMF on z-dat-s6-a is OK: OK - all services online [06:55:55] / on z-dat-s4-a is OK: DISK OK - free space: / 8226 MB (27% inode=85%): [06:56:11] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [06:56:12] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 116630 MB (12% inode=98%): [06:56:12] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 1771 MB (99% inode=99%): [06:56:12] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 75571 MB (18% inode=99%): [06:56:12] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 1773 MB (99% inode=99%): [06:56:12] Load avg. on z-dat-s6-a is OK: OK - load average: 2.28, 2.39, 2.96 [06:56:12] / on hyacinth is OK: DISK OK - free space: / 8226 MB (27% inode=85%): [06:56:12] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1772 MB (99% inode=99%): [06:56:13] Load avg. on z-dat-s7-a is OK: OK - load average: 2.32, 2.39, 2.96 [06:56:13] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 1810 MB (99% inode=99%): [06:56:21] Load avg. on z-dat-s4-a is OK: OK - load average: 2.57, 2.45, 2.98 [06:56:31] Free Memory on damiana is OK: OK - 7.3% (306452 kB) free. [06:56:31] SMTP on z-dat-s6-a is OK: SMTP OK - 0.132 sec. response time [06:56:31] SMTP on z-dat-s7-a is OK: SMTP OK - 0.031 sec. response time [06:58:28] hi s3 is down, please help restore. [06:59:23] looks like it is ok now. [07:00:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:01:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:02:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:13:31] Free Memory on damiana is WARNING: WARNING - 6.9% (287596 kB) free! [07:20:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:32:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 441193 MB (8% inode=42%): [07:41:41] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:48:02] Dr. Trigon * Re: [Toolserver-l] Write attempts to wiki API from TS timeout [07:50:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:52:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:53:11] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145224 [08:00:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:01:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:02:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:02:52] Load avg. on willow is WARNING: WARNING - load average: 16.14, 12.89, 10.16 [08:03:52] Load avg. on willow is OK: OK - load average: 10.92, 12.02, 10.03 [08:13:31] Free Memory on damiana is WARNING: WARNING - 6.1% (254088 kB) free! [08:20:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:21:12] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:23:12] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1956.000000 [08:32:12] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 441141 MB (8% inode=42%): [08:42:02] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:02] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:02] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:03] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:32] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:42:32] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:42:41] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:41] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:41] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:42:51] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:42:51] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:42:51] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:42:52] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:42:52] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:43:01] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:43:11] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [08:43:11] Load avg. on z-dat-s7-a is OK: OK - load average: 1.05, 1.65, 2.21 [08:43:21] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [08:43:21] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [08:43:21] Load avg. on z-dat-s3-a is OK: OK - load average: 1.42, 1.71, 2.23 [08:43:31] MySQL on z-dat-s3-a is OK: Uptime: 4018962 Threads: 27 Questions: 4208387762 Slow queries: 277526 Opens: 41772508 Flush tables: 1 Open tables: 16384 Queries per second avg: 1047.133 [08:43:31] MySQL slave on z-dat-s3-a is OK: Uptime: 4018962 Threads: 27 Questions: 4208387763 Slow queries: 277526 Opens: 41772508 Flush tables: 1 Open tables: 16384 Queries per second avg: 1047.133 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 159 [08:43:31] SMTP on z-dat-s6-a is OK: SMTP OK - 0.011 sec. response time [08:43:31] SMTP on z-dat-s7-a is OK: SMTP OK - 0.004 sec. response time [08:50:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:51:12] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [08:53:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:53:21] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145911 [08:56:51] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:00:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:01:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:02:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:14:31] Free Memory on damiana is WARNING: WARNING - 5.4% (227056 kB) free! [09:20:31] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:51:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:51:30] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7256.000000 [09:53:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:54:00] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 146261 [09:57:10] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1921.000000 [09:58:09] /sql on cassia is WARNING: DISK WARNING - free space: /sql 97595 MB (8% inode=99%): [10:00:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:02:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:02:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:20:44] 3(created) [TS-1403] toolserver.namespacename not up-to-date on s3/s4/s6/s7; Toolserver: Toolserver-DB; Bug <10https://jira.toolserver.org/browse/TS-1403> (merl) [10:21:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:25:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3603.000000 [10:36:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 442664 MB (8% inode=42%): [10:51:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:51:28] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10856.000000 [10:53:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:54:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145297 [11:01:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:02:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:03:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:21:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:25:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7203.000000 [11:36:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 440859 MB (8% inode=42%): [11:51:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:51:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14456.000000 [11:53:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:54:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 144776 [12:01:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:02:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:03:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:07:19] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:20:44] 3(assigned) [TS-1403] toolserver.namespacename not up-to-date on s3/s4/s6/s7 <10https://jira.toolserver.org/browse/TS-1403> (DaB.) [12:21:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:21:58] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:26:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10863.000000 [12:33:43] 3(resolved) [TS-1403] toolserver.namespacename not up-to-date on s3/s4/s6/s7 <10https://jira.toolserver.org/browse/TS-1403> (DaB.) [12:36:29] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 445569 MB (8% inode=42%): [12:44:52] @replag [12:44:52] DaBPunkt: s1-rr-a: 50s [+0.58 s/s]; s1-user: 46s [+0.00 s/s]; s2-user: 1w 2d 22h 14m 9s [+0.37 s/s]; s2-user-c: 4h 54m 22s [+0.99 s/s]; s3-rr-a: 26s [-0.41 s/s]; s3-user: 26s [-0.41 s/s]; s5-user-c: 4h 54m 23s [+1.00 s/s]; s6-rr-a: 1d 16h 17m 26s [+0.10 s/s] [12:44:53] DaBPunkt: s6-user: 1d 16h 17m 26s [+0.10 s/s] [12:48:42] 3(created) [MNT-1249] Restart replication of commons on s2; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1249> (DaB.) [12:51:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18059.000000 [12:52:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:53:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:55:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145098 [13:00:19] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:40] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:01:19] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [13:01:29] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [13:01:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:02:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:03:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:12:41] 3(commented) [MNT-1249] Restart replication of commons on s2 <10https://jira.toolserver.org/browse/MNT-1249> (DaB.) [13:17:42] 3(commented) [MNT-1249] Restart replication of commons on s2 <10https://jira.toolserver.org/browse/MNT-1249> (DaB.) [13:21:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:26:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14463.000000 [13:36:39] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 440660 MB (8% inode=42%): [13:51:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21663.000000 [13:52:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:53:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:53:51] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199818 MB (20% inode=99%): [13:55:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145431 [14:01:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:02:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:03:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:18:43] 3(created) [TS-1404] View commonswiki_p.globalimagelinks is missing on cassia; Toolserver: Databases; Bug <10https://jira.toolserver.org/browse/TS-1404> (merl) [14:22:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:26:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18064.000000 [14:36:39] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 439831 MB (8% inode=42%): [14:42:43] 3(updated) [DBQ-157] simple pagelinks count query <10https://jira.toolserver.org/browse/DBQ-157> [14:42:45] 3(updated) [DBQ-157] simple pagelinks count query <10https://jira.toolserver.org/browse/DBQ-157> [14:44:42] 3(updated) [DBQ-157] + <10https://jira.toolserver.org/browse/DBQ-157> [14:51:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25265.000000 [14:52:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:53:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:54:43] 3(created) [TS-1405] Are database indexes on s5 missing?; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1405> (Marlen Caemmerer) [14:54:45] 3(assigned) [TS-1405] Are database indexes on s5 missing? <10https://jira.toolserver.org/browse/TS-1405> (Marlen Caemmerer) [14:55:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 144987 [14:56:41] [[Special:Log/newusers]] create 10 * Apple-9L0-010-exam * (New user account) [15:02:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:02:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:03:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:08:59] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:Apple-9L0-010-exam10]] with an expiry time of infinite (account creation disabled): Spam) [15:09:12] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:Goutsolution10]] with an expiry time of infinite (account creation disabled): Spam) [15:09:23] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:Seodistribute610]] with an expiry time of infinite (account creation disabled): Spam) [15:09:34] [[Special:Log/block]] block 10 * Betacommand * (blocked [[02User:Apple-9L0-063-exam10]] with an expiry time of infinite (account creation disabled): Spam) [15:22:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:26:20] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21670.000000 [15:36:40] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 440727 MB (8% inode=42%): [15:50:44] 3(created) [TS-1406] nightshade and yarrow; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1406> (Marlen Caemmerer) [15:51:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28865.000000 [15:52:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:52:44] 3(created) [TS-1407] Add more RAM to TS; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1407> (Marlen Caemmerer) [15:52:46] 3(assigned) [TS-1407] Add more RAM to TS <10https://jira.toolserver.org/browse/TS-1407> (Marlen Caemmerer) [15:52:52] 3(created) [TS-1408] Add a switch to TS; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1408> (Marlen Caemmerer) [15:53:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:54:46] 3(assigned) [TS-1408] Add a switch to TS <10https://jira.toolserver.org/browse/TS-1408> (Marlen Caemmerer) [15:55:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 141761 [16:02:20] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:02:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:02:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:04:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:22:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:26:20] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25269.000000 [16:36:40] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 430730 MB (8% inode=41%): [16:47:09] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:51:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32466.000000 [16:52:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:53:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:55:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 141067 [17:02:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:03:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:05:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:08:39] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.380859/1.10, alarm hl:np_load_long=0.378906/1.55, alarm hl:mem_free=489.000000M/500M, alarm hl:tmp_free=13764M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.380859/1.00, alarm hl:np_load_long=0.378906/1.50, alarm hl:mem_free=489.000000M/600M, alarm hl:tmp_free= [17:09:38] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [17:16:39] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.363770/1.00, alarm hl:np_load_long=0.375976/1.50, alarm hl:mem_free=586.000000M/600M, alarm hl:tmp_free=13728M/100M, alarm hl:available=1/0 [17:22:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:27:19] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19045.000000 [17:36:38] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 429107 MB (8% inode=41%): [17:51:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36066.000000 [17:53:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:53:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:55:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 140259 [17:55:39] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.263184/1.00, alarm hl:np_load_long=0.286133/1.50, alarm hl:mem_free=522.000000M/600M, alarm hl:tmp_free=13659M/100M, alarm hl:available=1/0 [17:56:39] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [18:02:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:02:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:05:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:23:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:27:19] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8389.000000 [18:36:39] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 429004 MB (8% inode=41%): [18:51:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34812.000000 [18:53:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:53:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:55:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 138926 [19:02:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:03:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:05:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:06:19] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3549.000000 [19:10:49] hello, is there anyone around that could possibly help me with a TUSC verification issue? or is the only route to message Magnus? [19:20:19] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1636.000000 [19:23:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:37:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 441773 MB (8% inode=42%): [19:42:19] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:51:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29483.000000 [19:53:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:53:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:56:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 139207 [20:01:48] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:03:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:04:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:06:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:23:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:28:15] [[Special:Log/newusers]] create 10 * Salkd23lkjd * (New user account) [20:37:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 438418 MB (8% inode=41%): [20:45:58] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [20:46:39] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 287.000000 [20:51:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18804.000000 [20:53:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:53:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:56:19] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 139894 [21:03:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:04:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:06:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:24:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:29:18] @replag [21:29:18] Akoopal: s2-user: 1w 2d 10h 21m 36s [-2.22 s/s]; s2-user-c: 3h 40m 39s [-2.03 s/s]; s3-rr-a: 17s [-0.06 s/s]; s3-user: 17s [-0.06 s/s]; s5-user-c: 3h 40m 39s [-2.03 s/s]; s6-rr-a: 1d 14h 58m 43s [+0.18 s/s]; s6-user: 1d 14h 58m 43s [+0.18 s/s] [21:38:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 438306 MB (8% inode=41%): [21:41:43] 3(commented) [TS-1408] Add a switch to TS <10https://jira.toolserver.org/browse/TS-1408> (DaB.) [21:51:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10829.000000 [21:53:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:54:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:56:29] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 140690 [21:58:29] /sql on cassia is WARNING: DISK WARNING - free space: /sql 96275 MB (8% inode=99%): [21:58:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40402 MB (9% inode=99%): [22:03:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:04:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:06:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:13:40] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23903 MB (5% inode=99%): [22:16:39] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 48327 MB (11% inode=99%): [22:21:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40377 MB (9% inode=99%): [22:24:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:32:39] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23907 MB (5% inode=99%): [22:39:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 443046 MB (8% inode=42%): [22:48:59] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3574.000000 [22:49:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39867 MB (9% inode=99%): [22:52:40] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 66760 MB (16% inode=99%): [22:53:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:54:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:56:30] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 141713 [23:03:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:04:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:04:59] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1638.000000 [23:06:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:06:10] nacht ts [23:06:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42299 MB (10% inode=99%): [23:13:39] Pathoschild: poke [23:13:49] Hello. [23:14:32] My memory may be wrong, but isnt there a private Checkuser wiki? [23:14:44] There is. [23:15:07] https://checkuser.wikimedia.org [23:15:28] thanks [23:15:40] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 76718 MB (18% inode=99%): [23:15:41] Welcome. [23:25:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:39:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 436920 MB (8% inode=41%): [23:52:40] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:53:29] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [23:53:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:55:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:57:19] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:57:28] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 142976