[00:12:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:13:03] /sql on thyme is OK: DISK OK - free space: /sql 201533 MB (21% inode=99%): [00:14:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37785.000000 [00:15:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:16:03] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201351 MB (20% inode=99%): [00:16:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:19:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:29:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:30:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:46:16] /sql on thyme is OK: DISK OK - free space: /sql 201666 MB (21% inode=99%): [00:49:16] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201249 MB (20% inode=99%): [00:49:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 73209 MB (1% inode=10%): [01:07:16] /sql on thyme is OK: DISK OK - free space: /sql 202141 MB (21% inode=99%): [01:08:36] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:12:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:14:16] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33341.000000 [01:15:26] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:16:26] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:24:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:29:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:30:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:33:42] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [01:33:46] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 201.000000 [01:49:31] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 69927 MB (1% inode=10%): [02:00:38] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [02:01:30] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:12:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:14:29] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28014.000000 [02:15:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:16:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:24:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:29:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:49:40] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 68022 MB (1% inode=10%): [03:02:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:02:41] /sql on thyme is WARNING: DISK WARNING - free space: /sql 197122 MB (20% inode=99%): [03:07:40] /sql on thyme is OK: DISK OK - free space: /sql 201860 MB (21% inode=99%): [03:12:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:14:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19456.000000 [03:16:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:17:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:25:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:30:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:37:50] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201302 MB (20% inode=99%): [03:38:50] /sql on thyme is OK: DISK OK - free space: /sql 201718 MB (21% inode=99%): [03:49:51] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 67912 MB (1% inode=10%): [03:54:00] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:59:00] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:01:50] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200972 MB (20% inode=99%): [04:03:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:06:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41460 MB (10% inode=99%): [04:09:54] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 71292 MB (17% inode=99%): [04:12:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:14:52] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11392.000000 [04:16:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:17:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:26:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:29:21] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:29:54] /sql on thyme is OK: DISK OK - free space: /sql 201671 MB (21% inode=99%): [04:30:21] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [04:30:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:34:00] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 70111 MB (1% inode=10%): [04:54:23] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201420 MB (20% inode=99%): [04:56:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41392 MB (10% inode=99%): [04:59:23] /sql on thyme is OK: DISK OK - free space: /sql 201585 MB (21% inode=99%): [05:02:23] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201013 MB (20% inode=99%): [05:03:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:07:23] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24014 MB (5% inode=99%): [05:13:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:14:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4228.000000 [05:16:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:18:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:20:55] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3587.000000 [05:23:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [05:26:24] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41642 MB (10% inode=99%): [05:26:24] /sql on thyme is WARNING: DISK WARNING - free space: /sql 199086 MB (20% inode=99%): [05:26:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:28:24] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 70596 MB (17% inode=99%): [05:30:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:31:24] /sql on thyme is OK: DISK OK - free space: /sql 201802 MB (21% inode=99%): [05:38:25] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201407 MB (20% inode=99%): [05:40:24] /sql on thyme is OK: DISK OK - free space: /sql 201778 MB (21% inode=99%): [05:40:24] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:41:03] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:41:04] [[Screen]] ! 10https://wiki.toolserver.org/w/index.php?diff=7369&oldid=4998&rcid=9927 * 72.53.194.237 * (-774) (/* Quick-start */ ) [05:46:11] [[Screen]] ! 10https://wiki.toolserver.org/w/index.php?diff=7370&oldid=7369&rcid=9928 * 72.53.194.237 * (-444) (******************************************************) [05:46:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42225 MB (10% inode=99%): [05:49:31] [[Screen]] ! 10https://wiki.toolserver.org/w/index.php?diff=7371&oldid=7370&rcid=9929 * 72.53.194.237 * (-315) (:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::k) [05:50:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 68981 MB (1% inode=10%): [05:52:03] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3457.000000 [05:52:24] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 70763 MB (17% inode=99%): [06:00:04] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1739.000000 [06:03:26] [[Screen]] M 10https://wiki.toolserver.org/w/index.php?diff=7372&oldid=7371&rcid=9930 * Dispenser * (+1533) (Reverted edits by [[Special:Contributions/72.53.194.237|72.53.194.237]] ([[User talk:72.53.194.237|talk]]) to last revision by [[User:River|River]]) [06:03:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:13:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:14:23] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201134 MB (20% inode=99%): [06:16:23] /sql on thyme is OK: DISK OK - free space: /sql 201723 MB (21% inode=99%): [06:17:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:19:23] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201346 MB (20% inode=99%): [06:19:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:26:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:30:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 68877 MB (1% inode=10%): [07:03:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:13:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:17:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:19:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:26:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:30:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:42:44] 3(updated) [ACCAPP-515] Account for research and general curiosity for data mining of users and their contributions <10https://jira.toolserver.org/browse/ACCAPP-515> [07:49:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:51:13] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 68786 MB (1% inode=10%): [07:54:23] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:54:23] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:54:23] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:55:13] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [07:55:13] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [07:55:13] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [08:03:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:08:44] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:13:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:17:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:20:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:27:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:30:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:51:13] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 70292 MB (1% inode=10%): [09:03:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:05:33] /sql on thyme is OK: DISK OK - free space: /sql 201561 MB (21% inode=99%): [09:08:33] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201031 MB (20% inode=99%): [09:13:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:17:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:20:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:24:33] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42507 MB (10% inode=99%): [09:27:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:29:33] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 73228 MB (18% inode=99%): [09:30:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:32:23] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:43] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:52] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:33:14] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:33:33] SMTP on z-dat-s3-a is OK: SMTP OK - 0.003 sec. response time [09:33:43] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [09:34:43] 3(commented) [DBQ-186] List of good and excellent articles with the number of edits of the german Wikipedia <10https://jira.toolserver.org/browse/DBQ-186> (Clemens) [09:46:44] /sql on thyme is OK: DISK OK - free space: /sql 201931 MB (21% inode=99%): [09:49:43] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200729 MB (20% inode=99%): [09:51:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 68554 MB (1% inode=10%): [09:53:03] /sql on cassia is WARNING: DISK WARNING - free space: /sql 106124 MB (8% inode=99%): [10:04:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:13:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:17:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:17:43] /sql on thyme is OK: DISK OK - free space: /sql 202007 MB (21% inode=99%): [10:20:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:20:44] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200949 MB (20% inode=99%): [10:27:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:31:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:32:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38904 MB (9% inode=99%): [10:42:44] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24361 MB (5% inode=99%): [10:51:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 68414 MB (1% inode=10%): [10:53:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 31269 MB (7% inode=99%): [10:54:43] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 81134 MB (20% inode=99%): [11:02:43] /sql on thyme is OK: DISK OK - free space: /sql 201973 MB (21% inode=99%): [11:04:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:09:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40333 MB (9% inode=99%): [11:12:03] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.374512/1.10, alarm hl:np_load_long=0.363770/1.55, alarm hl:mem_free=497.000000M/500M, alarm hl:tmp_free=14234M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.374512/1.00, alarm hl:np_load_long=0.363770/1.50, alarm hl:mem_free=497.000000M/600M, alarm hl:tmp_free= [11:14:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:15:04] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [11:17:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:20:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:21:44] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 54974 MB (13% inode=99%): [11:27:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:31:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:37:03] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.347656/1.00, alarm hl:np_load_long=0.348633/1.50, alarm hl:mem_free=527.000000M/600M, alarm hl:tmp_free=14224M/100M, alarm hl:available=1/0 [11:37:33] Load avg. on adenia is WARNING: WARNING - load average: 17.12, 11.84, 7.72 [11:37:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42712 MB (10% inode=99%): [11:42:33] Load avg. on adenia is OK: OK - load average: 14.01, 14.02, 9.93 [11:43:13] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [11:43:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 76315 MB (18% inode=99%): [11:51:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 72251 MB (1% inode=10%): [11:59:02] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:04:03] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:04:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:06:33] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1904.000000 [12:07:52] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201245 MB (20% inode=99%): [12:10:53] /sql on thyme is OK: DISK OK - free space: /sql 201532 MB (21% inode=99%): [12:14:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:17:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:17:53] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201390 MB (20% inode=99%): [12:20:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:22:13] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.457520/1.00, alarm hl:np_load_long=0.349121/1.50, alarm hl:mem_free=515.000000M/600M, alarm hl:tmp_free=14193M/100M, alarm hl:available=1/0 [12:24:13] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [12:27:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:31:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:31:54] /sql on thyme is OK: DISK OK - free space: /sql 201602 MB (21% inode=99%): [12:32:53] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42314 MB (10% inode=99%): [12:41:23] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.354492/1.10, alarm hl:np_load_long=0.327149/1.55, alarm hl:mem_free=237.000000M/500M, alarm hl:tmp_free=14145M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.354492/1.00, alarm hl:np_load_long=0.327149/1.50, alarm hl:mem_free=237.000000M/600M, alarm hl:tmp_free= [12:41:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 66371 MB (16% inode=99%): [12:42:12] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1945 [12:42:24] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1959.000000 [12:46:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [12:47:54] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 28606 MB (7% inode=99%): [12:51:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 67353 MB (1% inode=9%): [12:51:43] toolserver.org HTTP on wolfsbane is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 7.904 second response time [12:52:43] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.007 second response time [12:53:53] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200605 MB (20% inode=99%): [13:04:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:13:16] [[Special:Log/newusers]] create 10 * Apple-9L0-063-exam * (New user account) [13:15:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:15:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3623 [13:15:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3636.000000 [13:17:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:20:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:24:03] [[Special:Log/newusers]] create 10 * Dewaltdcd * (New user account) [13:24:41] hello [13:24:43] @replag [13:24:46] MatmaRex_: s1-user: 1h 9m 58s [+0.06 s/s]; s2-user: 1w 1d 19h 26m 40s [+0.36 s/s]; s2-user-c: 1h 23m 29s [-0.28 s/s]; s3-rr-a: 1m 12s [+0.00 s/s]; s3-user: 1m 12s [+0.00 s/s]; s5-user-c: 1h 23m 29s [-0.28 s/s] [13:25:02] can someone tell me what's happening to s2? :( [13:25:54] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 409094 MB (7% inode=40%): [13:27:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:31:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:46:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5752.000000 [13:56:34] MatmaRex_: see status.toolserver [13:59:29] "Hight replag because of inserting of many SHA1-hashes" [13:59:35] well that totally expains *everything* [13:59:51] by which i mean, i have no idea what this indicates. [14:00:02] except that there is an error, and that i can see anyway :/ [14:00:12] MatmaRex_: its not an error [14:00:55] i'm not knowledgeable about this, maybe its not an error [14:01:07] but the replication clearly doesnt work :/ [14:01:16] is there any ETA as to when it might start working? [14:01:26] MatmaRex_: it is replicating just lagged [14:01:31] MatmaRex_: http://www.mediawiki.org/wiki/Manual:Revision_table#rev_sha1 [14:01:33] by a week [14:01:50] ah [14:01:51] so [14:01:59] the schema was changed, and it's replicating that change? [14:02:03] MatmaRex_: thats not really a big deal, at one point enwiki was lagged 6 months [14:02:09] MatmaRex_: yes [14:02:19] i finally know whats going on [14:02:19] thanks [14:02:31] i understand that replication is not critical [14:02:38] but some people extensively use tools that depend on it [14:02:41] like catscan [14:02:45] unlike the wmf we cant take the server out of rotation, run the update and bring it back [14:03:18] otherwise that would break a shit ton of things [14:03:26] i understand [14:03:31] there's a guy on pl.wiki right now complaining that it doesnt work and there no human-readable expanation anywhere, and he needs his catscan [14:03:44] and, well, he's right, that explanation is hardly an expalanation :) [14:04:09] MatmaRex_: it tells the toolserver users everything that we need to know [14:04:12] /sql on thyme is OK: DISK OK - free space: /sql 202605 MB (21% inode=99%): [14:04:26] it probably does [14:04:30] but for me it tells nothing [14:04:34] and i consider myself pretty technical [14:04:40] writing tools n bots n stuff [14:04:50] and that message tells me nothing [14:04:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:04:57] thats so high technicaaal [14:04:59] o hai [14:05:00] anyway, thanks [14:05:09] MatmaRex_: https://jira.toolserver.org/browse/TS-1398 [14:07:11] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201147 MB (20% inode=99%): [14:10:45] 3(commented) [TS-827] Database server out-of-sync? <10https://jira.toolserver.org/browse/TS-827> (Betacommand) [14:11:26] thanks. [14:12:01] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.135 second response time [14:13:02] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.016 second response time [14:14:10] /sql on thyme is OK: DISK OK - free space: /sql 201525 MB (21% inode=99%): [14:15:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:15:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5140 [14:16:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5147.000000 [14:16:02] John * [Toolserver-l] Bugzilla database replication [14:18:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:18:42] 3(assigned) [TS-1031] s1 replication issues <10https://jira.toolserver.org/browse/TS-1031> (Betacommand) [14:18:43] 3(resolved) [TS-1031] s1 replication issues <10https://jira.toolserver.org/browse/TS-1031> (Betacommand) [14:20:11] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38209 MB (9% inode=99%): [14:21:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:26:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410971 MB (7% inode=40%): [14:26:11] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23173 MB (5% inode=99%): [14:28:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:29:25] @replag [14:29:26] Akoopal: s1-user: 1h 23m 21s [+0.21 s/s]; s2-user: 1w 1d 20h 18m 40s [+0.80 s/s]; s2-user-c: 1h 59m 5s [+0.55 s/s]; s5-user-c: 1h 59m 5s [+0.55 s/s] [14:31:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:36:11] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200748 MB (20% inode=99%): [14:38:11] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38166 MB (9% inode=99%): [14:45:41] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.596191/1.10, alarm hl:np_load_long=0.477051/1.55, alarm hl:mem_free=476.000000M/500M, alarm hl:tmp_free=13988M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.596191/1.00, alarm hl:np_load_long=0.477051/1.50, alarm hl:mem_free=476.000000M/600M, alarm hl:tmp_free= [14:46:40] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [14:47:02] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7781.000000 [14:53:40] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.487793/1.00, alarm hl:np_load_long=0.468262/1.50, alarm hl:mem_free=583.000000M/600M, alarm hl:tmp_free=13994M/100M, alarm hl:available=1/0 [15:01:02] Krinkle * Re: [Toolserver-l] [Wikitech-l] Bugzilla database replication [15:05:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:15:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:15:31] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4074 [15:17:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4049.000000 [15:18:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:21:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:25:50] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:26:21] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:27:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 416185 MB (7% inode=40%): [15:28:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:31:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:39:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:47:03] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7826.000000 [15:58:34] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3571 [16:00:03] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3559.000000 [16:03:34] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:03:44] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.248047/1.10, alarm hl:np_load_long=0.242188/1.55, alarm hl:mem_free=355.000000M/500M, alarm hl:tmp_free=13910M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.248047/1.00, alarm hl:np_load_long=0.242188/1.50, alarm hl:mem_free=355.000000M/600M, alarm hl:tmp_free= [16:05:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:05:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3603 [16:06:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3629.000000 [16:07:44] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3592 [16:07:44] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [16:15:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:18:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:21:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:24:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:27:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 412790 MB (7% inode=40%): [16:27:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3697 [16:28:55] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:31:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:47:02] Platonides * Re: [Toolserver-l] [Wikitech-l] Bugzilla database replication [16:47:09] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10390.000000 [16:47:31] Krinkle, maybe you know how to disclose bugzilla bugs [16:47:51] Platonides: I think so [16:47:53] whats up [16:48:16] that was a question added up in the above mail above bugzilla [16:48:47] well, I didn't found a button to make the bug no longer private :P [16:50:39] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40933 MB (10% inode=99%): [16:51:09] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.203613/1.00, alarm hl:np_load_long=0.224609/1.50, alarm hl:mem_free=564.000000M/600M, alarm hl:tmp_free=13836M/100M, alarm hl:available=1/0 [16:52:10] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [17:01:39] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24124 MB (5% inode=99%): [17:06:09] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.300781/1.00, alarm hl:np_load_long=0.263184/1.50, alarm hl:mem_free=567.000000M/600M, alarm hl:tmp_free=13867M/100M, alarm hl:available=1/0 [17:06:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3737.000000 [17:06:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:14:40] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45702 MB (11% inode=99%): [17:15:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:19:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:22:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:25:10] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3587.000000 [17:25:40] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3581 [17:26:40] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3618 [17:27:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3614.000000 [17:27:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 408402 MB (7% inode=40%): [17:27:41] /sql on thyme is OK: DISK OK - free space: /sql 201528 MB (21% inode=99%): [17:27:41] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3589 [17:28:09] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3587.000000 [17:29:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:30:39] /sql on thyme is WARNING: DISK WARNING - free space: /sql 200466 MB (20% inode=99%): [17:32:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:43:23] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [17:47:23] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10823.000000 [18:00:22] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [18:01:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:02:31] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3060.000000 [18:15:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:19:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:22:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:27:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 401195 MB (7% inode=39%): [18:27:52] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2512 [18:30:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:32:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:41:31] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [18:44:03] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2531.000000 [18:45:43] 3(created) [TS-1400] Joining MMP "langlinksbot"; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1400> (reza) [18:47:22] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13289.000000 [19:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:09:42] /sql on thyme is OK: DISK OK - free space: /sql 201650 MB (21% inode=99%): [19:12:42] /sql on thyme is WARNING: DISK WARNING - free space: /sql 201128 MB (20% inode=99%): [19:16:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:20:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:22:52] MySQL slave on thyme is OK: Uptime: 2129890 Threads: 20 Questions: 900074472 Slow queries: 273682 Opens: 130844 Flush tables: 1 Open tables: 3305 Queries per second avg: 422.591 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1798 [19:23:02] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1796.000000 [19:23:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:27:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 400937 MB (7% inode=39%): [19:30:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:32:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:33:51] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1808 [19:34:02] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1810.000000 [19:43:02] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [19:47:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16008.000000 [20:01:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:02:12] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2055.000000 [20:04:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:08:35] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:16:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:20:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:23:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:27:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 392437 MB (7% inode=39%): [20:30:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:32:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:33:56] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2509 [20:37:42] 3(updated) [ACCAPP-413] I want to operate my global bot "GedawyBot" on toolserver in order to add and update interwikis <10https://jira.toolserver.org/browse/ACCAPP-413> (Mgedawy) [20:39:43] 3(commented) [ACCAPP-413] I want to operate my global bot "GedawyBot" on toolserver in order to add and update interwikis <10https://jira.toolserver.org/browse/ACCAPP-413> (Mgedawy) [20:47:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19090.000000 [21:01:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:02:46] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2788.000000 [21:17:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:20:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:23:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:27:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 384773 MB (7% inode=38%): [21:31:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:32:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:34:06] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3187 [21:47:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22273.000000 [21:52:02] Platonides * Re: [Toolserver-l] Update on IPv6 [21:53:16] /sql on cassia is WARNING: DISK WARNING - free space: /sql 106655 MB (9% inode=99%): [21:55:03] Platonides * Re: [Toolserver-l] Day of the open linux-server [22:00:16] SMF on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:01:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:01:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:02:46] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3450.000000 [22:17:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:20:26] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:27:06] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3609 [22:27:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [22:27:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 380114 MB (7% inode=38%): [22:31:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:32:57] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:44:57] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3578.000000 [22:45:15] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3553 [22:47:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25374.000000 [23:01:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:01:47] 3(created) [ACCAPP-527] Online converter for several vector formats to SVG; Account Approval; Minor New Account <10https://jira.toolserver.org/browse/ACCAPP-527> (Mark Blank) [23:02:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:17:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:21:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:28:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 374399 MB (7% inode=38%): [23:31:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:33:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:45:49] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2071.000000 [23:46:09] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2072 [23:47:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27807.000000 [23:51:09] MySQL slave on thyme is OK: Uptime: 2145987 Threads: 17 Questions: 906317096 Slow queries: 275451 Opens: 131799 Flush tables: 1 Open tables: 3314 Queries per second avg: 422.331 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1783 [23:51:48] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1742.000000