[00:10:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 168495 [00:17:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 226608 MB (4% inode=27%): [00:37:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:39:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:44:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:48:41] /sql on thyme is WARNING: DISK WARNING - free space: /sql 197856 MB (20% inode=99%): [00:53:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:55:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:59:39] /sql on thyme is OK: DISK OK - free space: /sql 228597 MB (23% inode=99%): [01:10:10] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 168985 [01:17:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 228892 MB (4% inode=27%): [01:37:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:39:29] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:44:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:53:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:55:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:09:50] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198607 MB (20% inode=99%): [02:10:20] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 169089 [02:17:21] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 228307 MB (4% inode=27%): [02:21:50] /sql on thyme is OK: DISK OK - free space: /sql 234979 MB (24% inode=99%): [02:38:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:39:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:45:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:53:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:55:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:10:22] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 170900 [03:17:22] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 226897 MB (4% inode=27%): [03:38:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:39:41] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:45:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:53:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:55:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:10:26] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 172979 [04:12:56] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.082031/1.95, alarm hl:tmp_free=39275M/100M, alarm hl:np_load_avg=1.037109/2.0, alarm hl:mem_free=325.000000M/350M, alarm hl:available=1/0 [04:13:56] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:17:46] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 226863 MB (4% inode=27%): [04:39:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:39:53] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:45:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:45:33] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [04:45:43] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 426.000000 [04:54:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:55:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:10:43] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 174573 [05:15:54] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.348633/1.10, alarm hl:np_load_long=0.651367/1.55, alarm hl:mem_free=10158.000000M/500M, alarm hl:tmp_free=14755M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.348633/1.00, alarm hl:np_load_long=0.651367/1.50, alarm hl:mem_free=10158.000000M/600M, alarm hl:tmp_free [05:16:54] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:17:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 226799 MB (4% inode=27%): [05:39:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:39:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:45:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:51:02] /sql on cassia is WARNING: DISK WARNING - free space: /sql 121758 MB (10% inode=99%): [05:54:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:55:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:02:03] /sql on thyme is WARNING: DISK WARNING - free space: /sql 192430 MB (20% inode=99%): [06:10:43] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 176431 [06:12:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.436524/1.95, alarm hl:tmp_free=38962M/100M, alarm hl:np_load_avg=1.422363/2.0, alarm hl:mem_free=269.000000M/350M, alarm hl:available=1/0 [06:14:53] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:17:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 226732 MB (4% inode=27%): [06:22:03] /sql on thyme is OK: DISK OK - free space: /sql 233989 MB (24% inode=99%): [06:24:22] Load avg. on adenia is WARNING: WARNING - load average: 16.92, 11.91, 6.82 [06:34:23] Load avg. on adenia is OK: OK - load average: 10.61, 14.14, 10.88 [06:39:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:40:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:45:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:45:53] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.424805/1.10, alarm hl:np_load_long=0.732422/1.55, alarm hl:mem_free=9194.000000M/500M, alarm hl:tmp_free=14647M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.424805/1.00, alarm hl:np_load_long=0.732422/1.50, alarm hl:mem_free=9194.000000M/600M, alarm hl:tmp_free=1 [06:47:53] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:54:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:56:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:02:42] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:10:43] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 178152 [07:12:33] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:17:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 226629 MB (4% inode=27%): [07:39:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:41:54] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:45:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:54:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:56:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:10:53] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180366 [08:17:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 225099 MB (4% inode=27%): [08:39:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:42:03] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:46:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:54:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:57:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:10:53] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180884 [09:17:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 224932 MB (4% inode=27%): [09:19:16] @replag all [09:19:16] nosy79: s1-rr-a: 2s [-0.01 s/s]; s1-user: 2s [-0.01 s/s]; s2-user: 5m 59s [+0.00 s/s]; s2-user-c: 2m 20s [+0.00 s/s]; s3-rr-a: 1s [-0.00 s/s]; s3-user: 1s [-0.00 s/s]; s4-rr-a: 0s [-0.00 s/s]; s4-user: 1s [-0.00 s/s] [09:19:17] nosy79: s5-rr-a: 3s [-0.00 s/s]; s5-user: 3s [-0.00 s/s]; s5-user-c: 2m 20s [+0.00 s/s]; s6-rr-a: 2d 2h 13m 28s [+0.28 s/s]; s6-user: 2d 2h 13m 28s [+0.28 s/s]; s7-rr-a: 1s [-]; s7-user: 1s [-] [09:19:46] @replag all [09:19:47] nosy79: s1-rr-a: 7s [+0.17 s/s]; s1-user: 7s [+0.17 s/s]; s2-user: 6m 20s [+0.70 s/s]; s2-user-c: 2m 11s [-0.30 s/s]; s3-rr-a: 13s [+0.40 s/s]; s3-user: 13s [+0.40 s/s]; s4-rr-a: 2s [+0.07 s/s]; s4-user: 2s [+0.03 s/s] [09:19:48] nosy79: s5-rr-a: 1s [-0.07 s/s]; s5-user: 1s [-0.07 s/s]; s5-user-c: 2m 11s [-0.30 s/s]; s6-rr-a: 2d 2h 13m 25s [-0.10 s/s]; s6-user: 2d 2h 13m 25s [-0.10 s/s]; s7-rr-a: 4s [+0.10 s/s]; s7-user: 4s [+0.10 s/s] [09:23:02] Merlijn van Deen * Re: [Toolserver-l] Toolserver user phe account expired [09:39:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:42:07] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:45:02] Magnus Manske * Re: [Toolserver-l] Toolserver user phe account expired [09:46:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:48:43] 3(created) [ACCAPP-514] Perform database queries across English and Telugu wiki projects for identifying quality improvement; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-514> (Arjuna Rao Chavala) [09:54:47] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:57:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:10:57] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180901 [10:12:43] 3(created) [MAGNUS-324] Catscan2 needs to support Telugu Wiki projects; Magnus' tools; Bug <10https://jira.toolserver.org/browse/MAGNUS-324> (Arjuna Rao Chavala) [10:17:57] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 223268 MB (4% inode=27%): [10:39:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:42:17] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:47:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:54:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:57:07] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:58:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.584961/1.95, alarm hl:tmp_free=38009M/100M, alarm hl:np_load_avg=0.594238/2.0, alarm hl:mem_free=316.000000M/350M, alarm hl:available=1/0 [11:01:07] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [11:05:17] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.265625/1.10, alarm hl:np_load_long=0.629883/1.55, alarm hl:mem_free=10812.000000M/500M, alarm hl:tmp_free=14257M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.265625/1.00, alarm hl:np_load_long=0.629883/1.50, alarm hl:mem_free=10812.000000M/600M, alarm hl:tmp_free [11:08:17] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:11:07] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 181613 [11:18:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 224258 MB (4% inode=27%): [11:39:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:42:27] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:43:17] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:54:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:57:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:57:48] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:12:07] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 182547 [12:13:17] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.664062/1.95, alarm hl:tmp_free=37797M/100M, alarm hl:np_load_avg=0.728516/2.0, alarm hl:mem_free=350.000000M/350M, alarm hl:available=1/0 [12:14:17] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [12:19:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 250213 MB (4% inode=29%): [12:31:02] Platonides * Re: [Toolserver-l] Toolserver user phe account expired [12:37:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.612305/1.95, alarm hl:tmp_free=37760M/100M, alarm hl:np_load_avg=0.608398/2.0, alarm hl:mem_free=287.000000M/350M, alarm hl:available=1/0 [12:39:18] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [12:39:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:42:26] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:47:36] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:54:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:57:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:12:17] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 183482 [13:16:18] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1886.000000 [13:17:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.474121/1.95, alarm hl:tmp_free=37429M/100M, alarm hl:np_load_avg=0.559082/2.0, alarm hl:mem_free=201.000000M/350M, alarm hl:available=1/0 [13:19:17] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 249188 MB (4% inode=29%): [13:20:27] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [13:39:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:42:37] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:47:47] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:54:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:55:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.659668/1.95, alarm hl:tmp_free=37634M/100M, alarm hl:np_load_avg=0.557617/2.0, alarm hl:mem_free=259.000000M/350M, alarm hl:available=1/0 [13:55:57] Load avg. on adenia is WARNING: WARNING - load average: 18.22, 11.48, 6.56 [13:57:27] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:57:56] Load avg. on adenia is OK: OK - load average: 9.03, 10.89, 6.99 [14:01:27] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [14:12:16] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 184315 [14:16:28] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2585.000000 [14:16:43] 3(created) [ACCAPP-515] Account for research and general curiosity for data mining of users and their contributions; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-515> [14:19:28] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 249204 MB (4% inode=29%): [14:22:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.555664/1.95, alarm hl:tmp_free=37590M/100M, alarm hl:np_load_avg=0.601562/2.0, alarm hl:mem_free=325.000000M/350M, alarm hl:available=1/0 [14:23:28] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [14:40:06] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:42:37] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:47:47] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:49:28] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1607.000000 [14:55:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:57:27] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:03:28] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.645508/1.95, alarm hl:tmp_free=37522M/100M, alarm hl:np_load_avg=0.608887/2.0, alarm hl:mem_free=283.000000M/350M, alarm hl:available=1/0 [15:04:28] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [15:12:17] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 184742 [15:13:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.575684/1.95, alarm hl:tmp_free=37503M/100M, alarm hl:np_load_avg=0.532715/2.0, alarm hl:mem_free=296.000000M/350M, alarm hl:available=1/0 [15:19:02] Magnus Manske * Re: [Toolserver-l] Toolserver user phe account expired [15:20:27] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 249049 MB (4% inode=29%): [15:36:44] 3(created) [ACCAPP-516] account for hosting of formula drawing applets used by wikipedia-de chemistry dept.; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-516> (Ralf Stephan) [15:37:27] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41923 MB (10% inode=99%): [15:39:27] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45724 MB (11% inode=99%): [15:40:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:42:28] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42524 MB (10% inode=99%): [15:42:37] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:45:56] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [15:47:47] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:51:17] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 551.000000 [15:55:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:55:28] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 54061 MB (13% inode=99%): [15:57:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:01:37] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.578613/1.95, alarm hl:tmp_free=37407M/100M, alarm hl:np_load_avg=0.553223/2.0, alarm hl:mem_free=304.000000M/350M, alarm hl:available=1/0 [16:10:27] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43435 MB (10% inode=99%): [16:11:28] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 61639 MB (15% inode=99%): [16:12:27] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 185569 [16:13:37] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [16:15:02] Lars Aronsson * Re: [Toolserver-l] Toolserver user phe account expired [16:19:36] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.609863/1.95, alarm hl:tmp_free=37369M/100M, alarm hl:np_load_avg=0.573731/2.0, alarm hl:mem_free=271.000000M/350M, alarm hl:available=1/0 [16:20:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 247204 MB (4% inode=29%): [16:40:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:43:22] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:48:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:55:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:58:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:03:57] [[Special:Log/newusers]] create 10 * Nonchalantelite * (New user account) [17:08:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:12:32] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 186144 [17:18:02] DaB. * Re: [Toolserver-l] Toolserver user phe account expired [17:18:22] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:20:31] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 247047 MB (4% inode=29%): [17:40:36] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:43:26] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [17:49:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:51:26] /sql on cassia is WARNING: DISK WARNING - free space: /sql 123891 MB (10% inode=99%): [17:55:36] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:59:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:10:27] Load avg. on adenia is WARNING: WARNING - load average: 16.01, 11.10, 6.58 [18:11:26] Load avg. on adenia is OK: OK - load average: 13.61, 11.33, 6.95 [18:12:46] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 186660 [18:16:06] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1925.000000 [18:17:02] Magnus Manske * Re: [Toolserver-l] Toolserver user phe account expired [18:21:06] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 246841 MB (4% inode=29%): [18:23:06] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.488770/1.95, alarm hl:tmp_free=36897M/100M, alarm hl:np_load_avg=0.529297/2.0, alarm hl:mem_free=255.000000M/350M, alarm hl:available=1/0 [18:26:06] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1789.000000 [18:26:07] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:40:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:43:26] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:49:00] [[Special:Log/newusers]] create 10 * Arubares7 * (New user account) [18:49:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:55:47] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:59:07] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:02:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.867676/1.95, alarm hl:tmp_free=36790M/100M, alarm hl:np_load_avg=0.727051/2.0, alarm hl:mem_free=318.000000M/350M, alarm hl:available=1/0 [19:03:17] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [19:12:47] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 187865 [19:13:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.598145/1.95, alarm hl:tmp_free=36732M/100M, alarm hl:np_load_avg=1.036133/2.0, alarm hl:mem_free=282.000000M/350M, alarm hl:available=1/0 [19:21:17] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 246685 MB (4% inode=29%): [19:27:49] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1847 [19:28:07] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1852.000000 [19:28:49] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:33:18] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:40:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:43:06] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [19:43:17] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2154.000000 [19:43:27] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [19:49:36] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:51:27] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:51:27] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:27] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:27] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:36] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:51:36] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:51:56] SMF on z-dat-s4-a is OK: OK - all services online [19:52:06] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:52:07] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 2731 MB (99% inode=99%): [19:52:18] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:52:18] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:52:18] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [19:55:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:56:02] Marcin Cieslak * Re: [Toolserver-l] Toolserver user phe account expired [19:59:16] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:11:16] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196395 MB (20% inode=99%): [20:13:06] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 189741 [20:21:17] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 246513 MB (4% inode=29%): [20:26:17] /sql on thyme is OK: DISK OK - free space: /sql 234904 MB (24% inode=99%): [20:27:57] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3172 [20:36:43] 3(created) [MAGNUS-325] "Upload at Commons" button should pass "&uselang=fromwikimedia" value to upload form; Magnus' tools: CommonsHelper; Critical Bug <10https://jira.toolserver.org/browse/MAGNUS-325> (Magog the Ogre) [20:38:59] Environment IPMI on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:39:49] Environment IPMI on adenia is OK: ok: temperature ok fan ok voltage ok chassis ok [20:40:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:42:44] 3(updated) [MAGNUS-325] "Upload at Commons" button should pass "&uselang=fromwikimedia" value to upload form <10https://jira.toolserver.org/browse/MAGNUS-325> (Magog the Ogre) [20:43:29] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3183.000000 [20:43:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:49:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:55:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:58:29] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.941406/1.10, alarm hl:np_load_long=0.735352/1.55, alarm hl:mem_free=9856.000000M/500M, alarm hl:tmp_free=13562M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.941406/1.00, alarm hl:np_load_long=0.735352/1.50, alarm hl:mem_free=9856.000000M/600M, alarm hl:tmp_free=1 [20:59:30] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:59:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:13:09] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 191930 [21:21:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 246310 MB (4% inode=29%): [21:27:59] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3270 [21:36:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.645996/1.95, alarm hl:tmp_free=36359M/100M, alarm hl:np_load_avg=0.656250/2.0, alarm hl:mem_free=286.000000M/350M, alarm hl:available=1/0 [21:37:29] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [21:41:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:42:29] /sql on thyme is WARNING: DISK WARNING - free space: /sql 193538 MB (20% inode=99%): [21:43:29] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3582.000000 [21:43:59] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3617 [21:44:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3641.000000 [21:44:29] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:49:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:51:29] /sql on thyme is OK: DISK OK - free space: /sql 214092 MB (22% inode=99%): [21:55:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:59:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:01:02] Platonides * Re: [Toolserver-l] Toolserver user phe account expired [22:05:02] Platonides * Re: [Toolserver-l] Toolserver user phe account expired [22:11:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.666992/1.95, alarm hl:tmp_free=36116M/100M, alarm hl:np_load_avg=0.613281/2.0, alarm hl:mem_free=218.000000M/350M, alarm hl:available=1/0 [22:13:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 193996 [22:14:39] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:18:49] 3(created) [ENWPONE-29] Web tool appears to list duplicate projects for individual names, and many projects pages don't actually appear to exist in the toolserver db; Wikipedia Release Version Tools; Bug <10https://jira.toolserver.org/browse/ENWPONE-29> (Michael Gilbert) [22:21:39] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 246271 MB (4% inode=29%): [22:41:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:42:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.455078/1.95, alarm hl:tmp_free=35870M/100M, alarm hl:np_load_avg=0.477051/2.0, alarm hl:mem_free=338.000000M/350M, alarm hl:available=1/0 [22:43:59] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4036 [22:44:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4033.000000 [22:44:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:47:39] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:49:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:56:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:59:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:13:18] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 194549 [23:21:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 246006 MB (4% inode=29%): [23:41:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:44:17] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3805 [23:44:37] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3825.000000 [23:44:57] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [23:50:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:56:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:58:36] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3600.000000 [23:59:17] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3591