[00:00:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:06:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:07:22] /sql on thyme is WARNING: DISK WARNING - free space: /sql 192625 MB (20% inode=99%): [00:07:44] 3(created) [CHTWO-44] Template:Self should add the author= field if none is already present; Commonshelper2: Transfer (XML), Upload Bot; Critical New Feature <10https://jira.toolserver.org/browse/CHTWO-44> (Magog the Ogre) [00:07:52] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23502 [00:09:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23566.000000 [00:09:52] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:10:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126127 MB (20% inode=99%): [00:19:01] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37310 [00:27:05] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 237283 MB (4% inode=28%): [00:30:23] /sql on thyme is OK: DISK OK - free space: /sql 228517 MB (23% inode=99%): [00:46:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:49:32] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:51:52] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [00:56:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:58:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.459473/1.95, alarm hl:tmp_free=46843M/100M, alarm hl:np_load_avg=0.472168/2.0, alarm hl:mem_free=295.000000M/350M, alarm hl:available=1/0 [01:00:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:01:52] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [01:06:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:07:53] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25665 [01:09:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25731.000000 [01:10:21] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126059 MB (20% inode=99%): [01:19:02] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31676 [01:28:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 234859 MB (4% inode=28%): [01:47:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:52:02] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [01:54:42] /sql on thyme is WARNING: DISK WARNING - free space: /sql 189802 MB (19% inode=99%): [01:57:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:01:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:06:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:08:01] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27379 [02:09:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27461.000000 [02:10:22] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126111 MB (20% inode=99%): [02:12:42] /sql on thyme is OK: DISK OK - free space: /sql 231405 MB (24% inode=99%): [02:19:13] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15789 [02:28:12] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 235632 MB (4% inode=28%): [02:47:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:51:12] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3358 [02:52:12] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [02:57:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:00:12] MySQL slave on z-dat-s6-a is OK: Uptime: 329871 Threads: 13 Questions: 75889550 Slow queries: 11466 Opens: 494798 Flush tables: 1 Open tables: 2193 Queries per second avg: 230.58 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1597 [03:01:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:04:54] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42569 MB (10% inode=99%): [03:07:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:08:02] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28453 [03:10:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28483.000000 [03:10:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 126059 MB (20% inode=99%): [03:18:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 59217 MB (14% inode=99%): [03:28:28] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 235185 MB (4% inode=28%): [03:45:22] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [03:45:53] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 223.000000 [03:47:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:52:12] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [03:58:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:01:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:07:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:08:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29663 [04:10:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29705.000000 [04:10:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125992 MB (20% inode=99%): [04:12:22] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.395996/1.10, alarm hl:np_load_long=0.403320/1.55, alarm hl:mem_free=236.000000M/500M, alarm hl:tmp_free=14174M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.395996/1.00, alarm hl:np_load_long=0.403320/1.50, alarm hl:mem_free=236.000000M/600M, alarm hl:tmp_free= [04:17:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.564453/1.95, alarm hl:tmp_free=46516M/100M, alarm hl:np_load_avg=0.675293/2.0, alarm hl:mem_free=337.000000M/350M, alarm hl:available=1/0 [04:18:22] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:19:32] [[Special:Log/newusers]] create 10 * Btiuser * (New user account) [04:25:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.704590/1.95, alarm hl:tmp_free=46505M/100M, alarm hl:np_load_avg=0.621582/2.0, alarm hl:mem_free=243.000000M/350M, alarm hl:available=1/0 [04:28:34] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 235150 MB (4% inode=28%): [04:46:22] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [04:47:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:52:22] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [04:58:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:01:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:07:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:08:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31684 [05:10:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31723.000000 [05:10:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125898 MB (20% inode=99%): [05:28:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 235106 MB (4% inode=28%): [05:47:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:52:11] /sql on cassia is WARNING: DISK WARNING - free space: /sql 119247 MB (10% inode=99%): [05:52:22] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [05:58:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:01:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:08:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:08:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33292 [06:10:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33345.000000 [06:10:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125761 MB (20% inode=99%): [06:13:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.388184/1.95, alarm hl:tmp_free=46165M/100M, alarm hl:np_load_avg=1.149902/2.0, alarm hl:mem_free=334.000000M/350M, alarm hl:available=1/0 [06:14:22] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [06:29:31] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 234986 MB (4% inode=28%): [06:45:53] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [06:46:12] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 251.000000 [06:47:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:52:22] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [06:58:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:01:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:08:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34683 [07:09:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:10:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34716.000000 [07:10:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125609 MB (20% inode=99%): [07:26:02] /sql on thyme is WARNING: DISK WARNING - free space: /sql 198539 MB (20% inode=99%): [07:29:32] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 234878 MB (4% inode=28%): [07:29:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:34:02] /sql on thyme is OK: DISK OK - free space: /sql 226056 MB (23% inode=99%): [07:34:22] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:47:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:52:23] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [07:58:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:01:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:08:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36098 [08:09:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:10:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36129.000000 [08:10:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125417 MB (20% inode=99%): [08:12:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.363770/1.95, alarm hl:tmp_free=45744M/100M, alarm hl:np_load_avg=1.038574/2.0, alarm hl:mem_free=213.000000M/350M, alarm hl:available=1/0 [08:14:23] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [08:29:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 233849 MB (4% inode=28%): [08:42:07] I've got a new request at https://jira.toolserver.org/browse/ACCAPP-502 that seems to be languishing. have i done something stupid to hold things up? of do i just need to be patient? [08:47:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:52:23] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [08:57:53] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:58:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:01:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:08:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36878 [09:09:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:10:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36884.000000 [09:11:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 125173 MB (20% inode=99%): [09:12:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.923828/1.95, alarm hl:tmp_free=45645M/100M, alarm hl:np_load_avg=0.774902/2.0, alarm hl:mem_free=308.000000M/350M, alarm hl:available=1/0 [09:13:23] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.002930/1.00, alarm hl:np_load_long=0.692383/1.50, alarm hl:mem_free=9404.000000M/600M, alarm hl:tmp_free=14632M/100M, alarm hl:available=1/0 [09:13:32] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:14:24] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:29:52] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 233669 MB (4% inode=28%): [09:41:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.691406/1.95, alarm hl:tmp_free=45460M/100M, alarm hl:np_load_avg=0.582519/2.0, alarm hl:mem_free=331.000000M/350M, alarm hl:available=1/0 [09:47:33] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [09:48:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:52:33] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [09:58:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:58:54] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:01:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:09:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37960 [10:09:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:11:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124903 MB (20% inode=99%): [10:11:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37960.000000 [10:13:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.638184/1.95, alarm hl:tmp_free=45346M/100M, alarm hl:np_load_avg=0.636719/2.0, alarm hl:mem_free=330.000000M/350M, alarm hl:available=1/0 [10:14:33] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [10:29:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:29:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 233380 MB (4% inode=28%): [10:34:23] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:48:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:52:42] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [10:56:32] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.367676/1.10, alarm hl:np_load_long=0.272461/1.55, alarm hl:mem_free=399.000000M/500M, alarm hl:tmp_free=13684M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.367676/1.00, alarm hl:np_load_long=0.272461/1.50, alarm hl:mem_free=399.000000M/600M, alarm hl:tmp_free= [10:57:33] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [10:58:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:59:52] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:01:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:07:33] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.466309/1.10, alarm hl:np_load_long=0.402344/1.55, alarm hl:mem_free=230.000000M/500M, alarm hl:tmp_free=13638M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.466309/1.00, alarm hl:np_load_long=0.402344/1.50, alarm hl:mem_free=230.000000M/600M, alarm hl:tmp_free= [11:09:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38437 [11:09:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:11:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124662 MB (20% inode=99%): [11:11:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38478.000000 [11:14:00] [[User:Shanmugamp7]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7294&rcid=9749 * Shanmugamp7 * (+65) (Created page with "[[wikipedia:user:shanmugamp7]] [[wikipedia:ta:user:shanmugamp7]]") [11:26:23] omg Username :p [11:27:07] how is that possible [11:27:14] I cannot access the toolserver wiki :/ [11:27:16] * ToAruShiroiNeko cries [11:29:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 235079 MB (4% inode=28%): [11:32:22] [[Special:Log/newusers]] create 10 * Ercolaniorval881 * (New user account) [11:43:33] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.841797/1.10, alarm hl:np_load_long=0.933594/1.55, alarm hl:mem_free=9350.000000M/500M, alarm hl:tmp_free=14392M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.841797/1.00, alarm hl:np_load_long=0.933594/1.50, alarm hl:mem_free=9350.000000M/600M, alarm hl:tmp_free=1 [11:45:33] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:46:33] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.428711/1.00, alarm hl:np_load_long=0.467774/1.50, alarm hl:mem_free=600.000000M/600M, alarm hl:tmp_free=13573M/100M, alarm hl:available=1/0 [11:47:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.379395/1.95, alarm hl:tmp_free=44992M/100M, alarm hl:np_load_avg=0.449219/2.0, alarm hl:mem_free=302.000000M/350M, alarm hl:available=1/0 [11:48:32] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [11:48:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:52:43] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [11:52:43] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [11:58:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:59:52] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:01:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.322266/1.10, alarm hl:np_load_long=0.892578/1.55, alarm hl:mem_free=9318.000000M/500M, alarm hl:tmp_free=14382M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.322266/1.00, alarm hl:np_load_long=0.892578/1.50, alarm hl:mem_free=9318.000000M/600M, alarm hl:tmp_free=1 [12:01:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:08:23] hello all [12:09:51] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39256 [12:09:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:11:19] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124438 MB (20% inode=99%): [12:11:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39281.000000 [12:28:45] 3(commented) [TS-1378] Create MMP 'ac' <10https://jira.toolserver.org/browse/TS-1378> (DaB.) [12:30:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 231505 MB (4% inode=27%): [12:30:44] 3(assigned) [TS-1387] Multi-maintainer project for Commons MIME type statistics <10https://jira.toolserver.org/browse/TS-1387> (DaB.) [12:35:05] [[Admin:Subversion]] 10https://wiki.toolserver.org/w/index.php?diff=7295&oldid=4563&rcid=9751 * Dab * (+47) (/* Creating a new repository */ I never can remember this) [12:39:44] 3(resolved) [TS-1387] Multi-maintainer project for Commons MIME type statistics <10https://jira.toolserver.org/browse/TS-1387> (DaB.) [12:39:45] 3(resolved) [TS-1386] Account expiration <10https://jira.toolserver.org/browse/TS-1386> (DaB.) [12:39:46] 3(assigned) [TS-1384] Add wikimania2013 wiki to database <10https://jira.toolserver.org/browse/TS-1384> (DaB.) [12:40:00] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:46:09] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [12:47:30] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 365.000000 [12:49:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:49:43] 3(resolved) [TS-1384] Add wikimania2013 wiki to database <10https://jira.toolserver.org/browse/TS-1384> (DaB.) [12:51:44] 3(resolved) [TS-1383] Add snowolf to stewardbots MMT <10https://jira.toolserver.org/browse/TS-1383> (DaB.) [12:53:00] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [12:54:43] 3(resolved) [TS-866] Account expired and can not login <10https://jira.toolserver.org/browse/TS-866> (DaB.) [12:56:43] 3(resolved) [TS-1385] S6 replag issues <10https://jira.toolserver.org/browse/TS-1385> (DaB.) [12:59:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:00:10] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:02:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:09:47] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [13:09:57] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40003 [13:10:17] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:11:37] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39992.000000 [13:12:17] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124225 MB (20% inode=99%): [13:19:43] @replag [13:19:43] Dispenser: s1-user: 11h 5m 18s [+0.19 s/s]; s2-user: 17m 12s [-0.21 s/s]; s3-rr-a: 11s [-0.01 s/s]; s3-user: 11s [-0.01 s/s] [13:30:34] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229579 MB (4% inode=27%): [13:45:22] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.135742/1.10, alarm hl:np_load_long=0.615235/1.55, alarm hl:mem_free=9682.000000M/500M, alarm hl:tmp_free=14237M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.135742/1.00, alarm hl:np_load_long=0.615235/1.50, alarm hl:mem_free=9682.000000M/600M, alarm hl:tmp_free=1 [13:46:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:49:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:53:21] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [13:59:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:00:21] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:02:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:09:59] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39641 [14:10:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:11:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39652.000000 [14:12:30] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 124052 MB (20% inode=99%): [14:30:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229396 MB (4% inode=27%): [14:37:22] Just curious, I submitted an account request two weeks ago, and have gotten zero updates on it. Is it normal to take this long? I know the page says to bug someone after three weeks, but is two considered to be a normal amount of time to wait? [14:49:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:53:30] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [14:59:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:00:40] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:02:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:10:10] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39366 [15:10:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:12:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39358.000000 [15:12:39] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123733 MB (20% inode=99%): [15:30:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229268 MB (4% inode=27%): [15:49:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:53:39] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [15:59:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:00:40] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:02:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:07:31] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.506836/1.10, alarm hl:np_load_long=0.693360/1.55, alarm hl:mem_free=9317.000000M/500M, alarm hl:tmp_free=14072M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.506836/1.00, alarm hl:np_load_long=0.693360/1.50, alarm hl:mem_free=9317.000000M/600M, alarm hl:tmp_free=1 [16:09:31] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [16:10:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39284 [16:10:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:12:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39280.000000 [16:12:49] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123405 MB (20% inode=99%): [16:14:11] [[Special:Log/newusers]] create 10 * Avaya3000 * (New user account) [16:31:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 235438 MB (4% inode=28%): [16:45:19] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [16:46:09] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 774.000000 [16:49:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:54:00] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [16:56:43] 3(commented) [TS-1378] Create MMP 'ac' <10https://jira.toolserver.org/browse/TS-1378> (Christian Thiele) [17:00:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:01:00] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:03:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:10:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:10:49] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39614 [17:12:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39595.000000 [17:13:48] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 123175 MB (20% inode=99%): [17:31:52] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 232815 MB (4% inode=28%): [17:50:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:52:17] /sql on cassia is WARNING: DISK WARNING - free space: /sql 120492 MB (10% inode=99%): [17:54:05] [[Avaya 3000 exam]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7296&rcid=9753 * Avaya3000 * (+224) (Created page with "ExamKiller Avaya 3000 exam , The Avaya 3000 test contains real Avaya 3000 Certification Questions and Answers, Money...") [17:54:48] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:01:26] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:01:47] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:03:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:11:07] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39003 [18:11:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.630859/1.95, alarm hl:tmp_free=44193M/100M, alarm hl:np_load_avg=0.584473/2.0, alarm hl:mem_free=190.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.630859/2.3, alarm hl:np_load_long=0.508789/2.5, alarm hl:cpu=46.200000/98, alarm hl:mem_free=190.000000M/200M, al [18:11:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:13:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39009.000000 [18:13:27] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [18:14:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122960 MB (20% inode=99%): [18:22:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.514160/1.95, alarm hl:tmp_free=44166M/100M, alarm hl:np_load_avg=0.543457/2.0, alarm hl:mem_free=246.000000M/350M, alarm hl:available=1/0 [18:31:57] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 230800 MB (4% inode=27%): [18:51:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:55:07] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [18:55:42] 3(created) [MNT-1243] Created myql-user tsdbsyncer with super-privilege; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1243> (DaB.) [18:55:47] 3(resolved) [MNT-1243] Created myql-user tsdbsyncer with super-privilege <10https://jira.toolserver.org/browse/MNT-1243> (DaB.) [19:01:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:02:08] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:04:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:11:07] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38844 [19:11:46] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:13:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38841.000000 [19:14:47] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122701 MB (20% inode=99%): [19:32:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229274 MB (4% inode=27%): [19:44:42] 3(created) [OSM-17] pg_dump is too old; OSM; Bug <10https://jira.toolserver.org/browse/OSM-17> (Kolossos ) [19:44:49] 3(assigned) [OSM-17] pg_dump is too old <10https://jira.toolserver.org/browse/OSM-17> (Kolossos ) [19:46:42] 3(assigned) [OSM-17] pg_dump is too old <10https://jira.toolserver.org/browse/OSM-17> (Kolossos ) [19:51:17] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:55:17] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:01:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:02:17] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:02:17] what server has s4 commons? [20:04:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:04:21] Im getting a ERROR 1 (HY000) at line 1: server name: 'sql_s4' doesn't exist! [20:08:14] sql-s4 ? [20:10:47] Nettrom: I can connect to the DB fine, using sql [20:10:58] however if I do: [20:11:17] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38251 [20:11:21] sql dbname $HOME/public_html/results.txt [20:11:29] I get the error [20:12:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:12:50] hm, I just tested that and it worked just fine [20:12:56] is there an error in our query? [20:13:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38240.000000 [20:15:07] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122517 MB (20% inode=99%): [20:17:23] Nettrom: no, I think I figured out the issue [20:17:35] the db im connecting to is on s3 and commons is on s4 [20:17:49] ah, I see [20:18:10] I was using "sql commonswiki_p", that explains it [20:18:40] @replag all [20:18:40] Betacommand: s1-rr-a: 4s [-0.14 s/s]; s1-user: 10h 36m 58s [-0.13 s/s]; s2-user: 8m 32s [+0.17 s/s]; s2-user-c: 4m 22s [+0.00 s/s]; s3-rr-a: 1m 8s [+0.01 s/s]; s3-user: 1m 8s [+0.01 s/s]; s4-rr-a: 1s [-]; s4-user: 1s [-] [20:18:41] Betacommand: s5-rr-a: 3s [+0.00 s/s]; s5-user: 3s [+0.00 s/s]; s5-user-c: 4m 22s [+0.00 s/s]; s6-rr-a: 4s [+0.00 s/s]; s6-user: 4s [+0.00 s/s]; s7-rr-a: 2s [-0.00 s/s]; s7-user: 2s [-0.00 s/s] [20:22:27] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.456055/1.95, alarm hl:tmp_free=43546M/100M, alarm hl:np_load_avg=0.422851/2.0, alarm hl:mem_free=338.000000M/350M, alarm hl:available=1/0 [20:26:28] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [20:31:31] is commons on the s3 cluster? [20:31:53] it's replicated to all clusters [20:32:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229043 MB (4% inode=27%): [20:33:18] Platonides: then why am I getting an error when running http://pastebin.com/33PPYNBe on s3 [20:33:42] ERROR 1 (HY000): server name: 'sql_s4' doesn't exist! [20:34:53] how are you running it? [20:35:04] I think that _ should be a - [20:35:08] sql simplewiki_p $HOME/public_html/simpleredlinks.txt [20:39:22] Platonides: thoughts? [20:40:35] I suspect the view points to a foreign server [20:40:38] which is missing [20:41:46] mysql> EXPLAIN commonswiki_p.page; [20:41:46] ERROR 1 (HY000): server name: 'sql_s4' doesn't exist! [20:42:26] seems ok from other servers [20:42:49] it may be wrongly configured on sql-s3 [20:43:25] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.040039/1.00, alarm hl:np_load_long=0.698242/1.50, alarm hl:mem_free=9165.000000M/600M, alarm hl:tmp_free=13716M/100M, alarm hl:available=1/0 [20:44:25] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:46:44] 3(created) [TS-1388] s3 and commonswiki_p mis-configured; Toolserver; Bug <10https://jira.toolserver.org/browse/TS-1388> (Betacommand) [20:46:46] 3(assigned) [TS-1388] s3 and commonswiki_p mis-configured <10https://jira.toolserver.org/browse/TS-1388> (Betacommand) [20:50:44] Betacommand: you plan to fix it? ;) [20:51:07] oh, it was assignted to me… [20:51:10] DaBPunkt: I cant [20:51:34] :) [20:51:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:51:48] river is the default for the TS project [20:51:54] however he is gone [20:53:55] AFAIR no-one is the default [20:55:25] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [20:56:15] i got Lead: River Tarnell mixed up [20:57:09] Project Team: [20:57:10] Project Lead: River Tarnell [20:57:12] Default Assignee: Unassigned [21:02:34] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:02:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:04:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:09:34] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:10:44] 3(resolved) [TS-1388] s3 and commonswiki_p mis-configured <10https://jira.toolserver.org/browse/TS-1388> (DaB.) [21:11:25] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37488 [21:12:11] DaBPunkt: thanks [21:12:15] np [21:12:25] you're always so efficient, DaB :) [21:12:39] I wish I would be… [21:12:48] it seems I always find the bugs [21:13:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:13:10] maybe the other people don't really use the toolserver :) [21:13:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37463.000000 [21:14:25] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1963.000000 [21:15:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122213 MB (20% inode=99%): [21:16:35] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:35] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:35] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:35] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:35] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:35] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:35] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:36] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:36] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:37] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:16:54] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:54] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:54] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:54] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:54] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:54] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:54] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:55] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:55] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:56] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:56] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:57] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:17:14] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:17:24] / on z-dat-s3-a is OK: DISK OK - free space: / 8289 MB (27% inode=85%): [21:17:24] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 142936 MB (14% inode=99%): [21:17:34] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:17:34] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:17:34] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:17:43] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [21:17:44] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [21:17:44] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [21:17:44] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [21:17:45] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [21:17:45] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [21:17:45] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [21:17:45] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [21:18:14] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [21:19:24] Load avg. on z-dat-s3-a is OK: OK - load average: 0.16, 0.91, 1.57 [21:19:24] MySQL on z-dat-s4-a is OK: Uptime: 2336322 Threads: 18 Questions: 233704314 Slow queries: 35503 Opens: 35223 Flush tables: 1 Open tables: 941 Queries per second avg: 100.30 [21:19:24] MySQL slave on z-dat-s4-a is OK: Uptime: 2336322 Threads: 18 Questions: 233704313 Slow queries: 35503 Opens: 35223 Flush tables: 1 Open tables: 941 Queries per second avg: 100.30 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 304 [21:19:24] MySQL on z-dat-s6-a is OK: Uptime: 395822 Threads: 17 Questions: 85583938 Slow queries: 16564 Opens: 578834 Flush tables: 1 Open tables: 2331 Queries per second avg: 216.218 [21:19:24] MySQL slave on z-dat-s6-a is OK: Uptime: 395822 Threads: 17 Questions: 85583938 Slow queries: 16564 Opens: 578834 Flush tables: 1 Open tables: 2331 Queries per second avg: 216.218 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 323 [21:19:25] SMTP on z-dat-s7-a is OK: SMTP OK - 0.003 sec. response time [21:19:25] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:19:26] SMTP on z-dat-s3-a is OK: SMTP OK - 0.004 sec. response time [21:19:26] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [21:19:27] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 90198 MB (22% inode=99%): [21:19:27] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 361 MB (99% inode=99%): [21:19:28] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 74792 MB (18% inode=99%): [21:19:28] SMTP on hyacinth is OK: SMTP OK - 0.077 sec. response time [21:19:29] SMF on hyacinth is OK: OK - all services online [21:20:04] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 367 MB (99% inode=99%): [21:20:04] Load avg. on z-dat-s6-a is OK: OK - load average: 1.02, 1.03, 1.58 [21:20:04] SMTP on z-dat-s4-a is OK: SMTP OK - 0.137 sec. response time [21:31:45] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:33:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 232428 MB (4% inode=27%): [21:41:15] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [21:45:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3638.000000 [21:51:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:55:46] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [21:56:34] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:57:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:57:05] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:02:55] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:04:47] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:08:14] nacht ts [22:11:47] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37469 [22:13:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:13:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37468.000000 [22:15:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121989 MB (20% inode=99%): [22:21:46] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1647.000000 [22:22:05] MySQL slave on z-dat-s4-a is OK: Uptime: 1639 Threads: 5 Questions: 1805 Slow queries: 0 Opens: 54 Flush tables: 1 Open tables: 44 Queries per second avg: 1.101 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1614 [22:22:05] MySQL on z-dat-s4-a is OK: Uptime: 1639 Threads: 4 Questions: 1808 Slow queries: 0 Opens: 54 Flush tables: 1 Open tables: 44 Queries per second avg: 1.103 [22:22:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.416504/1.95, alarm hl:tmp_free=42775M/100M, alarm hl:np_load_avg=0.458496/2.0, alarm hl:mem_free=241.000000M/350M, alarm hl:available=1/0 [22:25:46] 3(created) [TS-1389] SQL-S4-USER not connecting; Toolserver: Databases; Bug <10https://jira.toolserver.org/browse/TS-1389> (TParis) [22:26:03] ^^ Not sure if I put that in the right project/queue [22:26:55] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:27:46] 3(commented) [TS-1389] SQL-S4-USER not connecting <10https://jira.toolserver.org/browse/TS-1389> (TParis) [22:29:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.411621/1.95, alarm hl:tmp_free=42748M/100M, alarm hl:np_load_avg=0.414062/2.0, alarm hl:mem_free=215.000000M/350M, alarm hl:available=1/0 [22:33:56] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 229660 MB (4% inode=27%): [22:45:56] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6769.000000 [22:51:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:55:56] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs [22:57:44] 3(commented) [DBQ-184] I would like to know edit history(name of editor, page title and # of edits) for all pages in a category and related pages over a defined time period <10https://jira.toolserver.org/browse/DBQ-184> (Hoo man) [23:03:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:04:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:11:56] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37521 [23:13:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:14:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37524.000000 [23:15:35] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121804 MB (20% inode=99%): [23:26:42] is there something i need to do to progress https://jira.toolserver.org/browse/ACCAPP-502 or is it just a question of waiting? [23:29:10] snail: need to wait till dab reviews that queue [23:30:27] Betacommand: ok, that's fine. i just noticed that some new account requests submitted after mine had been actioned and wondered if i'd made a stupid mistake of some description or another [23:30:51] snail: No sometime DaB just misses things [23:33:56] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 226992 MB (4% inode=27%): [23:45:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9771.000000 [23:52:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:56:09] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d42 of mirror d40 is Needs and submirror d32 of mirror d30 is Needs and submirror d22 of mirror d20 is Needs and submirror d12 of mirror d10 is Needs