[00:00:28] Load avg. on willow is CRITICAL: CRITICAL - load average: 44.55, 23.50, 19.11 [00:02:28] Load avg. on willow is WARNING: WARNING - load average: 22.65, 22.73, 19.43 [00:05:27] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.05, 25.25, 20.83 [00:07:18] MySQL slave on thyme is WARNING: No slaves defined [00:22:28] Load avg. on willow is WARNING: WARNING - load average: 14.34, 17.79, 19.91 [00:30:29] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.21, 20.03, 19.41 [00:31:28] Load avg. on willow is WARNING: WARNING - load average: 22.61, 19.38, 19.21 [00:33:07] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:35:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [00:36:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 876072.000000 [00:45:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:45:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:45:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:49:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.700195/1.75, alarm hl:np_load_avg=1.960449/2.0, alarm hl:mem_free=285.000000M/350M, alarm hl:available=1/0 [00:52:47] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:53:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:58:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:07:28] MySQL slave on thyme is WARNING: No slaves defined [01:10:27] Load avg. on willow is CRITICAL: CRITICAL - load average: 38.74, 25.61, 21.39 [01:35:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [01:36:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 879678.000000 [01:45:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:45:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:45:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:50:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.060547/1.75, alarm hl:np_load_avg=2.879883/2.0, alarm hl:mem_free=227.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.060547/1.9, alarm hl:np_load_long=2.800293/2.25, alarm hl:mem_free=227.000000M/200M, alarm hl:available=1/0 [01:53:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:58:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:07:38] MySQL slave on thyme is WARNING: No slaves defined [02:10:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 42.62, 28.51, 25.05 [02:35:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [02:36:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 883291.000000 [02:45:26] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:45:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:45:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:51:07] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.425293/1.75, alarm hl:np_load_avg=3.157715/2.0, alarm hl:mem_free=245.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.425293/1.9, alarm hl:np_load_long=3.154297/2.25, alarm hl:mem_free=245.000000M/200M, alarm hl:available=1/0 [02:54:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:58:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:00:27] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 127294 MB (13% inode=99%): [03:07:51] MySQL slave on thyme is WARNING: No slaves defined [03:10:48] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.16, 27.99, 26.15 [03:23:54] 3(created) [ACCAPP-485] UTRS developer; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-485> (Chris Howie) [03:35:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [03:36:51] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 886895.000000 [03:45:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:46:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:46:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:51:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.937500/1.75, alarm hl:np_load_avg=3.172852/2.0, alarm hl:mem_free=289.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.937500/1.9, alarm hl:np_load_long=3.249023/2.25, alarm hl:mem_free=289.000000M/200M, alarm hl:available=1/0 [03:54:19] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:59:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:08:50] MySQL slave on thyme is WARNING: No slaves defined [04:10:48] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.01, 27.30, 26.01 [04:36:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [04:37:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 890512.000000 [04:42:07] Load avg. on willow is WARNING: WARNING - load average: 15.29, 17.21, 19.82 [04:45:36] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:45:36] * matthewrbowker hopes tsnag got that wrong... unknown database enwiki_p ? Wikipedia has disappeared! ;) [04:46:22] @replag [04:46:22] Joan: s1-rr-a: 1w 3d 7h 31m 18s [+1.00 s/s]; s1-rr-a-c: 22s [+0.00 s/s]; s1-user: 1w 3d 7h 31m 18s [+1.00 s/s]; s2-user: 10s [+0.00 s/s]; s2-user-c: 22s [+0.00 s/s]; s3-rr-a: 1m 31s [+0.00 s/s]; s3-user: 1m 31s [+0.00 s/s]; s4-rr-a: 22s [+0.00 s/s] [04:46:23] Joan: s4-user: 22s [+0.00 s/s]; s5-user-c: 22s [+0.00 s/s]; s7-rr-a: 11s [+0.00 s/s]; s7-user: 11s [+0.00 s/s] [04:47:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:47:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:51:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.524902/1.75, alarm hl:np_load_avg=2.453125/2.0, alarm hl:mem_free=812.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.524902/1.9, alarm hl:np_load_long=2.486328/2.25, alarm hl:mem_free=812.000000M/200M, alarm hl:available=1/0 [04:55:15] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:00:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:01:06] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.06, 21.86, 19.70 [05:02:07] Load avg. on willow is WARNING: WARNING - load average: 21.97, 20.99, 19.53 [05:05:07] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.07, 21.82, 20.11 [05:09:06] MySQL slave on thyme is WARNING: No slaves defined [05:36:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [05:38:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 894175.000000 [05:45:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:47:17] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:47:17] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:51:25] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.353027/1.75, alarm hl:np_load_avg=3.245605/2.0, alarm hl:mem_free=346.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.353027/1.9, alarm hl:np_load_long=3.166016/2.25, alarm hl:mem_free=346.000000M/200M, alarm hl:available=1/0 [05:55:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:00:16] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:02:54] 3(commented) [ACCAPP-477] Account for hosting Wikipedia-OpenStreetMap integration tools <10https://jira.toolserver.org/browse/ACCAPP-477> [06:06:05] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.39, 33.14, 30.06 [06:09:06] MySQL slave on thyme is WARNING: No slaves defined [06:36:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [06:38:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 897778.000000 [06:45:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:47:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:47:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:51:37] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.218262/1.75, alarm hl:np_load_avg=3.343750/2.0, alarm hl:mem_free=324.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.218262/1.9, alarm hl:np_load_long=3.495605/2.25, alarm hl:mem_free=324.000000M/200M, alarm hl:available=1/0 [06:55:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:00:16] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:06:17] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.80, 27.11, 26.97 [07:09:18] MySQL slave on thyme is WARNING: No slaves defined [07:30:18] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.013672/1.00, alarm hl:np_load_long=0.797852/1.50, alarm hl:mem_free=19941.000000M/350M, alarm hl:available=1/0 [07:33:17] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:36:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [07:38:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 901385.000000 [07:43:25] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.718750/1.10, alarm hl:np_load_long=0.982422/1.55, alarm hl:mem_free=20095.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.718750/1.00, alarm hl:np_load_long=0.982422/1.50, alarm hl:mem_free=20095.000000M/350M, alarm hl:available=1/0 [07:45:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:47:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:47:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:49:16] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1947 [07:51:46] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.131348/1.75, alarm hl:np_load_avg=3.298828/2.0, alarm hl:mem_free=195.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.131348/1.9, alarm hl:np_load_long=3.292969/2.25, alarm hl:mem_free=195.000000M/200M, alarm hl:available=1/0 [07:55:26] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:58:05] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:00:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:06:16] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.11, 26.47, 26.61 [08:09:17] MySQL slave on thyme is WARNING: No slaves defined [08:16:06] [[~soxred93/ec]] ! 10https://wiki.toolserver.org/w/index.php?diff=6966&oldid=5371&rcid=9172 * Ab2llah * (+17) (Expired Tool) [08:18:16] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3604 [08:36:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [08:38:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 904994.000000 [08:46:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:47:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:47:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:51:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.897949/1.75, alarm hl:np_load_avg=3.200684/2.0, alarm hl:mem_free=206.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.897949/1.9, alarm hl:np_load_long=3.062012/2.25, alarm hl:mem_free=206.000000M/200M, alarm hl:available=1/0 [08:52:45] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:55:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:00:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:06:47] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.79, 25.95, 25.26 [09:10:15] MySQL slave on thyme is WARNING: No slaves defined [09:18:09] [[~escaladix/larticles/larticles.php]] ! 10https://wiki.toolserver.org/w/index.php?diff=6967&oldid=4335&rcid=9173 * Ab2llah * (+12) () [09:19:15] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7124 [09:23:48] [[Main Page/lang]] ! 10https://wiki.toolserver.org/w/index.php?diff=6968&oldid=6935&rcid=9174 * Ab2llah * (+36) (+mzn) [09:37:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [09:38:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 908597.000000 [09:44:49] [[User:Ab2llah]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6969&rcid=9175 * Ab2llah * (+662) (Created page with "{| |style="border:solid #EFEFF1; -moz-border-radius: 10px; background:white; width:100%;"| '''Hi
My name is Me...") [09:46:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:46:51] [[User:Ab2llah/user mzn]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6970&rcid=9176 * Ab2llah * (+442) (Created page with "
{| cellspacing="0" style="width:238px;background:#C5FCDC" | style="width:45px;height:45px;background:#6EF7A7;text-ali...") [09:47:40] [[User:Ab2llah/User bot owner]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6971&rcid=9177 * Ab2llah * (+483) (Created page with "{{userbox | border-c = #808080 | id = [[Image:Crystal_Clear_action_run.svg|30x30px]]
([[{{{contribs|Specia...") [09:47:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:47:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:50:17] [[User:Ab2llah]] ! 10https://wiki.toolserver.org/w/index.php?diff=6972&oldid=6969&rcid=9178 * Ab2llah * (-28) () [09:51:47] [[User:Ab2llah/user glk-2]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6973&rcid=9179 * Ab2llah * (+489) (Created page with "
{| cellspacing="0" style="width:238px;background:#D0F8FF" | style="width:45px;height:45px;background:#77E0E8;text-ali...") [09:51:56] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.500000/1.75, alarm hl:np_load_avg=3.368164/2.0, alarm hl:mem_free=233.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.500000/1.9, alarm hl:np_load_long=3.255859/2.25, alarm hl:mem_free=233.000000M/200M, alarm hl:available=1/0 [09:53:23] [[User:Ab2llah/user fa]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6974&rcid=9180 * Ab2llah * (+413) (Created page with "
{| cellspacing="0" style="width:238px;background:#D0F8FF" | style="width:45px;height:45px;background:#77E0E8;text-ali...") [09:54:29] [[User:Ab2llah/user en-2]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6975&rcid=9181 * Ab2llah * (+517) (Created page with "
{| cellspacing="0" style="width:238px;background:#D0F8FF" | style="width:45px;height:45px;background:#77E0E8;text-ali...") [09:55:57] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:00:17] [[Main Page/lang]] ! 10https://wiki.toolserver.org/w/index.php?diff=6976&oldid=6968&rcid=9182 * Ab2llah * (+43) (+fa - plz don't remove and protect for auto confirmed users only) [10:00:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:01:40] [[صفحه اصلی]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6977&rcid=9183 * Ab2llah * (+33) (Redirected page to [[صفحهٔ اصلی]]) [10:02:59] 3(commented) [MNT-1225] Growing replag on S1 due to a database migration at WMF <10https://jira.toolserver.org/browse/MNT-1225> (Marlen Caemmerer) [10:06:47] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.14, 28.29, 27.53 [10:10:16] MySQL slave on thyme is WARNING: No slaves defined [10:19:16] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10648 [10:20:51] 3(commented) [TS-1343] s6 is currently not replicating <10https://jira.toolserver.org/browse/TS-1343> (Marlen Caemmerer) [10:23:05] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:28:05] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:37:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [10:37:46] Load avg. on willow is WARNING: WARNING - load average: 15.93, 17.72, 19.75 [10:38:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 912205.000000 [10:41:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.10, 19.57, 20.00 [10:42:46] Load avg. on willow is WARNING: WARNING - load average: 19.46, 19.34, 19.89 [10:47:09] no nosy here? [10:47:26] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:48:51] 3(commented) [TS-1343] s6 is currently not replicating <10https://jira.toolserver.org/browse/TS-1343> (Marlen Caemmerer) [10:48:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:48:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:52:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.285645/1.75, alarm hl:np_load_avg=2.422852/2.0, alarm hl:mem_free=642.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.285645/1.9, alarm hl:np_load_long=2.471680/2.25, alarm hl:mem_free=642.000000M/200M, alarm hl:available=1/0 [10:56:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:00:50] [[Toolserver:Sandbox]] ! 10https://wiki.toolserver.org/w/index.php?diff=6978&oldid=6613&rcid=9184 * 77.87.228.68 * (+15) () [11:01:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:05:46] Load avg. on willow is WARNING: WARNING - load average: 19.22, 18.18, 18.45 [11:10:16] MySQL slave on thyme is WARNING: No slaves defined [11:17:04] [[Interwiki bot MMP planning]] 10https://wiki.toolserver.org/w/index.php?diff=6979&oldid=6847&rcid=9185 * Merlissimo * (+905) (+ info about MerlIwBot) [11:19:16] MySQL slave on z-dat-s6-a is OK: Uptime: 2253 Threads: 11 Questions: 80500 Slow queries: 119 Opens: 294 Flush tables: 1 Open tables: 287 Queries per second avg: 35.730 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 928 [11:27:05] Hello all [11:27:55] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [11:29:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:32:52] 3(commented) [TS-1343] s6 is currently not replicating <10https://jira.toolserver.org/browse/TS-1343> (Marlen Caemmerer) [11:33:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.050781/1.75, alarm hl:np_load_avg=2.156738/2.0, alarm hl:mem_free=693.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.050781/1.9, alarm hl:np_load_long=2.190430/2.25, alarm hl:mem_free=693.000000M/200M, alarm hl:available=1/0 [11:37:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [11:38:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:38:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 915805.000000 [11:45:57] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.032226/1.00, alarm hl:np_load_long=0.933594/1.50, alarm hl:mem_free=19526.000000M/350M, alarm hl:available=1/0 [11:46:57] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:48:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:48:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:48:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:51:57] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.279297/1.10, alarm hl:np_load_long=0.970703/1.55, alarm hl:mem_free=19486.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.279297/1.00, alarm hl:np_load_long=0.970703/1.50, alarm hl:mem_free=19486.000000M/350M, alarm hl:available=1/0 [11:56:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:01:58] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:05:46] Load avg. on willow is WARNING: WARNING - load average: 20.68, 18.89, 17.17 [12:10:15] MySQL slave on thyme is WARNING: No slaves defined [12:16:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.238281/1.75, alarm hl:np_load_avg=2.291504/2.0, alarm hl:mem_free=429.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.238281/1.9, alarm hl:np_load_long=2.204102/2.25, alarm hl:mem_free=429.000000M/200M, alarm hl:available=1/0 [12:22:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:25:53] 3(created) [TS-1344] socat; Toolserver: Software installation; Minor Task <10https://jira.toolserver.org/browse/TS-1344> (Daniel Kinzler) [12:28:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.888672/1.75, alarm hl:np_load_avg=1.854980/2.0, alarm hl:mem_free=779.000000M/350M, alarm hl:available=1/0 [12:37:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:38:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [12:38:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 919413.000000 [12:48:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:48:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:48:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:54:46] Load avg. on willow is OK: OK - load average: 11.54, 13.60, 14.95 [12:57:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:57:46] Load avg. on willow is WARNING: WARNING - load average: 16.48, 15.05, 15.30 [13:01:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:10:25] MySQL slave on thyme is WARNING: No slaves defined [13:14:15] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.452637/1.75, alarm hl:np_load_avg=2.467285/2.0, alarm hl:mem_free=317.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.452637/1.9, alarm hl:np_load_long=2.248535/2.25, alarm hl:mem_free=317.000000M/200M, alarm hl:available=1/0 [13:22:15] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:22:57] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.320312/1.10, alarm hl:np_load_long=0.773438/1.55, alarm hl:mem_free=19767.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.320312/1.00, alarm hl:np_load_long=0.773438/1.50, alarm hl:mem_free=19767.000000M/350M, alarm hl:available=1/0 [13:23:02] Sumana Harihareswara * [Toolserver-l] 17 hours to submit a talk to Open Source Bridge, get WMF subsidy [13:24:58] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:32:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.056641/1.75, alarm hl:np_load_avg=1.922363/2.0, alarm hl:mem_free=509.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.056641/1.9, alarm hl:np_load_long=1.969238/2.25, alarm hl:mem_free=509.000000M/200M, alarm hl:available=1/0 [13:38:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [13:38:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 923014.000000 [13:48:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:49:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:49:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:57:16] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:57:46] Load avg. on willow is WARNING: WARNING - load average: 17.21, 18.01, 17.43 [14:02:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:10:25] MySQL slave on thyme is WARNING: No slaves defined [14:14:52] 3(commented) [TS-1343] s6 is currently not replicating <10https://jira.toolserver.org/browse/TS-1343> (Marlen Caemmerer) [14:24:16] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:28:15] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.935059/1.75, alarm hl:np_load_avg=1.940918/2.0, alarm hl:mem_free=904.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.935059/1.9, alarm hl:np_load_long=2.070312/2.25, alarm hl:mem_free=904.000000M/200M, alarm hl:available=1/0 [14:38:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [14:39:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 926627.000000 [14:45:10] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.200195/1.10, alarm hl:np_load_long=0.714844/1.55, alarm hl:mem_free=20724.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.200195/1.00, alarm hl:np_load_long=0.714844/1.50, alarm hl:mem_free=20724.000000M/350M, alarm hl:available=1/0 [14:46:09] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:49:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:49:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:49:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:52:55] 3(created) [LALM-1] Parsing OSM relations that have relations as members might fail; Look-and-Listen-Map; Minor Bug <10https://jira.toolserver.org/browse/LALM-1> (Peter Wendorff) [14:57:19] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:58:01] Load avg. on willow is WARNING: WARNING - load average: 15.84, 17.03, 17.06 [15:01:19] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 125316 MB (12% inode=99%): [15:02:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:10:30] MySQL slave on thyme is WARNING: No slaves defined [15:14:00] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.24, 23.32, 20.29 [15:28:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.085938/1.75, alarm hl:np_load_avg=3.525879/2.0, alarm hl:mem_free=224.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.085938/1.9, alarm hl:np_load_long=3.250488/2.25, alarm hl:mem_free=224.000000M/200M, alarm hl:available=1/0 [15:38:02] Sumana Harihareswara * [Toolserver-l] Please register for the Berlin hackathon - June 1-3, 2012 [15:38:38] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [15:39:10] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 930235.000000 [15:43:11] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.433594/1.10, alarm hl:np_load_long=0.888672/1.55, alarm hl:mem_free=20287.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.433594/1.00, alarm hl:np_load_long=0.888672/1.50, alarm hl:mem_free=20287.000000M/350M, alarm hl:available=1/0 [15:46:11] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:49:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:49:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:49:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:55:52] 3(created) [ACCAPP-486] Developing tool for transliteration Serbian names from latin <-> cyrilic; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-486> (Predrag Milanovic) [15:57:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:58:11] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.768555/1.10, alarm hl:np_load_long=1.082031/1.55, alarm hl:mem_free=20600.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.768555/1.00, alarm hl:np_load_long=1.082031/1.50, alarm hl:mem_free=20600.000000M/350M, alarm hl:available=1/0 [16:02:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:10:29] MySQL slave on thyme is WARNING: No slaves defined [16:14:09] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.23, 33.99, 32.05 [16:28:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=5.173828/1.75, alarm hl:np_load_avg=4.896484/2.0, alarm hl:mem_free=157.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=5.173828/1.9, alarm hl:np_load_long=4.396973/2.25, alarm hl:mem_free=157.000000M/200M, alarm hl:available=1/0 [16:38:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [16:39:10] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 933837.000000 [16:49:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:49:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:49:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:57:30] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:02:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:11:01] MySQL slave on thyme is WARNING: No slaves defined [17:14:21] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.53, 36.25, 34.94 [17:18:44] * Dispenser pokes DaBPunkt about s1's status [17:29:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.713379/1.75, alarm hl:np_load_avg=3.873047/2.0, alarm hl:mem_free=346.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.713379/1.9, alarm hl:np_load_long=4.107910/2.25, alarm hl:mem_free=346.000000M/200M, alarm hl:available=1/0 [17:39:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [17:39:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 937443.000000 [17:49:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:49:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:49:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:58:01] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:02:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:12:01] MySQL slave on thyme is WARNING: No slaves defined [18:14:30] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.11, 31.86, 31.22 [18:20:32] re [18:21:03] Dispenser: the redirect-table is importing at the moment. [18:24:53] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [18:29:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.991211/1.75, alarm hl:np_load_avg=3.306641/2.0, alarm hl:mem_free=670.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.991211/1.9, alarm hl:np_load_long=3.611816/2.25, alarm hl:mem_free=670.000000M/200M, alarm hl:available=1/0 [18:33:22] @replag [18:33:22] matthewrbowker: s1-rr-a: 1w 3d 21h 18m 18s [+1.00 s/s]; s1-user: 1w 3d 21h 18m 18s [+1.00 s/s]; s3-rr-a: 1m 5s [+0.00 s/s]; s3-user: 1m 5s [+0.00 s/s] [18:39:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [18:39:21] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 941051.000000 [18:49:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:49:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:49:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:58:00] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:02:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:12:01] MySQL slave on thyme is WARNING: No slaves defined [19:14:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.87, 30.98, 31.09 [19:23:50] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.618164/1.10, alarm hl:np_load_long=0.799805/1.55, alarm hl:mem_free=21271.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.618164/1.00, alarm hl:np_load_long=0.799805/1.50, alarm hl:mem_free=21271.000000M/350M, alarm hl:available=1/0 [19:29:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.720703/1.75, alarm hl:np_load_avg=2.879883/2.0, alarm hl:mem_free=677.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.720703/1.9, alarm hl:np_load_long=3.293457/2.25, alarm hl:mem_free=677.000000M/200M, alarm hl:available=1/0 [19:38:31] /tmp on willow is WARNING: DISK WARNING - free space: / 21975 MB (20% inode=99%): [19:38:51] / on willow is WARNING: DISK WARNING - free space: / 21970 MB (20% inode=99%): [19:39:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [19:40:20] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 944712.000000 [19:40:40] @replag [19:40:40] Thehelpfulone: s1-rr-a: 1w 3d 22h 25m 36s [+1.00 s/s]; s1-user: 1w 3d 22h 25m 36s [+1.00 s/s]; s3-rr-a: 21s [-0.01 s/s]; s3-user: 21s [-0.01 s/s] [19:42:51] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [19:49:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:49:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:50:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:58:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:01:12] @replag [20:01:12] matthewrbowker: s1-rr-a: 1w 3d 22h 46m 7s [+1.00 s/s]; s1-user: 1w 3d 22h 46m 7s [+1.00 s/s]; s3-rr-a: 11s [-0.01 s/s]; s3-user: 11s [-0.01 s/s]; s5-rr-a: 11s [+0.00 s/s]; s5-user: 11s [+0.00 s/s] [20:02:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:03:30] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:13:00] MySQL slave on thyme is WARNING: No slaves defined [20:13:31] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:14:30] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.44, 23.88, 24.24 [20:27:30] Load avg. on willow is WARNING: WARNING - load average: 14.30, 16.56, 19.80 [20:28:30] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:29:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.961426/1.75, alarm hl:np_load_avg=2.047852/2.0, alarm hl:mem_free=491.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.961426/1.9, alarm hl:np_load_long=2.433105/2.25, alarm hl:mem_free=491.000000M/200M, alarm hl:available=1/0 [20:30:30] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.62, 18.96, 20.03 [20:31:31] Load avg. on willow is WARNING: WARNING - load average: 21.14, 18.67, 19.86 [20:38:32] /tmp on willow is WARNING: DISK WARNING - free space: / 21378 MB (20% inode=99%): [20:38:51] / on willow is WARNING: DISK WARNING - free space: / 21374 MB (20% inode=99%): [20:39:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [20:40:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 948318.000000 [20:50:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:50:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:50:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:58:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:00:32] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.91, 21.96, 19.32 [21:01:31] Load avg. on willow is WARNING: WARNING - load average: 23.21, 21.32, 19.26 [21:03:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:10:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.42, 23.76, 20.92 [21:13:01] MySQL slave on thyme is WARNING: No slaves defined [21:23:41] Load avg. on willow is WARNING: WARNING - load average: 12.95, 17.97, 19.90 [21:29:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.052246/1.75, alarm hl:np_load_avg=2.116699/2.0, alarm hl:mem_free=361.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.052246/1.9, alarm hl:np_load_long=2.354492/2.25, alarm hl:mem_free=361.000000M/200M, alarm hl:available=1/0 [21:38:41] /tmp on willow is WARNING: DISK WARNING - free space: / 20734 MB (19% inode=98%): [21:39:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [21:39:51] / on willow is WARNING: DISK WARNING - free space: / 20721 MB (19% inode=98%): [21:40:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 951927.000000 [21:50:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:50:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:51:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:53:46] nacht ts [21:58:11] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:00:50] / on willow is OK: DISK OK - free space: / 27649 MB (26% inode=99%): [22:01:41] /tmp on willow is OK: DISK OK - free space: / 27577 MB (26% inode=99%): [22:03:01] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:05:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.10, 21.36, 20.23 [22:06:40] Load avg. on willow is WARNING: WARNING - load average: 18.22, 20.20, 19.89 [22:10:41] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.32, 21.61, 20.46 [22:13:10] MySQL slave on thyme is WARNING: No slaves defined [22:21:41] Load avg. on willow is WARNING: WARNING - load average: 15.52, 19.32, 19.98 [22:28:21] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:31:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.571777/1.75, alarm hl:np_load_avg=2.155762/2.0, alarm hl:mem_free=315.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.571777/1.9, alarm hl:np_load_long=2.268555/2.25, alarm hl:mem_free=315.000000M/200M, alarm hl:available=1/0 [22:32:40] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:20] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:33:41] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:38:42] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:39:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [22:40:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 955532.000000 [22:47:50] @replag [22:47:50] Thehelpfulone: s1-rr-a: 1w 4d 1h 32m 46s [+1.00 s/s]; s1-user: 1w 4d 1h 32m 46s [+1.00 s/s]; s3-rr-a: 1m 9s [+0.01 s/s]; s3-user: 1m 9s [+0.01 s/s]; s5-rr-a: 10s [-0.00 s/s]; s5-user: 10s [-0.00 s/s] [22:50:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:50:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:51:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:53:08] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:58:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:01:02] Platonides * Re: [Toolserver-l] Dementia [23:03:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:05:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.376953/1.75, alarm hl:np_load_avg=2.218750/2.0, alarm hl:mem_free=337.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.376953/1.9, alarm hl:np_load_long=2.110840/2.25, alarm hl:mem_free=337.000000M/200M, alarm hl:available=1/0 [23:13:40] MySQL slave on thyme is WARNING: No slaves defined [23:21:41] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:21:48] Load avg. on willow is WARNING: WARNING - load average: 12.50, 14.99, 16.33 [23:26:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.976562/1.75, alarm hl:np_load_avg=1.772949/2.0, alarm hl:mem_free=292.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.976562/1.9, alarm hl:np_load_long=1.938476/2.25, alarm hl:mem_free=292.000000M/200M, alarm hl:available=1/0 [23:28:48] Load avg. on willow is OK: OK - load average: 12.68, 13.38, 15.00 [23:31:48] Load avg. on willow is WARNING: WARNING - load average: 17.24, 16.27, 15.91 [23:33:42] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:37:01] Load avg. on willow is OK: OK - load average: 12.67, 13.92, 14.93 [23:40:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [23:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 959152.000000 [23:51:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:51:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:52:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:58:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1