[00:03:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.729980/1.75, alarm hl:np_load_avg=1.635742/2.0, alarm hl:mem_free=248.000000M/350M, alarm hl:available=1/0 [00:07:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:09:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:13:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.843262/1.75, alarm hl:np_load_avg=1.868652/2.0, alarm hl:mem_free=469.000000M/350M, alarm hl:available=1/0 [00:13:42] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [00:44:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:45:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [00:46:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:47:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:52:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 484182 MB (9% inode=45%): [00:59:15] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 607970.000000 [01:01:56] can anyone help? [01:09:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:09:26] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.439941/1.75, alarm hl:np_load_avg=1.569336/2.0, alarm hl:mem_free=239.000000M/350M, alarm hl:available=1/0 [01:13:43] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:15:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:21:13] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 34561 MB (8% inode=99%): [01:25:13] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22869 MB (5% inode=99%): [01:41:13] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 29117 MB (7% inode=99%): [01:42:14] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 21573 MB (5% inode=99%): [01:44:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:46:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [01:46:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:47:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:51:14] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40519 MB (9% inode=99%): [01:52:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 484160 MB (9% inode=45%): [01:59:24] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 611581.000000 [02:03:13] Load avg. on willow is WARNING: WARNING - load average: 15.09, 15.82, 13.59 [02:03:36] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.873535/1.75, alarm hl:np_load_avg=1.977539/2.0, alarm hl:mem_free=396.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.873535/2.1, alarm hl:np_load_long=1.695801/2.5, alarm hl:cpu=97.100000/97, alarm hl:mem_free=396.000000M/200M, alarm hl:available=1/0 [02:08:13] Load avg. on willow is OK: OK - load average: 13.73, 14.98, 13.87 [02:09:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:12:12] Load avg. on willow is WARNING: WARNING - load average: 18.94, 17.56, 15.20 [02:13:43] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:22:35] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:23:14] Load avg. on willow is OK: OK - load average: 11.78, 14.68, 14.98 [02:44:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:46:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [02:47:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:47:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:50:20] chicocvenancio: I think you file a JIRA request [02:50:44] Dispenser: I sent an email to the admins email... [02:51:30] Do you think that it is insuficient? [02:52:25] Not sure how often they check it... Its probably sufficient if you can wait up to 2 weeks. [02:53:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 483135 MB (9% inode=45%): [03:00:25] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 615242.000000 [03:02:23] Dispenser: would a JIRA request be any quicker? [03:02:35] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.024902/1.75, alarm hl:np_load_avg=1.861328/2.0, alarm hl:mem_free=395.000000M/350M, alarm hl:available=1/0 [03:03:35] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:03:43] Possibility, fastest is to bug an admin when they're online here [03:09:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:12:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.746094/1.75, alarm hl:np_load_avg=1.617188/2.0, alarm hl:mem_free=247.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.746094/2.1, alarm hl:np_load_long=1.605469/2.5, alarm hl:cpu=99.500000/97, alarm hl:mem_free=247.000000M/200M, alarm hl:available=1/0 [03:13:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:39:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.333496/1.75, alarm hl:np_load_avg=1.517578/2.0, alarm hl:mem_free=317.000000M/350M, alarm hl:available=1/0 [03:42:46] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:44:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:46:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [03:47:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:47:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:50:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.024902/1.75, alarm hl:np_load_avg=1.595215/2.0, alarm hl:mem_free=256.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.024902/2.1, alarm hl:np_load_long=1.567871/2.5, alarm hl:cpu=98.600000/97, alarm hl:mem_free=256.000000M/200M, alarm hl:available=1/0 [03:52:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:53:24] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 482231 MB (9% inode=45%): [04:01:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 618901.000000 [04:09:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:12:13] Load avg. on willow is WARNING: WARNING - load average: 18.07, 16.47, 14.62 [04:13:54] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:17:13] Load avg. on willow is OK: OK - load average: 12.40, 14.59, 14.38 [04:30:14] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7011&oldid=6992&rcid=9238 * 91.198.174.202 * (+2) (updated page) [04:44:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:46:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [04:47:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:47:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:53:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 482198 MB (9% inode=45%): [05:01:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 622504.000000 [05:03:03] someone should ban Alchimista [05:10:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:12:14] Load avg. on willow is WARNING: WARNING - load average: 17.05, 15.71, 14.37 [05:14:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:14:13] Load avg. on willow is OK: OK - load average: 13.26, 14.84, 14.22 [05:14:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.629395/1.75, alarm hl:np_load_avg=1.844238/2.0, alarm hl:mem_free=303.000000M/350M, alarm hl:available=1/0 [05:21:53] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:27:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.797363/1.75, alarm hl:np_load_avg=1.766113/2.0, alarm hl:mem_free=409.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.797363/2.1, alarm hl:np_load_long=1.757324/2.5, alarm hl:cpu=97.700000/97, alarm hl:mem_free=409.000000M/200M, alarm hl:available=1/0 [05:29:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:32:15] Load avg. on willow is WARNING: WARNING - load average: 19.05, 16.74, 15.12 [05:44:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:47:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [05:47:14] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:47:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:54:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 482139 MB (9% inode=45%): [06:01:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 626105.000000 [06:11:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:15:04] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:30:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.059082/1.75, alarm hl:np_load_avg=2.428223/2.0, alarm hl:mem_free=516.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.059082/2.1, alarm hl:np_load_long=2.171875/2.5, alarm hl:cpu=97.400000/97, alarm hl:mem_free=516.000000M/200M, alarm hl:available=1/0 [06:32:34] Load avg. on willow is WARNING: WARNING - load average: 15.45, 17.50, 16.89 [06:33:54] 3(created) [ACC-245] passwort reset should link to http://en.wikipedia.org/wiki/Special:PasswordReset; ACC: Internal interface; Trivial Bug <10https://jira.toolserver.org/browse/ACC-245> (mabdul) [06:45:44] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.928711/1.10, alarm hl:np_load_long=1.095703/1.55, alarm hl:mem_free=19836.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.928711/1.00, alarm hl:np_load_long=1.095703/1.50, alarm hl:mem_free=19836.000000M/350M, alarm hl:available=1/0 [06:45:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:47:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:47:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:48:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [06:48:44] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:54:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 482038 MB (9% inode=45%): [06:59:53] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:01:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 629708.000000 [07:02:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.699707/1.75, alarm hl:np_load_avg=1.902344/2.0, alarm hl:mem_free=243.000000M/350M, alarm hl:available=1/0 [07:04:12] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:11:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:15:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:33:33] Load avg. on willow is WARNING: WARNING - load average: 14.33, 15.07, 15.61 [07:48:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:48:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:48:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [07:48:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:53:46] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:54:46] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481952 MB (9% inode=45%): [08:01:44] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 633317.000000 [08:03:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.910156/1.75, alarm hl:np_load_avg=2.165527/2.0, alarm hl:mem_free=296.000000M/350M, alarm hl:available=1/0 [08:09:46] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.372070/1.10, alarm hl:np_load_long=0.781250/1.55, alarm hl:mem_free=19258.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.372070/1.00, alarm hl:np_load_long=0.781250/1.50, alarm hl:mem_free=19258.000000M/350M, alarm hl:available=1/0 [08:11:45] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:12:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:15:13] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:21:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:25:34] Load avg. on willow is OK: OK - load average: 13.07, 13.27, 14.88 [08:43:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.683105/1.75, alarm hl:np_load_avg=1.628418/2.0, alarm hl:mem_free=301.000000M/350M, alarm hl:available=1/0 [08:43:45] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.018555/1.00, alarm hl:np_load_long=0.744140/1.50, alarm hl:mem_free=19680.000000M/350M, alarm hl:available=1/0 [08:44:46] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:46:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:48:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [08:48:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:49:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:52:33] Load avg. on willow is WARNING: WARNING - load average: 18.53, 16.01, 14.55 [08:55:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481858 MB (9% inode=45%): [09:01:45] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 636918.000000 [09:03:14] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 784 MB (20% inode=99%): [09:04:15] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1587 MB (34% inode=99%): [09:12:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:15:14] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:19:54] 3(commented) [ACC-245] passwort reset should link to http://en.wikipedia.org/wiki/Special:PasswordReset <10https://jira.toolserver.org/browse/ACC-245> (Manish Goregaokar) [09:26:14] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:26:33] Load avg. on willow is OK: OK - load average: 11.20, 12.75, 14.91 [09:32:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.852539/1.75, alarm hl:np_load_avg=1.724609/2.0, alarm hl:mem_free=179.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.852539/2.1, alarm hl:np_load_long=1.825195/2.5, alarm hl:cpu=97.600000/97, alarm hl:mem_free=179.000000M/200M, alarm hl:available=1/0 [09:33:15] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:46:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:49:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [09:49:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:49:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:55:45] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481708 MB (9% inode=45%): [09:56:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.611816/1.75, alarm hl:np_load_avg=1.600586/2.0, alarm hl:mem_free=210.000000M/350M, alarm hl:available=1/0 [10:02:45] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 640578.000000 [10:12:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:12:33] Load avg. on willow is WARNING: WARNING - load average: 19.65, 16.32, 14.69 [10:13:53] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:14:23] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:15:14] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:17:33] Load avg. on willow is OK: OK - load average: 11.94, 14.55, 14.50 [10:18:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:19:52] 3(updated) [ACC-245] password reset should link to http://en.wikipedia.org/wiki/Special:PasswordReset <10https://jira.toolserver.org/browse/ACC-245> (Manish Goregaokar) [10:32:33] Load avg. on willow is WARNING: WARNING - load average: 23.57, 16.50, 14.73 [10:33:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.981445/1.75, alarm hl:np_load_avg=2.133301/2.0, alarm hl:mem_free=260.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.981445/2.1, alarm hl:np_load_long=1.870605/2.5, alarm hl:cpu=100.000000/97, alarm hl:mem_free=260.000000M/200M, alarm hl:available=1/0 [10:36:15] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:39:34] Load avg. on willow is OK: OK - load average: 11.92, 14.63, 14.64 [10:46:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:49:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:50:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [10:50:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:56:44] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481628 MB (9% inode=45%): [11:01:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.242188/1.10, alarm hl:np_load_long=0.737305/1.55, alarm hl:mem_free=19718.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.242188/1.00, alarm hl:np_load_long=0.737305/1.50, alarm hl:mem_free=19718.000000M/350M, alarm hl:available=1/0 [11:02:34] Load avg. on willow is WARNING: WARNING - load average: 16.60, 14.38, 13.79 [11:02:45] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 644179.000000 [11:03:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.864258/1.75, alarm hl:np_load_avg=1.770020/2.0, alarm hl:mem_free=644.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.864258/2.1, alarm hl:np_load_long=1.715820/2.5, alarm hl:cpu=97.500000/97, alarm hl:mem_free=644.000000M/200M, alarm hl:available=1/0 [11:03:33] Load avg. on willow is OK: OK - load average: 13.46, 13.88, 13.65 [11:04:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:05:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:10:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.524414/1.10, alarm hl:np_load_long=0.997070/1.55, alarm hl:mem_free=19553.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.524414/1.00, alarm hl:np_load_long=0.997070/1.50, alarm hl:mem_free=19553.000000M/350M, alarm hl:available=1/0 [11:13:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:13:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.791016/1.75, alarm hl:np_load_avg=1.777832/2.0, alarm hl:mem_free=446.000000M/350M, alarm hl:available=1/0 [11:15:14] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [11:44:13] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:47:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:50:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [11:50:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:50:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:53:53] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:56:54] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481511 MB (9% inode=45%): [11:58:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.416504/1.75, alarm hl:np_load_avg=1.456543/2.0, alarm hl:mem_free=349.000000M/350M, alarm hl:available=1/0 [12:00:14] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:00:54] 3(created) [WIKISENSE-60] Add slwikiversity; WikiSense: CommonSense; Trivial Bug <10https://jira.toolserver.org/browse/WIKISENSE-60> (David Crochet) [12:02:35] Load avg. on willow is WARNING: WARNING - load average: 16.43, 14.82, 13.28 [12:03:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:03:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.873535/1.75, alarm hl:np_load_avg=1.831055/2.0, alarm hl:mem_free=299.000000M/350M, alarm hl:available=1/0 [12:03:34] Load avg. on willow is OK: OK - load average: 14.82, 14.64, 13.31 [12:03:44] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 647842.000000 [12:06:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.473633/1.10, alarm hl:np_load_long=1.400391/1.55, alarm hl:mem_free=19728.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.473633/1.00, alarm hl:np_load_long=1.400391/1.50, alarm hl:mem_free=19728.000000M/350M, alarm hl:available=1/0 [12:12:33] Load avg. on willow is WARNING: WARNING - load average: 15.97, 14.76, 13.87 [12:13:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:16:13] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:20:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:38:58] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (Marlen Caemmerer) [12:47:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:50:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [12:50:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:50:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:53:12] [[Special:Log/newusers]] create 10 * Tokai Csaba * (New user account) [12:56:53] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481415 MB (9% inode=45%): [13:02:34] Load avg. on willow is WARNING: WARNING - load average: 14.60, 15.18, 14.32 [13:03:33] Load avg. on willow is OK: OK - load average: 13.52, 14.74, 14.22 [13:03:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 615750.000000 [13:07:34] Load avg. on willow is WARNING: WARNING - load average: 16.11, 15.37, 14.59 [13:13:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:13:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.025879/1.75, alarm hl:np_load_avg=1.931641/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.025879/2.1, alarm hl:np_load_long=1.849609/2.5, alarm hl:cpu=99.300000/97, alarm hl:mem_free=125.000000M/200M, alarm hl:available=1/0 [13:14:53] @replag [13:14:54] Joan: No replag found. Use "@replag all" to see all lags. [13:16:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2867008 [13:16:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:39:52] 3(created) [DBQ-183] SQL Query Issue; Database Queries; Critical Task <10https://jira.toolserver.org/browse/DBQ-183> (Franz Herbach) [13:47:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:50:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:51:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [13:51:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:56:53] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481304 MB (9% inode=45%): [13:59:02] hello all [14:02:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.796875/1.75, alarm hl:np_load_avg=1.652832/2.0, alarm hl:mem_free=712.000000M/350M, alarm hl:available=1/0 [14:03:14] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:03:44] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 562647.000000 [14:12:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.886230/1.75, alarm hl:np_load_avg=1.707031/2.0, alarm hl:mem_free=210.000000M/350M, alarm hl:available=1/0 [14:13:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:16:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2843986 [14:35:07] Somebody mind updating status.toolserver.org? [14:42:58] 3(commented) [DBQ-182] List of superseded images still used in the article name space <10https://jira.toolserver.org/browse/DBQ-182> (Danhash) [14:47:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:50:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:50:55] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [14:50:57] 3(commented) [DBQ-182] List of superseded images still used in the article name space <10https://jira.toolserver.org/browse/DBQ-182> (Hoo man) [14:51:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [14:51:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:57:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481181 MB (9% inode=45%): [15:02:44] Load avg. on willow is WARNING: WARNING - load average: 15.24, 13.98, 13.05 [15:03:44] Load avg. on willow is OK: OK - load average: 12.24, 13.34, 12.88 [15:03:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 492712.000000 [15:05:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.448242/1.75, alarm hl:np_load_avg=1.610840/2.0, alarm hl:mem_free=193.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.448242/2.1, alarm hl:np_load_long=1.593750/2.5, alarm hl:cpu=92.700000/97, alarm hl:mem_free=193.000000M/200M, alarm hl:available=1/0 [15:13:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:16:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2822918 [15:17:24] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:25:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.827637/1.75, alarm hl:np_load_avg=1.693359/2.0, alarm hl:mem_free=539.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.827637/2.1, alarm hl:np_load_long=1.665527/2.5, alarm hl:cpu=97.800000/97, alarm hl:mem_free=539.000000M/200M, alarm hl:available=1/0 [15:26:24] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:32:43] Load avg. on willow is WARNING: WARNING - load average: 15.16, 14.59, 13.77 [15:33:44] Load avg. on willow is OK: OK - load average: 13.86, 14.30, 13.72 [15:42:44] Load avg. on willow is WARNING: WARNING - load average: 15.36, 14.29, 13.79 [15:48:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.823242/1.75, alarm hl:np_load_avg=1.754883/2.0, alarm hl:mem_free=173.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.823242/2.1, alarm hl:np_load_long=1.726562/2.5, alarm hl:cpu=84.300000/97, alarm hl:mem_free=173.000000M/200M, alarm hl:available=1/0 [15:48:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:50:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:52:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [15:52:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:57:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481517 MB (9% inode=45%): [15:59:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:02:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.057129/1.75, alarm hl:np_load_avg=1.964355/2.0, alarm hl:mem_free=219.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.057129/2.1, alarm hl:np_load_long=1.805664/2.5, alarm hl:cpu=99.800000/97, alarm hl:mem_free=219.000000M/200M, alarm hl:available=1/0 [16:04:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 451161.000000 [16:13:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:16:56] 3(created) [TS-1353] Switch SSH key for my toolserver account; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1353> (Daniel Salciccioli) [16:17:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2783990 [16:18:55] 3(updated) [TS-1353] Switch SSH key for my toolserver account <10https://jira.toolserver.org/browse/TS-1353> (Daniel Salciccioli) [16:21:02] DaB. * Re: [Toolserver-announce] [Toolserver-l] News about s1/enwiki [16:21:33] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:23:53] 3(commented) [TS-1353] Switch SSH key for my toolserver account <10https://jira.toolserver.org/browse/TS-1353> (Daniel Salciccioli) [16:26:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.850098/1.75, alarm hl:np_load_avg=1.732422/2.0, alarm hl:mem_free=887.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.850098/2.1, alarm hl:np_load_long=1.812500/2.5, alarm hl:cpu=100.000000/97, alarm hl:mem_free=887.000000M/200M, alarm hl:available=1/0 [16:28:33] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:28:57] 3(resolved) [TS-1353] Switch SSH key for my toolserver account <10https://jira.toolserver.org/browse/TS-1353> (DaB.) [16:28:59] 3(assigned) [TS-1353] Switch SSH key for my toolserver account <10https://jira.toolserver.org/browse/TS-1353> (DaB.) [16:42:44] Load avg. on willow is WARNING: WARNING - load average: 15.32, 15.52, 15.25 [16:48:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:51:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:52:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [16:52:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:55:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.542480/1.75, alarm hl:np_load_avg=2.427246/2.0, alarm hl:mem_free=479.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.542480/2.1, alarm hl:np_load_long=2.157715/2.5, alarm hl:cpu=100.000000/97, alarm hl:mem_free=479.000000M/200M, alarm hl:available=1/0 [16:57:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481452 MB (9% inode=45%): [17:04:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 383525.000000 [17:09:53] 3(commented) [UTRS-96] HTTP 404 makes a rather concerning privacy policy <10https://jira.toolserver.org/browse/UTRS-96> (Martijn Hoekstra) [17:13:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:15:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:17:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2757763 [17:24:44] Load avg. on willow is OK: OK - load average: 11.58, 12.84, 14.84 [17:31:52] 3(commented) [UTRS-87] Log timestamps are sometimes incorrect <10https://jira.toolserver.org/browse/UTRS-87> (Martijn Hoekstra) [17:33:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.405762/1.75, alarm hl:np_load_avg=1.605957/2.0, alarm hl:mem_free=230.000000M/350M, alarm hl:available=1/0 [17:42:35] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:46:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.859863/1.75, alarm hl:np_load_avg=1.726074/2.0, alarm hl:mem_free=270.000000M/350M, alarm hl:available=1/0 [17:48:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:51:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:52:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [17:52:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:53:17] DaBPunkt: need help fixing ^^ ? [17:55:54] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:56:23] SMF on z-dat-s4-a is OK: OK - all services online [17:57:05] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481369 MB (9% inode=45%): [18:02:45] Load avg. on willow is WARNING: WARNING - load average: 15.33, 15.49, 14.23 [18:03:44] Load avg. on willow is OK: OK - load average: 13.78, 14.98, 14.13 [18:05:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 295350.000000 [18:07:44] Load avg. on willow is WARNING: WARNING - load average: 15.61, 15.65, 14.62 [18:09:00] bonne nuit~ [18:13:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.746582/1.75, alarm hl:np_load_avg=1.850586/2.0, alarm hl:mem_free=151.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.746582/2.1, alarm hl:np_load_long=1.816895/2.5, alarm hl:cpu=96.600000/97, alarm hl:mem_free=151.000000M/200M, alarm hl:available=1/0 [18:14:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:17:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2731425 [18:23:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:27:35] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.566406/1.75, alarm hl:np_load_avg=1.587402/2.0, alarm hl:mem_free=137.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.566406/2.1, alarm hl:np_load_long=1.691406/2.5, alarm hl:cpu=97.700000/97, alarm hl:mem_free=137.000000M/200M, alarm hl:available=1/0 [18:33:35] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:33:54] 3(resolved) [UTRS-96] HTTP 404 makes a rather concerning privacy policy <10https://jira.toolserver.org/browse/UTRS-96> (Chris Howie) [18:39:14] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:48:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:51:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:52:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [18:52:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:54:13] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [18:55:35] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.952637/1.75, alarm hl:np_load_avg=1.534668/2.0, alarm hl:mem_free=185.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.952637/2.1, alarm hl:np_load_long=1.530762/2.5, alarm hl:cpu=92.000000/97, alarm hl:mem_free=185.000000M/200M, alarm hl:available=1/0 [18:57:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481257 MB (9% inode=45%): [18:57:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:01:36] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.238770/1.75, alarm hl:np_load_avg=1.797851/2.0, alarm hl:mem_free=864.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.238770/2.1, alarm hl:np_load_long=1.621582/2.5, alarm hl:cpu=94.700000/97, alarm hl:mem_free=864.000000M/200M, alarm hl:available=1/0 [19:02:35] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:05:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 260149.000000 [19:09:14] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:14:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:17:35] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2697303 [19:48:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:51:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:52:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:53:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [19:57:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481157 MB (9% inode=45%): [20:06:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207024.000000 [20:12:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.783203/1.75, alarm hl:np_load_avg=1.637207/2.0, alarm hl:mem_free=228.000000M/350M, alarm hl:available=1/0 [20:14:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:15:46] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:17:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2676131 [20:24:46] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.586426/1.75, alarm hl:np_load_avg=1.557617/2.0, alarm hl:mem_free=207.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.586426/2.1, alarm hl:np_load_long=1.530762/2.5, alarm hl:cpu=98.400000/97, alarm hl:mem_free=207.000000M/200M, alarm hl:available=1/0 [20:48:45] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:51:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:52:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:53:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [20:57:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 481015 MB (9% inode=45%): [20:57:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.478027/1.75, alarm hl:np_load_avg=1.467285/2.0, alarm hl:mem_free=279.000000M/350M, alarm hl:available=1/0 [21:02:57] 3(created) [ACCAPP-493] I am a teaching assistant in stanford cs341 mining massive datasets, some user projects might become toolserver tools; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-493> (keith siilats) [21:03:46] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:06:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 164775.000000 [21:07:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.709961/1.75, alarm hl:np_load_avg=1.743164/2.0, alarm hl:mem_free=184.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.709961/2.1, alarm hl:np_load_long=1.657715/2.5, alarm hl:cpu=97.600000/97, alarm hl:mem_free=184.000000M/200M, alarm hl:available=1/0 [21:12:54] Load avg. on willow is WARNING: WARNING - load average: 15.66, 16.04, 14.40 [21:14:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:14:54] Load avg. on willow is OK: OK - load average: 11.17, 14.44, 14.02 [21:17:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2656025 [21:19:14] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:32:54] Load avg. on willow is WARNING: WARNING - load average: 16.05, 17.83, 15.88 [21:38:54] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [21:42:45] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:43:54] Load avg. on willow is OK: OK - load average: 11.93, 14.23, 14.97 [21:45:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.074707/1.75, alarm hl:np_load_avg=1.844726/2.0, alarm hl:mem_free=241.000000M/350M, alarm hl:available=1/0 [21:48:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:51:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:51:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:52:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:53:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [21:57:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 480949 MB (9% inode=45%): [22:06:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98028.000000 [22:12:52] 3(created) [ACCAPP-494] copycat Defconbot; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-494> (pixeltoo) [22:12:54] Load avg. on willow is WARNING: WARNING - load average: 18.97, 16.80, 15.12 [22:13:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.346191/1.75, alarm hl:np_load_avg=2.124512/2.0, alarm hl:mem_free=399.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.346191/2.1, alarm hl:np_load_long=1.907226/2.5, alarm hl:cpu=99.900000/97, alarm hl:mem_free=399.000000M/200M, alarm hl:available=1/0 [22:15:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:16:53] Load avg. on willow is OK: OK - load average: 11.88, 14.51, 14.60 [22:17:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2625148 [22:18:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:24:46] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.890625/1.75, alarm hl:np_load_avg=1.721680/2.0, alarm hl:mem_free=372.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.890625/2.1, alarm hl:np_load_long=1.756836/2.5, alarm hl:cpu=100.000000/97, alarm hl:mem_free=372.000000M/200M, alarm hl:available=1/0 [22:27:19] bonne nuit =_= [22:28:45] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:32:54] Load avg. on willow is WARNING: WARNING - load average: 13.34, 15.41, 14.87 [22:48:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:51:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:52:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:53:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [22:57:06] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 480903 MB (9% inode=45%): [23:06:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18700.000000 [23:09:14] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:13:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.031738/1.75, alarm hl:np_load_avg=2.062500/2.0, alarm hl:mem_free=216.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.031738/2.1, alarm hl:np_load_long=1.987305/2.5, alarm hl:cpu=98.000000/97, alarm hl:mem_free=216.000000M/200M, alarm hl:available=1/0 [23:15:45] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:17:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2608183 [23:17:45] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:19:04] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3191.000000 [23:20:05] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1622.000000 [23:38:46] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:40:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.931152/1.75, alarm hl:np_load_avg=1.647949/2.0, alarm hl:mem_free=525.000000M/350M, alarm hl:available=1/0 [23:41:46] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:48:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:51:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:52:46] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.270508/1.75, alarm hl:np_load_avg=1.408691/2.0, alarm hl:mem_free=321.000000M/350M, alarm hl:available=1/0 [23:52:46] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:53:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [23:57:15] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 480810 MB (9% inode=45%):