[00:05:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580981 MB (10% inode=49%): [00:09:47] [[Toolserver database]] !M 10https://wiki.toolserver.org/w/index.php?diff=6996&oldid=6527&rcid=9214 * Wiegels * (-1) (ortho) [00:15:59] Load avg. on willow is WARNING: WARNING - load average: 17.43, 16.59, 16.61 [00:19:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:25:00] /aux0 on hemlock is OK: DISK OK - free space: /aux0 582967 MB (11% inode=50%): [00:26:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [00:27:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:30:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:36:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 260976.000000 [00:44:09] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:44:30] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [00:44:30] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:46:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:06:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.420898/1.75, alarm hl:np_load_avg=2.213379/2.0, alarm hl:mem_free=432.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.420898/1.9, alarm hl:np_load_long=2.174316/2.25, alarm hl:mem_free=432.000000M/200M, alarm hl:available=1/0 [01:08:49] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:09:08] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:12:09] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.058105/1.75, alarm hl:np_load_avg=2.134766/2.0, alarm hl:mem_free=309.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.058105/1.9, alarm hl:np_load_long=2.135742/2.25, alarm hl:mem_free=309.000000M/200M, alarm hl:available=1/0 [01:15:09] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:16:01] Load avg. on willow is WARNING: WARNING - load average: 17.87, 16.76, 16.87 [01:20:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:24:30] [[Special:Log/newusers]] create 10 * Munez * (New user account) [01:26:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [01:27:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:30:21] is there a bulk upload tool for flickr? [01:30:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:32:13] ToAruShiroiNeko: https://toolserver.org/~magnus/ [01:34:45] Hi everybody! [01:36:00] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 264577.000000 [01:36:36] You back! [01:36:41] Your back! [01:37:15] You're back! [01:39:03] dschwen: When will you ever merge my zoom-by-sigfigs patch into WMA [01:39:15] uhm [01:39:21] what patch? [01:39:39] what does it do? [01:39:43] ist it in JIRA? [01:40:05] I have been super busy with WMA lately [01:40:06] the one I email you in June 2011 [01:40:18] one sec, let me look [01:41:53] found it. sorry I don't know how I could have missed that [01:42:01] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:01] Load avg. on willow is OK: OK - load average: 9.66, 13.07, 14.78 [01:42:09] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:42:09] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:09] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:18] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:42:18] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:42:19] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:42:28] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:42:39] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:42:47] lets see if that patch still applies :-) [01:42:49] SMTP on z-dat-s4-a is OK: SMTP OK - 0.003 sec. response time [01:42:49] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 186505 MB (19% inode=99%): [01:42:49] / on z-dat-s6-a is OK: DISK OK - free space: / 8516 MB (28% inode=85%): [01:42:49] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 2763 MB (99% inode=99%): [01:42:59] Load avg. on z-dat-s6-a is OK: OK - load average: 1.95, 2.41, 2.90 [01:43:00] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:43:00] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:44:40] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:45:29] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:45:52] hunk 3 failed :-( [01:46:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:48:00] Load avg. on willow is WARNING: WARNING - load average: 13.41, 14.71, 15.04 [01:49:00] Load avg. on willow is OK: OK - load average: 13.38, 14.50, 14.94 [01:55:01] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:01] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:02] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:02] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:09] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:10] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:10] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:11] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:11] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:11] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:19] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:20] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:20] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:29] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:30] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:30] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:30] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:43] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:50] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:50] SMTP on hyacinth is OK: SMTP OK - 2.083 sec. response time [01:55:50] SMTP on z-dat-s3-a is OK: SMTP OK - 2.773 sec. response time [01:55:50] SMTP on z-dat-s7-a is OK: SMTP OK - 2.955 sec. response time [01:56:00] Load avg. on z-dat-s4-a is OK: OK - load average: 0.56, 1.52, 2.26 [01:56:00] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [01:56:00] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:56:00] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:56:00] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 2712 MB (99% inode=99%): [01:56:00] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 103289 MB (25% inode=99%): [01:56:09] SMF on z-dat-s6-a is OK: OK - all services online [01:56:31] I'm applying it by hand [01:56:40] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [01:56:41] thanks by the way [01:57:19] any roots around? [02:01:44] Dispenser, have you noticed the new features and new Basemap in WMA? [02:02:11] I been spending most of my time with disambiguation [02:03:02] And of that figuring out way to get useful data to present [02:03:50] no more geo stuff? [02:04:13] what are you aiming at with the disambiguations? [02:05:58] anyhow, I submitted a talk for Wikimania. If it gets accepted I'll be going to DC [02:06:00] Improve the automatic creation of disambiguation pages (Dabfix) and stream those improvements to my disambiguating tool (Dab solver) [02:06:41] I'll be at HOPE in NYC, same weekend unfortunately [02:06:51] oh [02:10:59] Load avg. on willow is WARNING: WARNING - load average: 22.96, 18.39, 16.91 [02:11:28] I'll return to geo stuff once I get someone to coordinate participation in disambiguation [02:12:32] but your extraction project is still running, is it? [02:12:36] http://enwp.org/WT:DPL#Dab_solver_topic_director [02:12:48] oh yes, three hours every day [02:16:58] 3(created) [DBQ-180] Active Bots; Database Queries; Critical Task <10https://jira.toolserver.org/browse/DBQ-180> (Franz Herbach) [02:18:51] 3(updated) [DBQ-180] Database Query (question) <10https://jira.toolserver.org/browse/DBQ-180> (Franz Herbach) [02:20:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:20:54] 3(updated) [DBQ-180] Database Query (question) <10https://jira.toolserver.org/browse/DBQ-180> (Franz Herbach) [02:26:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [02:27:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:28:59] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 581719 MB (10% inode=49%): [02:30:49] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:37:00] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 268237.000000 [02:44:41] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:45:40] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:46:19] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:52:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.073242/1.75, alarm hl:np_load_avg=2.096680/2.0, alarm hl:mem_free=329.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.073242/1.9, alarm hl:np_load_long=2.041016/2.25, alarm hl:mem_free=329.000000M/200M, alarm hl:available=1/0 [02:59:18] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:02:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.031738/1.75, alarm hl:np_load_avg=2.073730/2.0, alarm hl:mem_free=209.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.031738/1.9, alarm hl:np_load_long=2.044434/2.25, alarm hl:mem_free=209.000000M/200M, alarm hl:available=1/0 [03:03:19] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:10:59] Load avg. on willow is WARNING: WARNING - load average: 14.69, 14.51, 15.28 [03:18:01] Load avg. on willow is OK: OK - load average: 11.49, 13.77, 14.81 [03:20:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:26:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [03:27:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:29:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 581019 MB (10% inode=49%): [03:31:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:37:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 271838.000000 [03:45:01] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:46:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:46:39] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:03:09] Load avg. on willow is WARNING: WARNING - load average: 12.98, 15.03, 14.38 [04:04:10] Load avg. on willow is OK: OK - load average: 10.86, 14.07, 14.08 [04:08:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.285156/1.10, alarm hl:np_load_long=0.752930/1.55, alarm hl:mem_free=17838.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.285156/1.00, alarm hl:np_load_long=0.752930/1.50, alarm hl:mem_free=17838.000000M/350M, alarm hl:available=1/0 [04:13:11] Load avg. on willow is WARNING: WARNING - load average: 15.20, 14.96, 14.44 [04:14:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.630859/1.75, alarm hl:np_load_avg=1.802246/2.0, alarm hl:mem_free=347.000000M/350M, alarm hl:available=1/0 [04:21:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:23:10] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:23:52] 3(created) [TS-1348] Replicated zhwiki_p seems broken.; Toolserver: Databases; Bug <10https://jira.toolserver.org/browse/TS-1348> (Liangent) [04:27:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [04:28:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:29:11] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580884 MB (10% inode=49%): [04:29:29] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:30:12] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.181641/1.10, alarm hl:np_load_long=1.077149/1.55, alarm hl:mem_free=16849.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.181641/1.00, alarm hl:np_load_long=1.077149/1.50, alarm hl:mem_free=16849.000000M/350M, alarm hl:available=1/0 [04:31:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:32:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.940430/1.75, alarm hl:np_load_avg=2.019043/2.0, alarm hl:mem_free=400.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.940430/1.9, alarm hl:np_load_long=1.903809/2.25, alarm hl:mem_free=400.000000M/200M, alarm hl:available=1/0 [04:33:14] [[Special:Log/newusers]] create 10 * SatuSuro * (New user account) [04:33:28] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:34:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:37:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 275447.000000 [04:45:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:46:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:46:39] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:02:10] Load avg. on willow is WARNING: WARNING - load average: 16.70, 16.45, 15.26 [05:09:10] Load avg. on willow is OK: OK - load average: 12.62, 14.68, 15.00 [05:13:10] Load avg. on willow is WARNING: WARNING - load average: 15.14, 15.42, 15.28 [05:21:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:27:10] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [05:28:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:29:19] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580823 MB (10% inode=49%): [05:31:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:33:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.973633/1.75, alarm hl:np_load_avg=2.308594/2.0, alarm hl:mem_free=562.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.973633/1.9, alarm hl:np_load_long=2.091797/2.25, alarm hl:mem_free=562.000000M/200M, alarm hl:available=1/0 [05:38:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 279108.000000 [05:42:09] Load avg. on willow is WARNING: WARNING - load average: 13.32, 15.63, 16.22 [05:43:18] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.170899/1.10, alarm hl:np_load_long=0.720703/1.55, alarm hl:mem_free=19076.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.170899/1.00, alarm hl:np_load_long=0.720703/1.50, alarm hl:mem_free=19076.000000M/350M, alarm hl:available=1/0 [05:44:19] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:45:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:46:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:46:38] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:53:29] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:56:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.926758/1.75, alarm hl:np_load_avg=1.961426/2.0, alarm hl:mem_free=600.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.926758/1.9, alarm hl:np_load_long=2.019531/2.25, alarm hl:mem_free=600.000000M/200M, alarm hl:available=1/0 [05:59:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:21:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:24:19] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.188476/1.10, alarm hl:np_load_long=0.841797/1.55, alarm hl:mem_free=19241.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.188476/1.00, alarm hl:np_load_long=0.841797/1.50, alarm hl:mem_free=19241.000000M/350M, alarm hl:available=1/0 [06:25:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:27:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [06:29:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:29:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580692 MB (10% inode=49%): [06:31:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:34:19] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.101562/1.10, alarm hl:np_load_long=0.859375/1.55, alarm hl:mem_free=19250.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.101562/1.00, alarm hl:np_load_long=0.859375/1.50, alarm hl:mem_free=19250.000000M/350M, alarm hl:available=1/0 [06:38:20] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 282712.000000 [06:42:20] Load avg. on willow is WARNING: WARNING - load average: 13.75, 17.55, 18.21 [06:43:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.881348/1.75, alarm hl:np_load_avg=2.181152/2.0, alarm hl:mem_free=337.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.881348/1.9, alarm hl:np_load_long=2.268066/2.25, alarm hl:mem_free=337.000000M/200M, alarm hl:available=1/0 [06:45:19] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:46:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:46:39] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:54:29] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:58:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.409668/1.75, alarm hl:np_load_avg=2.574707/2.0, alarm hl:mem_free=357.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.409668/1.9, alarm hl:np_load_long=2.354004/2.25, alarm hl:mem_free=357.000000M/200M, alarm hl:available=1/0 [06:59:20] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.95, 23.40, 19.96 [07:22:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:27:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [07:29:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:30:19] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580575 MB (10% inode=49%): [07:31:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:38:20] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 286312.000000 [07:39:08] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:42:20] Load avg. on willow is WARNING: WARNING - load average: 14.15, 17.17, 19.90 [07:45:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:46:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:46:40] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:49:19] Load avg. on willow is CRITICAL: CRITICAL - load average: 29.52, 21.24, 20.26 [07:58:30] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.666016/1.10, alarm hl:np_load_long=1.000000/1.55, alarm hl:mem_free=18711.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.666016/1.00, alarm hl:np_load_long=1.000000/1.50, alarm hl:mem_free=18711.000000M/350M, alarm hl:available=1/0 [07:58:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.186035/1.75, alarm hl:np_load_avg=3.000488/2.0, alarm hl:mem_free=397.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.186035/1.9, alarm hl:np_load_long=2.730957/2.25, alarm hl:mem_free=397.000000M/200M, alarm hl:available=1/0 [08:00:29] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:07:21] Load avg. on willow is WARNING: WARNING - load average: 13.67, 17.80, 19.83 [08:14:40] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:22:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:22:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.878906/1.75, alarm hl:np_load_avg=1.803711/2.0, alarm hl:mem_free=530.000000M/350M, alarm hl:available=1/0 [08:24:40] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:27:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [08:28:59] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:29:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:30:20] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580352 MB (10% inode=49%): [08:31:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:38:19] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 289918.000000 [08:45:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:46:49] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:47:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:57:20] Load avg. on willow is OK: OK - load average: 13.00, 14.23, 14.92 [09:02:21] Load avg. on willow is WARNING: WARNING - load average: 15.61, 15.88, 15.36 [09:19:22] Load avg. on willow is OK: OK - load average: 12.67, 14.03, 14.86 [09:22:23] Load avg. on willow is WARNING: WARNING - load average: 13.02, 14.91, 15.16 [09:22:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:24:22] Load avg. on willow is OK: OK - load average: 14.20, 14.44, 14.93 [09:27:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [09:29:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:30:23] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580022 MB (10% inode=49%): [09:31:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:38:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 293522.000000 [09:45:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:46:51] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:47:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:57:33] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.025391/1.00, alarm hl:np_load_long=0.764648/1.50, alarm hl:mem_free=19061.000000M/350M, alarm hl:available=1/0 [09:58:21] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:58:32] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:58:51] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:09:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:17:32] Load avg. on willow is WARNING: WARNING - load average: 11.29, 14.27, 15.07 [10:18:33] Load avg. on willow is OK: OK - load average: 12.28, 14.03, 14.94 [10:22:33] Load avg. on willow is WARNING: WARNING - load average: 14.00, 14.84, 15.12 [10:22:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:24:31] Load avg. on willow is OK: OK - load average: 13.71, 14.32, 14.88 [10:27:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [10:28:41] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:29:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:30:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579894 MB (10% inode=49%): [10:32:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:38:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 297126.000000 [10:45:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:46:52] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:48:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:05:32] Load avg. on willow is WARNING: WARNING - load average: 20.87, 17.25, 16.34 [11:05:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.564941/1.75, alarm hl:np_load_avg=2.134277/2.0, alarm hl:mem_free=440.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.564941/1.9, alarm hl:np_load_long=2.034180/2.25, alarm hl:mem_free=440.000000M/200M, alarm hl:available=1/0 [11:19:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:22:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:22:52] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.234375/1.75, alarm hl:np_load_avg=2.083008/2.0, alarm hl:mem_free=642.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.234375/1.9, alarm hl:np_load_long=2.099121/2.25, alarm hl:mem_free=642.000000M/200M, alarm hl:available=1/0 [11:24:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:27:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [11:29:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:30:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579724 MB (10% inode=49%): [11:32:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:38:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 300726.000000 [11:38:35] Load avg. on willow is OK: OK - load average: 12.29, 13.91, 14.99 [11:46:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:47:51] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [11:49:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:52:33] Load avg. on willow is WARNING: WARNING - load average: 15.90, 15.57, 15.01 [11:53:33] Load avg. on willow is OK: OK - load average: 13.32, 14.90, 14.81 [12:04:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:17:33] Load avg. on willow is WARNING: WARNING - load average: 14.29, 16.25, 16.14 [12:22:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:27:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [12:29:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:30:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579493 MB (10% inode=49%): [12:32:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:33:33] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.318359/1.10, alarm hl:np_load_long=0.708008/1.55, alarm hl:mem_free=19341.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.318359/1.00, alarm hl:np_load_long=0.708008/1.50, alarm hl:mem_free=19341.000000M/350M, alarm hl:available=1/0 [12:33:41] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:34:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:39:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 304387.000000 [12:43:32] Load avg. on willow is OK: OK - load average: 12.03, 13.86, 14.86 [12:46:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:47:32] Load avg. on willow is WARNING: WARNING - load average: 13.63, 14.73, 15.04 [12:48:51] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:49:32] Load avg. on willow is OK: OK - load average: 14.74, 14.64, 14.96 [12:49:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:22:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:27:12] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [13:29:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:30:41] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579376 MB (10% inode=49%): [13:32:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:40:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 308050.000000 [13:46:51] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:48:50] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:49:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:52:43] Load avg. on willow is WARNING: WARNING - load average: 14.58, 15.20, 14.54 [13:53:41] Load avg. on willow is OK: OK - load average: 13.17, 14.74, 14.43 [14:22:51] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:27:12] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [14:29:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:30:42] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579292 MB (10% inode=49%): [14:32:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:32:41] Load avg. on willow is WARNING: WARNING - load average: 14.97, 16.18, 14.85 [14:34:41] Load avg. on willow is OK: OK - load average: 11.54, 14.61, 14.43 [14:41:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 311712.000000 [14:46:52] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:48:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [14:49:52] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:22:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:28:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [15:29:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:30:41] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579170 MB (10% inode=49%): [15:32:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:32:41] Load avg. on willow is WARNING: WARNING - load average: 18.41, 16.75, 14.80 [15:37:41] Load avg. on willow is OK: OK - load average: 13.70, 14.87, 14.50 [15:41:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 315312.000000 [15:42:41] Load avg. on willow is WARNING: WARNING - load average: 17.01, 15.90, 14.99 [15:44:01] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.005859/1.00, alarm hl:np_load_long=0.739258/1.50, alarm hl:mem_free=18800.000000M/350M, alarm hl:available=1/0 [15:45:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:47:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:49:02] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [15:50:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:23:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:24:01] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.018555/1.00, alarm hl:np_load_long=0.770508/1.50, alarm hl:mem_free=19073.000000M/350M, alarm hl:available=1/0 [16:25:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [16:28:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [16:30:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:30:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 578540 MB (10% inode=49%): [16:32:21] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:33:01] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.267578/1.10, alarm hl:np_load_long=0.856445/1.55, alarm hl:mem_free=18669.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.267578/1.00, alarm hl:np_load_long=0.856445/1.50, alarm hl:mem_free=18669.000000M/350M, alarm hl:available=1/0 [16:41:41] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 318913.000000 [16:47:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:50:02] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [16:50:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:52:02] @replag [16:52:03] Danny_B|backup: s2-user: 55s [+0.00 s/s] [17:01:54] 3(created) [TS-1349] modify cronie of mmp giftbot; Toolserver; Trivial Task <10https://jira.toolserver.org/browse/TS-1349> (gifti) [17:02:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.784668/1.75, alarm hl:np_load_avg=2.220215/2.0, alarm hl:mem_free=973.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.784668/1.9, alarm hl:np_load_long=1.887695/2.25, alarm hl:mem_free=973.000000M/200M, alarm hl:available=1/0 [17:02:52] Load avg. on willow is WARNING: WARNING - load average: 16.63, 16.88, 14.96 [17:05:06] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:08:51] Load avg. on willow is OK: OK - load average: 12.83, 14.87, 14.71 [17:12:51] Load avg. on willow is WARNING: WARNING - load average: 14.46, 15.33, 14.97 [17:13:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.803223/1.75, alarm hl:np_load_avg=1.915039/2.0, alarm hl:mem_free=324.000000M/350M, alarm hl:available=1/0 [17:24:01] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.044922/1.00, alarm hl:np_load_long=0.878906/1.50, alarm hl:mem_free=18244.000000M/350M, alarm hl:available=1/0 [17:24:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:25:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:28:20] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [17:30:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:30:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 578280 MB (10% inode=49%): [17:32:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:42:38] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 322574.000000 [17:47:17] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:50:17] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [17:50:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:02:17] Load avg. on willow is WARNING: WARNING - load average: 16.58, 16.11, 14.62 [18:03:18] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.775391/1.75, alarm hl:np_load_avg=1.949707/2.0, alarm hl:mem_free=931.000000M/350M, alarm hl:available=1/0 [18:04:58] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:07:15] Load avg. on willow is OK: OK - load average: 14.52, 14.75, 14.41 [18:09:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.763672/1.75, alarm hl:np_load_avg=1.868164/2.0, alarm hl:mem_free=533.000000M/350M, alarm hl:available=1/0 [18:10:15] Load avg. on willow is WARNING: WARNING - load average: 19.47, 15.85, 14.88 [18:24:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:28:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [18:30:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:31:14] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 578133 MB (10% inode=49%): [18:33:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:38:56] 3(assigned) [TS-1349] modify cronie of mmp giftbot <10https://jira.toolserver.org/browse/TS-1349> (DaB.) [18:40:54] 3(resolved) [TS-1349] modify cronie of mmp giftbot <10https://jira.toolserver.org/browse/TS-1349> (DaB.) [18:42:57] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 326193.000000 [18:47:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:50:57] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [18:50:57] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:06:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.831543/1.75, alarm hl:np_load_avg=2.089355/2.0, alarm hl:mem_free=398.000000M/350M, alarm hl:available=1/0 [19:07:58] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:10:15] Load avg. on willow is WARNING: WARNING - load average: 20.52, 16.61, 17.00 [19:11:56] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.313965/1.75, alarm hl:np_load_avg=2.147949/2.0, alarm hl:mem_free=317.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.313965/1.9, alarm hl:np_load_long=2.150391/2.25, alarm hl:mem_free=317.000000M/200M, alarm hl:available=1/0 [19:21:58] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:24:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:28:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [19:30:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:31:15] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 578022 MB (10% inode=49%): [19:33:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:36:27] any TS users around? [19:37:05] 06 14:40:12 < jeremyb> could someone please `git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git` in an arbitrary (temp) dir on willow and tell me what happens? [19:37:10] 06 14:40:21 < jeremyb> see https://bugzilla.wikimedia.org/35709 [19:39:15] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:43:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 329799.000000 [19:44:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.884277/1.75, alarm hl:np_load_avg=2.068359/2.0, alarm hl:mem_free=849.000000M/350M, alarm hl:available=1/0 [19:48:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:48:23] mzmcbride@willow:~$ git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git [19:48:26] Cloning into core... [19:48:28] error: SSL certificate problem, verify that the CA cert is OK. Details: [19:48:31] error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https://gerrit.wikimedia.org/r/p/mediawiki/core.git/info/refs [19:48:35] fatal: HTTP request failed [19:51:55] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:51:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:59:06] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:01:56] @replag [20:01:57] Joan: No replag found. Use "@replag all" to see all lags. [20:02:00] @replag all [20:02:01] Joan: s1-rr-a: 1s [-0.00 s/s]; s1-user: 1s [-0.00 s/s]; s2-user: 1s [-0.00 s/s]; s2-user-c: 4s [-0.00 s/s]; s3-rr-a: 9s [-0.00 s/s]; s3-user: 9s [-0.00 s/s]; s4-rr-a: 4s [-0.00 s/s]; s4-user: 4s [-0.00 s/s] [20:02:02] Joan: s5-rr-a: 1s [-0.00 s/s]; s5-user: 1s [-0.00 s/s]; s5-user-c: 4s [-0.00 s/s]; s6-rr-a: 2s [-0.00 s/s]; s6-user: 2s [-0.00 s/s]; s7-rr-a: 1s [-0.00 s/s]; s7-user: 1s [-0.00 s/s] [20:02:06] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.211426/1.75, alarm hl:np_load_avg=2.221191/2.0, alarm hl:mem_free=1048.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.211426/1.9, alarm hl:np_load_long=2.146484/2.25, alarm hl:mem_free=1048.000000M/200M, alarm hl:available=1/0 [20:10:15] Load avg. on willow is WARNING: WARNING - load average: 18.01, 16.71, 16.87 [20:13:22] Joan: this is debian? [20:14:10] willow? [20:14:19] yes [20:14:21] mzmcbride@willow:~/scripts/database-reports$ uname -a [20:14:21] SunOS willow 5.10 Generic_147441-09 i86pc i386 i86pc Solaris [20:14:27] hrmm... [20:14:37] Only Debian host is nightshade, I think? [20:14:40] And it's still offline. [20:14:43] i've no idea [20:14:55] can you look for a /etc/ssl/certs or equivalent? [20:15:49] and then try something like GIT_SSL_CAPATH=/etc/path/to/certs git clone https://gerrit.wikimedia.org/r/p/mediawiki/core.git [20:16:59] * jeremyb has to run away in a few mins [20:20:34] * jeremyb runs away [20:23:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:25:06] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:27:06] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.211914/1.75, alarm hl:np_load_avg=2.085449/2.0, alarm hl:mem_free=845.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.211914/1.9, alarm hl:np_load_long=2.098633/2.25, alarm hl:mem_free=845.000000M/200M, alarm hl:available=1/0 [20:29:56] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [20:30:54] 3(commented) [ACCAPP-481] I am active user in http://fa.wiktionary.org and i want this account to run bot on this wiki also updating it's statics. <10https://jira.toolserver.org/browse/ACCAPP-481> (Darafsh Kaviyani) [20:31:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:31:15] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 577854 MB (10% inode=49%): [20:33:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:33:57] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:35:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:43:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 333404.000000 [20:48:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:52:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:52:55] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:10:26] Load avg. on willow is WARNING: WARNING - load average: 24.38, 17.46, 17.03 [21:13:08] [[User:Erdemaslancan]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=6997&rcid=9217 * Erdemaslancan * (+3) (Created page with "...") [21:26:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:30:55] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [21:31:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 577702 MB (10% inode=49%): [21:32:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:33:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:43:06] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 337003.000000 [21:48:15] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:52:55] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:53:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:04:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.115723/1.75, alarm hl:np_load_avg=2.171387/2.0, alarm hl:mem_free=291.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.115723/1.9, alarm hl:np_load_long=2.093750/2.25, alarm hl:mem_free=291.000000M/200M, alarm hl:available=1/0 [22:10:25] Load avg. on willow is WARNING: WARNING - load average: 28.10, 19.11, 17.57 [22:18:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:23:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.884766/1.75, alarm hl:np_load_avg=1.970215/2.0, alarm hl:mem_free=380.000000M/350M, alarm hl:available=1/0 [22:24:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:27:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:27:25] Load avg. on willow is OK: OK - load average: 9.86, 12.54, 14.84 [22:30:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [22:31:26] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 577569 MB (10% inode=49%): [22:32:26] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:33:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:35:25] Load avg. on willow is WARNING: WARNING - load average: 22.38, 16.02, 15.29 [22:37:15] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.169922/1.10, alarm hl:np_load_long=0.981445/1.55, alarm hl:mem_free=18731.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.169922/1.00, alarm hl:np_load_long=0.981445/1.50, alarm hl:mem_free=18731.000000M/350M, alarm hl:available=1/0 [22:39:15] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [22:43:15] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 340609.000000 [22:48:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:52:56] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [22:53:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:27:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:31:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [23:32:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 577514 MB (10% inode=49%): [23:33:26] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:33:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:35:25] Load avg. on willow is WARNING: WARNING - load average: 17.84, 15.52, 15.58 [23:38:27] Load avg. on willow is OK: OK - load average: 12.79, 14.10, 14.98 [23:42:28] Load avg. on willow is WARNING: WARNING - load average: 17.01, 15.59, 15.36 [23:43:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 344214.000000 [23:48:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:53:18] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [23:53:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:57:56] 3(resolved) [DBQ-180] Database Query (question) <10https://jira.toolserver.org/browse/DBQ-180> (Hoo man)