[00:02:13] Load avg. on willow is WARNING: WARNING - load average: 23.04, 19.38, 16.17 [00:07:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:10:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.75, 22.38, 18.52 [00:11:37] Load avg. on willow is WARNING: WARNING - load average: 22.45, 21.05, 18.29 [00:12:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [00:14:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31285.000000 [00:16:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:19:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:23:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87429.000000 [00:30:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.00, 21.43, 19.12 [00:30:52] 3(updated) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [00:31:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [00:31:35] Load avg. on willow is WARNING: WARNING - load average: 21.08, 20.34, 18.88 [00:32:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:34:36] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30501 [00:35:03] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:43:46] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.485351/1.10, alarm hl:np_load_long=0.797852/1.55, alarm hl:mem_free=19347.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.485351/1.00, alarm hl:np_load_long=0.797852/1.50, alarm hl:mem_free=19347.000000M/350M, alarm hl:available=1/0 [00:44:06] @replag [00:44:07] Betacommand: s1-rr-a: 8h 23m 31s [-0.84 s/s]; s1-user: 8h 23m 31s [-0.84 s/s]; s3-rr-a: 37s [-0.00 s/s]; s3-user: 37s [-0.00 s/s]; s6-rr-a: 1m 1s [+0.00 s/s]; s6-user: 1m 1s [+0.00 s/s] [00:45:46] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [00:53:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:01:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.736328/1.75, alarm hl:np_load_avg=2.027832/2.0, alarm hl:mem_free=838.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.736328/1.9, alarm hl:np_load_long=2.031738/2.25, alarm hl:mem_free=838.000000M/200M, alarm hl:available=1/0 [01:03:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:07:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:13:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [01:15:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28747.000000 [01:16:20] j #ubuntu [01:17:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:19:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:23:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91031.000000 [01:27:56] @replag [01:27:57] Thehelpfulone: s1-rr-a: 7h 56m 5s [-0.63 s/s]; s1-user: 7h 56m 5s [-0.63 s/s]; s2-user-c: 14s [-0.01 s/s]; s3-rr-a: 20s [-0.01 s/s]; s3-user: 20s [-0.01 s/s]; s4-rr-a: 14s [-]; s4-user: 14s [-]; s5-rr-a: 13s [-] [01:27:58] Thehelpfulone: s5-user: 13s [-]; s5-user-c: 14s [-0.01 s/s] [01:31:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:32:38] Load avg. on willow is WARNING: WARNING - load average: 14.92, 15.56, 15.75 [01:32:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:35:03] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:35:34] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28450 [01:39:36] Load avg. on willow is OK: OK - load average: 12.95, 13.86, 14.89 [01:42:34] Load avg. on willow is WARNING: WARNING - load average: 16.43, 15.11, 15.21 [01:57:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.325684/1.75, alarm hl:np_load_avg=2.214355/2.0, alarm hl:mem_free=305.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.325684/1.9, alarm hl:np_load_long=2.055664/2.25, alarm hl:mem_free=305.000000M/200M, alarm hl:available=1/0 [02:00:34] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.81, 20.95, 17.79 [02:01:35] Load avg. on willow is WARNING: WARNING - load average: 20.29, 19.56, 17.50 [02:07:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:13:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [02:14:04] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:15:34] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27246.000000 [02:17:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:18:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.005859/1.75, alarm hl:np_load_avg=2.060059/2.0, alarm hl:mem_free=384.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.005859/1.9, alarm hl:np_load_long=2.121582/2.25, alarm hl:mem_free=384.000000M/200M, alarm hl:available=1/0 [02:19:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:24:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94692.000000 [02:31:13] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:32:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:35:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:35:35] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26171 [02:50:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:53:34] Load avg. on willow is OK: OK - load average: 10.79, 12.79, 14.75 [02:57:35] Load avg. on willow is WARNING: WARNING - load average: 16.67, 14.94, 15.19 [02:58:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.012207/1.75, alarm hl:np_load_avg=1.864746/2.0, alarm hl:mem_free=317.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.012207/1.9, alarm hl:np_load_long=1.896484/2.25, alarm hl:mem_free=317.000000M/200M, alarm hl:available=1/0 [03:00:33] Load avg. on willow is CRITICAL: CRITICAL - load average: 37.37, 20.64, 17.15 [03:07:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:12:04] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:13:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [03:15:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.785645/1.75, alarm hl:np_load_avg=1.934570/2.0, alarm hl:mem_free=553.000000M/350M, alarm hl:available=1/0 [03:15:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23080.000000 [03:16:36] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 991 MB (18% inode=99%): [03:17:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:19:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:20:35] Load avg. on willow is WARNING: WARNING - load average: 15.11, 14.98, 15.61 [03:22:34] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1240 MB (21% inode=99%): [03:25:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98352.000000 [03:31:13] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:33:04] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:33:34] Load avg. on willow is OK: OK - load average: 11.52, 13.89, 14.84 [03:33:34] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 1054 MB (19% inode=99%): [03:35:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:36:34] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20911 [03:47:45] Load avg. on willow is WARNING: WARNING - load average: 15.92, 15.51, 15.04 [03:50:57] @replag [03:50:57] Dispenser: s1-rr-a: 5h 29m 6s [-1.03 s/s]; s1-user: 5h 29m 6s [-1.03 s/s]; s2-user: 26s [+0.00 s/s]; s3-rr-a: 16s [-0.00 s/s]; s3-user: 16s [-0.00 s/s] [03:51:44] Load avg. on willow is OK: OK - load average: 12.21, 14.74, 14.88 [03:53:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.947754/1.75, alarm hl:np_load_avg=1.906738/2.0, alarm hl:mem_free=179.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.947754/1.9, alarm hl:np_load_long=1.881836/2.25, alarm hl:mem_free=179.000000M/200M, alarm hl:available=1/0 [04:05:04] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:08:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:11:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.631836/1.75, alarm hl:np_load_avg=2.173828/2.0, alarm hl:mem_free=364.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.631836/1.9, alarm hl:np_load_long=2.029297/2.25, alarm hl:mem_free=364.000000M/200M, alarm hl:available=1/0 [04:13:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [04:15:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16842.000000 [04:17:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:20:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:20:44] Load avg. on willow is WARNING: WARNING - load average: 16.93, 15.71, 15.80 [04:26:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102014.000000 [04:31:13] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:33:05] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:35:17] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:35:17] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:36:34] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16766 [04:39:55] Load avg. on willow is OK: OK - load average: 10.24, 13.40, 14.84 [04:55:24] @replag [04:55:25] Joan: s1-rr-a: 4h 29m 2s [-0.93 s/s]; s1-user: 4h 29m 3s [-0.93 s/s]; s3-rr-a: 46s [+0.01 s/s]; s3-user: 46s [+0.01 s/s] [05:01:02] Ja Ga * [Toolserver-l] use of include_once in PHP changed [05:05:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.944824/1.75, alarm hl:np_load_avg=1.853027/2.0, alarm hl:mem_free=637.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.944824/1.9, alarm hl:np_load_long=1.826172/2.25, alarm hl:mem_free=637.000000M/200M, alarm hl:available=1/0 [05:05:55] Load avg. on willow is WARNING: WARNING - load average: 19.36, 16.26, 15.14 [05:08:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:08:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:08:55] Load avg. on willow is OK: OK - load average: 12.70, 14.57, 14.67 [05:14:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [05:15:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14552.000000 [05:17:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:17:14] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.002441/1.75, alarm hl:np_load_avg=1.914062/2.0, alarm hl:mem_free=158.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.002441/1.9, alarm hl:np_load_long=1.859863/2.25, alarm hl:mem_free=158.000000M/200M, alarm hl:available=1/0 [05:17:55] Load avg. on willow is WARNING: WARNING - load average: 16.65, 15.56, 14.98 [05:20:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:26:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105613.000000 [05:28:14] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:31:14] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:33:14] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:36:13] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:36:34] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13590 [05:44:55] Load avg. on willow is WARNING: WARNING - load average: 19.43, 20.82, 19.20 [05:50:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.073242/1.75, alarm hl:np_load_avg=2.235840/2.0, alarm hl:mem_free=156.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.073242/1.9, alarm hl:np_load_long=2.294922/2.25, alarm hl:mem_free=156.000000M/200M, alarm hl:available=1/0 [06:08:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:10:53] Load avg. on willow is CRITICAL: CRITICAL - load average: 38.28, 23.32, 20.29 [06:14:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [06:16:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8611.000000 [06:17:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:20:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:26:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109217.000000 [06:31:23] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:33:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:36:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:36:34] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3562 [06:36:53] Load avg. on willow is WARNING: WARNING - load average: 15.43, 18.34, 19.97 [06:36:53] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3517.000000 [06:43:53] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [06:44:34] MySQL slave on thyme is OK: Uptime: 560472 Threads: 20 Questions: 81062432 Slow queries: 83023 Opens: 18509 Flush tables: 3 Open tables: 2577 Queries per second avg: 144.632 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1684 [06:44:54] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.61, 21.18, 20.13 [06:51:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.349121/1.75, alarm hl:np_load_avg=3.153809/2.0, alarm hl:mem_free=139.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.349121/1.9, alarm hl:np_load_long=2.815918/2.25, alarm hl:mem_free=139.000000M/200M, alarm hl:available=1/0 [06:52:02] Marlen Caemmerer * Re: [Toolserver-l] use of include_once in PHP changed [06:52:47] @replag [06:52:47] Dispenser: s1-rr-a: 1m 56s [-2.28 s/s]; s1-user: 1m 56s [-2.28 s/s]; s3-rr-a: 1m 15s [+0.00 s/s]; s3-user: 1m 15s [+0.00 s/s] [06:59:53] Load avg. on willow is WARNING: WARNING - load average: 16.89, 17.96, 19.93 [07:00:52] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.51, 20.87, 20.86 [07:08:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:10:39] @replag [07:10:40] Joan: s2-user: 17s [-0.00 s/s]; s7-rr-a: 12s [-]; s7-user: 12s [-] [07:10:43] o: [07:10:49] Vicotry. [07:10:51] Victory, even. [07:14:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [07:17:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:20:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:25:54] Load avg. on willow is WARNING: WARNING - load average: 21.65, 19.36, 18.68 [07:26:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112820.000000 [07:31:24] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:33:53] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 1015 MB (18% inode=99%): [07:34:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:34:54] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1408 MB (23% inode=99%): [07:37:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:44:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:48:53] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 1156 MB (20% inode=99%): [07:51:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.091797/1.75, alarm hl:np_load_avg=2.603516/2.0, alarm hl:mem_free=432.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.091797/1.9, alarm hl:np_load_long=2.329590/2.25, alarm hl:mem_free=432.000000M/200M, alarm hl:available=1/0 [07:59:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:02:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.495117/1.75, alarm hl:np_load_avg=2.323730/2.0, alarm hl:mem_free=521.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.495117/1.9, alarm hl:np_load_long=2.229004/2.25, alarm hl:mem_free=521.000000M/200M, alarm hl:available=1/0 [08:04:06] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:08:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:08:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:14:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [08:17:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:17:46] Hi. Is it possible to connect to a ts project jira from netbeans? if so: how is the jira url assembled? [08:19:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:19:08] I got it to work for the complete jira server, and I am able to add issues now [08:21:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:25:54] Load avg. on willow is WARNING: WARNING - load average: 15.84, 16.28, 16.46 [08:26:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116425.000000 [08:31:23] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:31:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.183594/1.75, alarm hl:np_load_avg=2.035645/2.0, alarm hl:mem_free=226.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.183594/1.9, alarm hl:np_load_long=2.034180/2.25, alarm hl:mem_free=226.000000M/200M, alarm hl:available=1/0 [08:34:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:37:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:39:24] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:42:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.902344/1.75, alarm hl:np_load_avg=1.905762/2.0, alarm hl:mem_free=274.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.902344/1.9, alarm hl:np_load_long=1.945801/2.25, alarm hl:mem_free=274.000000M/200M, alarm hl:available=1/0 [08:52:25] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:57:25] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:08:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:14:35] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.850586/1.75, alarm hl:np_load_avg=1.924805/2.0, alarm hl:mem_free=271.000000M/350M, alarm hl:available=1/0 [09:14:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [09:17:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:21:43] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:25:54] Load avg. on willow is WARNING: WARNING - load average: 14.63, 14.75, 15.29 [09:26:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120026.000000 [09:28:53] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 1172 MB (20% inode=99%): [09:31:35] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:31:53] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1920 MB (29% inode=99%): [09:34:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:37:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:42:54] Load avg. on willow is OK: OK - load average: 10.26, 13.54, 14.71 [09:57:45] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:57:55] Load avg. on willow is WARNING: WARNING - load average: 17.33, 15.54, 14.90 [10:08:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:14:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.680664/1.75, alarm hl:np_load_avg=2.584473/2.0, alarm hl:mem_free=163.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.680664/1.9, alarm hl:np_load_long=2.309082/2.25, alarm hl:mem_free=163.000000M/200M, alarm hl:available=1/0 [10:14:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [10:17:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:21:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:26:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123629.000000 [10:27:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:31:44] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:32:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.056152/1.75, alarm hl:np_load_avg=2.132324/2.0, alarm hl:mem_free=693.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.056152/1.9, alarm hl:np_load_long=2.137207/2.25, alarm hl:mem_free=693.000000M/200M, alarm hl:available=1/0 [10:33:45] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:34:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:37:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:41:53] 3(created) [ET-46] April Newsletter; JamesR's Tools; Minor Task <10https://jira.toolserver.org/browse/ET-46> (Keith Dorey) [10:43:43] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.217774/1.10, alarm hl:np_load_long=0.820312/1.55, alarm hl:mem_free=17931.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.217774/1.00, alarm hl:np_load_long=0.820312/1.50, alarm hl:mem_free=17931.000000M/350M, alarm hl:available=1/0 [10:44:44] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [10:55:05] [[Special:Log/newusers]] create 10 * Chihonglee * (New user account) [10:57:53] Load avg. on willow is WARNING: WARNING - load average: 15.58, 15.14, 15.87 [10:58:45] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:00:54] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.80, 20.08, 17.51 [11:01:54] Load avg. on willow is WARNING: WARNING - load average: 18.09, 18.42, 17.08 [11:08:58] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 1059 MB (19% inode=99%): [11:09:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:09:58] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1298 MB (22% inode=99%): [11:12:59] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 965 MB (17% inode=99%): [11:15:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [11:17:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:18:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.019043/1.75, alarm hl:np_load_avg=2.170410/2.0, alarm hl:mem_free=270.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.019043/1.9, alarm hl:np_load_long=2.168457/2.25, alarm hl:mem_free=270.000000M/200M, alarm hl:available=1/0 [11:21:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:27:50] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127288.000000 [11:32:39] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [11:35:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:37:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:39:04] hello all [11:41:21] hi [11:45:40] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.031250/1.00, alarm hl:np_load_long=0.838867/1.50, alarm hl:mem_free=17678.000000M/350M, alarm hl:available=1/0 [11:46:39] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:48:34] hi DaBPunkt [11:48:57] DaBPunkt: could you process https://jira.toolserver.org/browse/TS-1347 and https://jira.toolserver.org/browse/TS-1346 please? [11:54:02] Platonides * Re: [Toolserver-l] use of include_once in PHP changed [11:54:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:57:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.901367/1.75, alarm hl:np_load_avg=2.026367/2.0, alarm hl:mem_free=382.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.901367/1.9, alarm hl:np_load_long=2.066895/2.25, alarm hl:mem_free=382.000000M/200M, alarm hl:available=1/0 [11:57:53] 3(assigned) [TS-1347] sudo access to "unblock" user <10https://jira.toolserver.org/browse/TS-1347> (DaB.) [11:57:58] 3(assigned) [TS-1346] SVN write access for UTRS <10https://jira.toolserver.org/browse/TS-1346> (DaB.) [11:58:02] 3(resolved) [TS-1347] sudo access to "unblock" user <10https://jira.toolserver.org/browse/TS-1347> (DaB.) [11:58:03] 3(work started) [TS-1347] sudo access to "unblock" user <10https://jira.toolserver.org/browse/TS-1347> (DaB.) [11:58:06] 3(work started) [TS-1346] SVN write access for UTRS <10https://jira.toolserver.org/browse/TS-1346> (DaB.) [11:58:10] 3(resolved) [TS-1346] SVN write access for UTRS <10https://jira.toolserver.org/browse/TS-1346> (DaB.) [11:58:49] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:00:49] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:02:09] Load avg. on willow is WARNING: WARNING - load average: 17.08, 17.84, 17.17 [12:04:02] Marcin Cieslak * Re: [Toolserver-l] UnicodeEncodeError [12:10:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:16:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [12:17:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:18:09] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 1048 MB (18% inode=99%): [12:21:07] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1291 MB (22% inode=99%): [12:22:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:24:08] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 1132 MB (20% inode=99%): [12:28:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130903.000000 [12:28:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:33:39] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:33:40] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:36:10] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: /tmp 518 MB (10% inode=99%): [12:36:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:37:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:39:49] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.038086/1.00, alarm hl:np_load_long=0.893555/1.50, alarm hl:mem_free=17434.000000M/350M, alarm hl:available=1/0 [12:41:40] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:44:39] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.560547/1.10, alarm hl:np_load_long=1.056641/1.55, alarm hl:mem_free=18409.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.560547/1.00, alarm hl:np_load_long=1.056641/1.50, alarm hl:mem_free=18409.000000M/350M, alarm hl:available=1/0 [12:53:49] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.209473/1.10, alarm hl:np_load_long=0.285645/1.55, alarm hl:mem_free=174.000000M/300M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.209473/1.00, alarm hl:np_load_long=0.285645/1.50, alarm hl:mem_free=174.000000M/350M, alarm hl:available=1/0 [12:57:49] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [12:57:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.718262/1.75, alarm hl:np_load_avg=2.185059/2.0, alarm hl:mem_free=261.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.718262/1.9, alarm hl:np_load_long=2.332031/2.25, alarm hl:mem_free=261.000000M/200M, alarm hl:available=1/0 [13:03:09] Load avg. on willow is WARNING: WARNING - load average: 18.73, 20.36, 19.66 [13:03:49] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.260254/1.00, alarm hl:np_load_long=0.252441/1.50, alarm hl:mem_free=344.000000M/350M, alarm hl:available=1/0 [13:10:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:10:39] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.017578/1.00, alarm hl:np_load_long=0.935547/1.50, alarm hl:mem_free=16023.000000M/350M, alarm hl:available=1/0 [13:11:09] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.00, 21.90, 20.32 [13:11:40] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:14:38] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580601 MB (10% inode=49%): [13:16:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [13:17:39] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.077149/1.00, alarm hl:np_load_long=1.001953/1.50, alarm hl:mem_free=15940.000000M/350M, alarm hl:available=1/0 [13:17:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:19:08] Load avg. on willow is WARNING: WARNING - load average: 18.57, 19.84, 19.99 [13:23:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:24:39] /aux0 on hemlock is OK: DISK OK - free space: /aux0 619287 MB (11% inode=51%): [13:28:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134503.000000 [13:34:38] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:36:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:38:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:52:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:55:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.504883/1.75, alarm hl:np_load_avg=2.133301/2.0, alarm hl:mem_free=256.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.504883/1.9, alarm hl:np_load_long=2.188965/2.25, alarm hl:mem_free=256.000000M/200M, alarm hl:available=1/0 [14:10:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:16:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [14:17:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:19:08] Load avg. on willow is WARNING: WARNING - load average: 17.84, 18.64, 18.92 [14:23:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:29:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138164.000000 [14:31:38] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580569 MB (10% inode=49%): [14:33:54] 3(updated) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [14:33:54] 3(updated) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [14:33:54] 3(updated) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [14:33:55] 3(updated) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [14:34:39] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [14:35:51] 3(updated) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [14:35:51] 3(updated) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [14:36:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:39:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:52:54] 3(updated) [ACCAPP-487] Analysing the development of Bots <10https://jira.toolserver.org/browse/ACCAPP-487> (Franz Herbach) [14:52:55] 3(updated) [ACCAPP-487] Analysing the development of Bots <10https://jira.toolserver.org/browse/ACCAPP-487> (Franz Herbach) [14:54:53] 3(updated) [ACCAPP-461] Analysing the development of Bots <10https://jira.toolserver.org/browse/ACCAPP-461> (Franz Herbach) [14:54:54] 3(updated) [ACCAPP-461] Analysing the development of Bots <10https://jira.toolserver.org/browse/ACCAPP-461> (Franz Herbach) [14:54:55] 3(updated) [ACCAPP-461] Analysing the development of Bots <10https://jira.toolserver.org/browse/ACCAPP-461> (Franz Herbach) [14:55:21] DaBPunkt, you're adding the new accounts today, mhm? [14:55:23] @replag [14:55:27] Firebolt: s2-user: 32s [+0.00 s/s] [14:55:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.187500/1.75, alarm hl:np_load_avg=2.615234/2.0, alarm hl:mem_free=259.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.187500/1.9, alarm hl:np_load_long=2.476562/2.25, alarm hl:mem_free=259.000000M/200M, alarm hl:available=1/0 [15:00:21] who runs tsbot? [15:00:52] the ts admins, I'd assum [15:06:19] [[Logging in]] ! 10https://wiki.toolserver.org/w/index.php?diff=6995&oldid=5955&rcid=9212 * 70.28.63.126 * (+1) (Fixed typo) [15:10:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:13:48] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.041992/1.00, alarm hl:np_load_long=0.827148/1.50, alarm hl:mem_free=18626.000000M/350M, alarm hl:available=1/0 [15:14:48] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:17:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [15:17:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:19:09] Load avg. on willow is WARNING: WARNING - load average: 17.09, 18.96, 19.40 [15:23:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:29:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 141765.000000 [15:32:38] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 549725 MB (10% inode=48%): [15:34:41] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [15:36:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:39:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:44:48] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.004883/1.00, alarm hl:np_load_long=0.856445/1.50, alarm hl:mem_free=19052.000000M/350M, alarm hl:available=1/0 [15:45:49] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:55:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.464844/1.75, alarm hl:np_load_avg=2.410645/2.0, alarm hl:mem_free=136.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.464844/1.9, alarm hl:np_load_long=2.397949/2.25, alarm hl:mem_free=136.000000M/200M, alarm hl:available=1/0 [16:08:18] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.27, 21.52, 20.16 [16:08:54] 3(commented) [UTRS-78] Many links have interfering JavaScript <10https://jira.toolserver.org/browse/UTRS-78> (Chris Howie) [16:10:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:17:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [16:17:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:24:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:30:07] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 145425.000000 [16:30:51] /aux0 on hemlock is OK: DISK OK - free space: /aux0 599007 MB (11% inode=50%): [16:34:50] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [16:36:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:40:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:43:18] Load avg. on willow is WARNING: WARNING - load average: 17.07, 18.46, 19.91 [16:46:18] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.63, 20.23, 20.27 [16:54:18] Load avg. on willow is WARNING: WARNING - load average: 16.36, 19.01, 19.84 [16:56:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.457031/1.75, alarm hl:np_load_avg=2.428223/2.0, alarm hl:mem_free=270.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.457031/1.9, alarm hl:np_load_long=2.487305/2.25, alarm hl:mem_free=270.000000M/200M, alarm hl:available=1/0 [17:01:19] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.48, 20.93, 20.24 [17:11:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:17:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [17:18:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:24:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:30:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 149025.000000 [17:33:59] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:34:50] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [17:35:19] Load avg. on willow is WARNING: WARNING - load average: 17.03, 15.98, 17.31 [17:36:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:37:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.918457/1.75, alarm hl:np_load_avg=1.961914/2.0, alarm hl:mem_free=797.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.918457/1.9, alarm hl:np_load_long=2.126953/2.25, alarm hl:mem_free=797.000000M/200M, alarm hl:available=1/0 [17:40:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:42:58] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:11:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:18:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [18:19:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:21:16] @replag [18:21:16] Sebaso_WMDE: s3-rr-a: 49s [-0.00 s/s]; s3-user: 49s [-0.00 s/s]; s5-rr-a: 10s [-0.00 s/s]; s5-user: 10s [-0.00 s/s] [18:24:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:30:08] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 152628.000000 [18:35:48] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [18:36:19] Load avg. on willow is WARNING: WARNING - load average: 15.39, 16.61, 16.93 [18:36:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:40:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:06:09] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.421875/1.75, alarm hl:np_load_avg=2.091797/2.0, alarm hl:mem_free=154.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.421875/1.9, alarm hl:np_load_long=2.084473/2.25, alarm hl:mem_free=154.000000M/200M, alarm hl:available=1/0 [19:12:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:18:19] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [19:20:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:24:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:25:08] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:30:20] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 156232.000000 [19:30:20] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.19, 18.73, 17.30 [19:31:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.917969/1.75, alarm hl:np_load_avg=2.459473/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.917969/1.9, alarm hl:np_load_long=2.208008/2.25, alarm hl:mem_free=125.000000M/200M, alarm hl:available=1/0 [19:31:19] Load avg. on willow is WARNING: WARNING - load average: 26.41, 19.78, 17.78 [19:35:50] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:38:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:41:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:48:02] Ja Ga * Re: [Toolserver-l] use of include_once in PHP changed [20:03:45] nighty~ [20:03:58] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:12:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:15:19] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.65, 21.67, 19.50 [20:16:19] Load avg. on willow is WARNING: WARNING - load average: 23.04, 21.32, 19.53 [20:18:20] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [20:20:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:24:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:30:19] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 159834.000000 [20:30:20] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.36, 21.00, 19.62 [20:31:21] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.077637/1.75, alarm hl:np_load_avg=2.589355/2.0, alarm hl:mem_free=125.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.077637/1.9, alarm hl:np_load_long=2.449219/2.25, alarm hl:mem_free=125.000000M/200M, alarm hl:available=1/0 [20:35:49] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:38:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:41:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:43:38] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:50:21] Load avg. on willow is WARNING: WARNING - load average: 28.62, 20.45, 18.82 [20:52:08] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.395508/1.10, alarm hl:np_load_long=0.822265/1.55, alarm hl:mem_free=19261.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.395508/1.00, alarm hl:np_load_long=0.822265/1.50, alarm hl:mem_free=19261.000000M/350M, alarm hl:available=1/0 [20:53:08] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:00:20] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.27, 21.15, 19.27 [21:01:21] Load avg. on willow is WARNING: WARNING - load average: 23.34, 21.09, 19.39 [21:12:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:18:19] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [21:20:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:24:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:29:23] nacht ts [21:30:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 163435.000000 [21:32:20] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.305664/1.75, alarm hl:np_load_avg=2.370117/2.0, alarm hl:mem_free=347.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.305664/1.9, alarm hl:np_load_long=2.311035/2.25, alarm hl:mem_free=347.000000M/200M, alarm hl:available=1/0 [21:35:59] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:38:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:41:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:50:21] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:56:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.000000/1.75, alarm hl:np_load_avg=2.035156/2.0, alarm hl:mem_free=612.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.000000/1.9, alarm hl:np_load_long=2.117188/2.25, alarm hl:mem_free=612.000000M/200M, alarm hl:available=1/0 [21:57:21] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:02:21] Load avg. on willow is WARNING: WARNING - load average: 17.48, 17.02, 16.95 [22:13:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:19:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [22:19:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.787598/1.75, alarm hl:np_load_avg=2.048828/2.0, alarm hl:mem_free=467.000000M/350M, alarm hl:available=1/0 [22:20:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:22:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:24:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:27:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.047363/1.75, alarm hl:np_load_avg=1.886230/2.0, alarm hl:mem_free=468.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.047363/1.9, alarm hl:np_load_long=2.000000/2.25, alarm hl:mem_free=468.000000M/200M, alarm hl:available=1/0 [22:28:59] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:31:19] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 167097.000000 [22:33:29] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:33:38] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:36:59] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [22:38:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:41:30] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:56:38] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.154297/1.75, alarm hl:np_load_avg=2.102539/2.0, alarm hl:mem_free=517.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.154297/1.9, alarm hl:np_load_long=2.068848/2.25, alarm hl:mem_free=517.000000M/200M, alarm hl:available=1/0 [22:57:39] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:02:38] Load avg. on willow is WARNING: WARNING - load average: 16.12, 16.77, 16.45 [23:03:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.979492/1.75, alarm hl:np_load_avg=2.078613/2.0, alarm hl:mem_free=386.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.979492/1.9, alarm hl:np_load_long=2.050781/2.25, alarm hl:mem_free=386.000000M/200M, alarm hl:available=1/0 [23:13:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:19:53] 3(assigned) [UTRS-96] HTTP 404 makes a rather concerning privacy policy <10https://jira.toolserver.org/browse/UTRS-96> (DeltaQuad) [23:19:56] 3(commented) [UTRS-96] HTTP 404 makes a rather concerning privacy policy <10https://jira.toolserver.org/browse/UTRS-96> (DeltaQuad) [23:20:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [23:20:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:21:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:24:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.930664/1.75, alarm hl:np_load_avg=1.967285/2.0, alarm hl:mem_free=712.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.930664/1.9, alarm hl:np_load_long=2.040039/2.25, alarm hl:mem_free=712.000000M/200M, alarm hl:available=1/0 [23:24:58] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:26:39] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:32:18] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 170759.000000 [23:36:58] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [23:39:28] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:41:38] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:49:48] Load avg. on willow is OK: OK - load average: 9.73, 13.23, 14.96 [23:52:48] Load avg. on willow is WARNING: WARNING - load average: 15.13, 14.87, 15.37