[00:09:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.952637/1.75, alarm hl:np_load_avg=0.873047/2.0, alarm hl:mem_free=181.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.952637/1.9, alarm hl:np_load_long=0.777832/2.25, alarm hl:mem_free=181.000000M/200M, alarm hl:available=1/0 [00:10:15] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:19:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 702240.000000 [00:20:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 702335.000000 [00:33:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:33:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:34:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:37:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:43:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:54:14] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2074.000000 [01:09:16] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1794.000000 [01:18:33] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:20:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 705901.000000 [01:20:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 705940.000000 [01:23:56] [[Special:Log/newusers]] create 10 * Thsevier * (New user account) [01:27:24] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 13.252 second response time [01:33:24] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:34:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:34:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:38:15] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:38:24] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1872.000000 [01:43:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:48:53] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:18:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.009277/1.75, alarm hl:np_load_avg=1.112793/2.0, alarm hl:mem_free=253.000000M/350M, alarm hl:available=1/0 [02:20:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 709507.000000 [02:21:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 709554.000000 [02:28:55] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:33:16] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.077149/1.00, alarm hl:np_load_long=0.731445/1.50, alarm hl:mem_free=20164.000000M/350M, alarm hl:available=1/0 [02:33:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:34:15] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:34:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:34:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:38:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:38:25] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1997.000000 [02:41:55] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 272 bytes in 9.853 second response time [02:43:15] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.386719/1.10, alarm hl:np_load_long=0.817383/1.55, alarm hl:mem_free=19989.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.386719/1.00, alarm hl:np_load_long=0.817383/1.50, alarm hl:mem_free=19989.000000M/350M, alarm hl:available=1/0 [02:44:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:59:15] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 136854 MB (14% inode=99%): [03:00:25] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1788.000000 [03:20:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713111.000000 [03:21:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713162.000000 [03:33:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:35:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:37:54] /tmp on willow is WARNING: DISK WARNING - free space: / 21983 MB (20% inode=99%): [03:38:17] / on willow is WARNING: DISK WARNING - free space: / 21980 MB (20% inode=99%): [03:39:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:44:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:46:15] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 33493 MB (3% inode=99%): [04:13:16] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.125976/1.10, alarm hl:np_load_long=0.716797/1.55, alarm hl:mem_free=20497.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.125976/1.00, alarm hl:np_load_long=0.716797/1.50, alarm hl:mem_free=20497.000000M/350M, alarm hl:available=1/0 [04:15:15] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:20:24] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 716712.000000 [04:21:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 716762.000000 [04:30:11] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6960&oldid=6911&rcid=9163 * 91.198.174.202 * (+0) (updated page) [04:33:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:35:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:37:55] /tmp on willow is WARNING: DISK WARNING - free space: / 21561 MB (20% inode=99%): [04:39:14] / on willow is WARNING: DISK WARNING - free space: / 21550 MB (20% inode=99%): [04:40:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:45:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:15:55] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.165527/1.75, alarm hl:np_load_avg=1.293945/2.0, alarm hl:mem_free=202.000000M/350M, alarm hl:available=1/0 [05:20:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 720311.000000 [05:21:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 720361.000000 [05:23:24] Load avg. on willow is WARNING: WARNING - load average: 15.29, 13.89, 11.67 [05:24:24] Load avg. on willow is OK: OK - load average: 13.81, 13.70, 11.74 [05:29:34] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1962.000000 [05:30:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:32:23] Load avg. on willow is WARNING: WARNING - load average: 18.00, 15.84, 13.31 [05:33:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.310547/1.75, alarm hl:np_load_avg=2.003418/2.0, alarm hl:mem_free=136.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.310547/1.9, alarm hl:np_load_long=1.675293/2.25, alarm hl:mem_free=136.000000M/200M, alarm hl:available=1/0 [05:33:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:38:54] /tmp on willow is WARNING: DISK WARNING - free space: / 21142 MB (20% inode=98%): [05:39:14] / on willow is WARNING: DISK WARNING - free space: / 21139 MB (20% inode=98%): [05:40:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:46:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:07:14] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1901.000000 [06:20:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [06:21:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 723970.000000 [06:21:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 723972.000000 [06:33:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:35:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:37:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:39:16] / on willow is WARNING: DISK WARNING - free space: / 20719 MB (19% inode=98%): [06:39:55] /tmp on willow is WARNING: DISK WARNING - free space: / 20716 MB (19% inode=98%): [06:40:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:46:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:59:24] Load avg. on willow is WARNING: WARNING - load average: 15.98, 16.24, 16.18 [07:00:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.970215/1.75, alarm hl:np_load_avg=2.020508/2.0, alarm hl:mem_free=311.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.970215/1.9, alarm hl:np_load_long=2.019043/2.25, alarm hl:mem_free=311.000000M/200M, alarm hl:available=1/0 [07:02:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000 [07:03:54] /tmp on willow is OK: DISK OK - free space: / 22802 MB (21% inode=99%): [07:04:15] / on willow is OK: DISK OK - free space: / 22769 MB (21% inode=99%): [07:06:24] Load avg. on willow is CRITICAL: CRITICAL - load average: 32.66, 25.36, 20.19 [07:21:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 727572.000000 [07:21:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 727571.000000 [07:21:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6882.000000 [07:26:24] Load avg. on willow is WARNING: WARNING - load average: 15.89, 18.59, 19.78 [07:34:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:41:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:46:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:46:55] @replag [07:46:56] kevin_brown: s1-rr-a: 1w 1d 10h 31m 51s [+1.00 s/s]; s1-rr-a-c: 1h 59m 59s [+0.20 s/s]; s1-user: 1w 1d 10h 31m 51s [+1.00 s/s]; s2-user-c: 1h 21m 10s [+0.13 s/s]; s3-rr-a: 15s [-0.00 s/s]; s3-user: 15s [-0.00 s/s]; s5-user-c: 1h 21m 10s [+0.13 s/s] [08:00:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.925293/1.75, alarm hl:np_load_avg=2.032227/2.0, alarm hl:mem_free=630.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.925293/1.9, alarm hl:np_load_long=2.166992/2.25, alarm hl:mem_free=630.000000M/200M, alarm hl:available=1/0 [08:02:15] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5386.000000 [08:09:55] 3(updated) [MNT-1225] Growing replag on S1 due to a database migration at WMF <10https://jira.toolserver.org/browse/MNT-1225> (Anonymous) [08:11:55] 3(updated) [MNT-1225] Growing replag on S1 due to a database migration at WMF <10https://jira.toolserver.org/browse/MNT-1225> (Marlen Caemmerer) [08:12:26] I'm currently getting MySQL error 1290 on s1-user for any insert query, any ideas on why? [08:12:56] according to google this looks to be some kind of permissions error [08:13:40] note I'm trying to do inserts for my own user database, not for a view... [08:14:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:21:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7706.000000 [08:21:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731198.000000 [08:22:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731232.000000 [08:23:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.226562/1.75, alarm hl:np_load_avg=1.987305/2.0, alarm hl:mem_free=933.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.226562/1.9, alarm hl:np_load_long=2.026367/2.25, alarm hl:mem_free=933.000000M/200M, alarm hl:available=1/0 [08:27:23] Load avg. on willow is WARNING: WARNING - load average: 15.13, 15.90, 16.12 [08:33:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:34:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:38:23] Load avg. on willow is OK: OK - load average: 12.36, 13.70, 14.86 [08:41:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:43:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:02:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6800.000000 [09:21:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8050.000000 [09:21:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 734799.000000 [09:22:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 734832.000000 [09:34:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:41:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:43:02] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:45:14] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.292969/1.10, alarm hl:np_load_long=0.832031/1.55, alarm hl:mem_free=20943.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.292969/1.00, alarm hl:np_load_long=0.832031/1.50, alarm hl:mem_free=20943.000000M/350M, alarm hl:available=1/0 [09:46:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:46:14] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:58:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.020020/1.75, alarm hl:np_load_avg=1.707520/2.0, alarm hl:mem_free=319.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.020020/1.9, alarm hl:np_load_long=1.634766/2.25, alarm hl:mem_free=319.000000M/200M, alarm hl:available=1/0 [10:02:24] Load avg. on willow is WARNING: WARNING - load average: 16.37, 14.83, 13.70 [10:03:15] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8273.000000 [10:04:25] Load avg. on willow is OK: OK - load average: 14.58, 14.73, 13.80 [10:08:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:21:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8116.000000 [10:21:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 738399.000000 [10:23:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 738491.000000 [10:34:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:37:34] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:41:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:45:23] Load avg. on willow is WARNING: WARNING - load average: 15.18, 14.34, 13.49 [10:46:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:46:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.851074/1.75, alarm hl:np_load_avg=1.786133/2.0, alarm hl:mem_free=833.000000M/350M, alarm hl:available=1/0 [10:49:23] Load avg. on willow is OK: OK - load average: 13.73, 14.64, 13.85 [10:50:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:52:24] Load avg. on willow is WARNING: WARNING - load average: 17.89, 16.80, 14.89 [10:53:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.183105/1.75, alarm hl:np_load_avg=2.094238/2.0, alarm hl:mem_free=867.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.183105/1.9, alarm hl:np_load_long=1.862793/2.25, alarm hl:mem_free=867.000000M/200M, alarm hl:available=1/0 [11:03:15] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10249.000000 [11:21:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 741998.000000 [11:22:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6572.000000 [11:24:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 742151.000000 [11:26:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.838867/1.75, alarm hl:np_load_avg=1.958008/2.0, alarm hl:mem_free=1348.000000M/350M, alarm hl:available=1/0 [11:26:23] Load avg. on willow is WARNING: WARNING - load average: 14.22, 15.43, 15.07 [11:27:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:30:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.753418/1.75, alarm hl:np_load_avg=1.873047/2.0, alarm hl:mem_free=1245.000000M/350M, alarm hl:available=1/0 [11:33:23] Load avg. on willow is OK: OK - load average: 12.75, 14.54, 14.86 [11:35:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:35:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:37:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:42:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:46:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:00:24] Load avg. on willow is WARNING: WARNING - load average: 18.74, 15.42, 14.13 [12:01:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.163086/1.75, alarm hl:np_load_avg=1.925293/2.0, alarm hl:mem_free=1170.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.163086/1.9, alarm hl:np_load_long=1.770996/2.25, alarm hl:mem_free=1170.000000M/200M, alarm hl:available=1/0 [12:02:23] Load avg. on willow is OK: OK - load average: 13.42, 14.69, 14.03 [12:03:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:04:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13276.000000 [12:17:25] Load avg. on willow is WARNING: WARNING - load average: 16.64, 13.86, 13.32 [12:18:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.037109/1.75, alarm hl:np_load_avg=1.754395/2.0, alarm hl:mem_free=1073.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.037109/1.9, alarm hl:np_load_long=1.674805/2.25, alarm hl:mem_free=1073.000000M/200M, alarm hl:available=1/0 [12:22:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5880.000000 [12:22:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 745658.000000 [12:24:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 745752.000000 [12:35:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:35:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:43:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:46:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:48:23] Load avg. on willow is WARNING: WARNING - load average: 15.90, 15.01, 14.13 [12:49:24] Load avg. on willow is OK: OK - load average: 13.71, 14.57, 14.03 [12:56:46] hello all [13:04:15] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16405.000000 [13:21:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.998047/1.75, alarm hl:np_load_avg=1.792480/2.0, alarm hl:mem_free=1364.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.998047/1.9, alarm hl:np_load_long=1.654297/2.25, alarm hl:mem_free=1364.000000M/200M, alarm hl:available=1/0 [13:22:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:22:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5743.000000 [13:22:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 749258.000000 [13:24:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 749352.000000 [13:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:35:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:43:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.971680/1.75, alarm hl:np_load_avg=1.785645/2.0, alarm hl:mem_free=751.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.971680/1.9, alarm hl:np_load_long=1.691895/2.25, alarm hl:mem_free=751.000000M/200M, alarm hl:available=1/0 [13:43:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:46:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:02:23] Load avg. on willow is WARNING: WARNING - load average: 15.59, 14.13, 13.70 [14:04:23] Load avg. on willow is OK: OK - load average: 12.85, 13.80, 13.65 [14:05:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19547.000000 [14:10:25] Load avg. on willow is WARNING: WARNING - load average: 18.43, 15.40, 14.29 [14:14:59] @replag [14:14:59] DaBPunkt: s1-rr-a: 1w 1d 16h 59m 54s [+1.00 s/s]; s1-rr-a-c: 2h 10m 33s [+0.73 s/s]; s1-user: 1w 1d 16h 59m 54s [+1.00 s/s]; s2-user: 22m 43s [+0.63 s/s]; s2-user-c: 5h 35m 16s [+0.94 s/s]; s3-rr-a: 24s [+0.04 s/s]; s3-user: 24s [+0.04 s/s]; s5-user-c: 5h 35m 16s [+0.94 s/s] [14:16:53] 3(resolved) [MNT-1196] ptolemy is not longer accessable by ipv6 <10https://jira.toolserver.org/browse/MNT-1196> (Marlen Caemmerer) [14:23:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8169.000000 [14:23:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 752918.000000 [14:24:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 752952.000000 [14:24:57] 3(closed) [MNT-1196] ptolemy is not longer accessable by ipv6 <10https://jira.toolserver.org/browse/MNT-1196> (DaB.) [14:25:59] DaBPunkt: is flup installed? [14:26:16] flup? [14:26:22] https://wiki.toolserver.org/view/Python_WSGI#Flup [14:26:50] i'm tryng to use it, but with no success [14:32:14] Alchimista: there is at least code in our svn. I will look for the package if it is installed [14:32:47] DaBPunkt: oki doki. do you know who uses to use python on webpage tools? [14:33:17] AFAIS it is installed. [14:33:24] Alchimista: no idea, sory [14:33:44] hehe, seems that DispenserAFK does. i'll try talk to him later, thanks [14:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:35:35] brb [14:35:51] re [14:36:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:37:09] kevin_brown: 1290 mean the database is in read-only mode. The WMF is adding a column that'll be done sometime next month. [14:37:56] Alchimista: All my python web tools are working well [14:43:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:47:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:50:51] Dispenser: well, so i'm doing something wrong :s Is it possible to use Bottlepy (http://bottlepy.org) wich can use flup server? [14:51:08] what's the path to your script? [14:53:50] /home/alchimista/public_html/cgi-bin [14:53:53] 3(assigned) [TS-1339] Multi-maintainer project request: citegen <10https://jira.toolserver.org/browse/TS-1339> [14:54:27] Can't list directory contents [14:56:07] But ~/public_html/page.py has the wrong extension and missing the execute bit [14:56:30] /home/alchimista/public_html/cgi-bin/web.py [14:56:42] that was for testing [14:56:57] 3(assigned) [TS-1339] Multi-maintainer project request: citegen <10https://jira.toolserver.org/browse/TS-1339> [14:59:14] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 134110 MB (13% inode=99%): [14:59:39] You missing the shebang and its execute bit is off. You might want to add: import cgitb;cgitb.enable() [15:01:39] Dispenser: on web.py, or page.py? [15:01:49] web.py [15:02:15] you can reference /home/dispenser/public_html/cgi-bin/related.py [15:05:15] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22073.000000 [15:15:53] Dispenser: with exec and imprt... still not working. Fcgi do not work on cgi right? Seems that it uses FlupFCGIServer [15:18:55] 3(assigned) [TS-1339] Multi-maintainer project request: citegen <10https://jira.toolserver.org/browse/TS-1339> (DaB.) [15:20:58] 3(resolved) [TS-1339] Multi-maintainer project request: citegen <10https://jira.toolserver.org/browse/TS-1339> (DaB.) [15:23:14] Alchimista: To get page.py working: mv page.py page.fcgi; chmod +x page.fcgi [15:23:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10524.000000 [15:23:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 756521.000000 [15:25:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 756611.000000 [15:30:20] Dispenser: that worked! what about web.py? on /home/alchimista/public_html/cgi-bin/web.py [15:30:49] I'm reading the flup source code [15:32:16] it has a basic WSGIRefServer wich uses wsgiref.simple_server [15:33:27] isn't ZWS configured such that nothing in cgi-bin is FastCGI-compatible? [15:34:01] at least that's what https://wiki.toolserver.org/view/Web_hosting says [15:35:00] If possible, you should use FastCGI, since it's faster, and reduces the load on the web server. [15:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:35:24] Load avg. on willow is WARNING: WARNING - load average: 20.06, 14.57, 12.66 [15:35:41] oh yah, they should be placed on public_html. [15:36:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.471191/1.75, alarm hl:np_load_avg=1.880371/2.0, alarm hl:mem_free=860.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.471191/1.9, alarm hl:np_load_long=1.610840/2.25, alarm hl:mem_free=860.000000M/200M, alarm hl:available=1/0 [15:36:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:37:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:38:28] FastCGI is hard to development with since old process linger around for a while [15:42:22] i've moved web.py to public_html, and used bottle's FlupFCGIServer, but still nothing :s [15:43:00] is there any other basic framework like bottle who works fine on ts? [15:43:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:47:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:47:14] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 76541 MB (7% inode=99%): [16:00:24] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.38, 22.40, 20.20 [16:05:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22053.000000 [16:06:24] Load avg. on willow is WARNING: WARNING - load average: 17.93, 20.14, 19.98 [16:08:24] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.50, 20.48, 20.11 [16:11:25] Apparently the flup doesn't like ZWS, http://trac.saddi.com/flup/ticket/55 describes it as Apache missing mod_fcgid [16:14:19] isn't any other option who likes more of ZWS? [16:15:36] Yes, you could override some function inside flup [16:17:14] BTW, mv ~/public_html/web.py ~/public_html/web.fcgi [16:22:05] that uses bottlepy [16:23:06] it should work without fcgi [16:23:48] DaBPunkt: If replag is projected to be 36 days on enwiki; is there any possible relief for all the batch jobs kicking off on the 31st? [16:23:51] *.fcgi [16:24:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13557.000000 [16:24:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 760181.000000 [16:25:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 760212.000000 [16:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:36:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.148438/1.75, alarm hl:np_load_avg=2.354980/2.0, alarm hl:mem_free=1085.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.148438/1.9, alarm hl:np_load_long=2.487793/2.25, alarm hl:mem_free=1085.000000M/200M, alarm hl:available=1/0 [16:36:23] Load avg. on willow is WARNING: WARNING - load average: 18.58, 19.00, 19.92 [16:37:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:37:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:44:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:47:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:55:42] [[GetWikiAPI]] 10https://wiki.toolserver.org/w/index.php?diff=6961&oldid=5587&rcid=9164 * Krinkle * (-3) (update url) [16:55:53] 3(created) [TS-1341] Webserver FastCGI interface does not set $HOME; Toolserver: Webserver: General/Unknown; Bug <10https://jira.toolserver.org/browse/TS-1341> [17:00:45] [[Template:Toolserver url]] M 10https://wiki.toolserver.org/w/index.php?diff=6962&oldid=5330&rcid=9165 * Krinkle * (+1) (https, (I was going to make it protocol-relative, //, but this wiki is https already, and // only works when used in [// .. .. ] construct, not as literal and this template is used in both ways)_) [17:05:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22316.000000 [17:23:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:24:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10052.000000 [17:24:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 763780.000000 [17:25:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 763812.000000 [17:26:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.978516/1.75, alarm hl:np_load_avg=1.987305/2.0, alarm hl:mem_free=526.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.978516/1.9, alarm hl:np_load_long=2.185547/2.25, alarm hl:mem_free=526.000000M/200M, alarm hl:available=1/0 [17:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:35:55] 3(updated) [ACCAPP-441] SVN, Database access and storage <10https://jira.toolserver.org/browse/ACCAPP-441> [17:36:23] Load avg. on willow is WARNING: WARNING - load average: 15.59, 16.14, 16.92 [17:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:38:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:41:58] 3(commented) [TS-1339] Multi-maintainer project request: citegen <10https://jira.toolserver.org/browse/TS-1339> [17:44:14] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.083984/1.00, alarm hl:np_load_long=0.669922/1.50, alarm hl:mem_free=20509.000000M/350M, alarm hl:available=1/0 [17:45:14] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:45:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:47:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:50:24] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.72, 22.45, 20.07 [17:53:12] [[Special:Log/newusers]] create 10 * Brandon Sky Pimenta * (New user account) [17:56:15] [[User talk:Gifti]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6963&rcid=9167 * Brandon Sky Pimenta * (+86) (Created page with "==Request for undeletion== why deleted the unblock template? users could be unblocked?") [18:05:15] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22458.000000 [18:24:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9091.000000 [18:24:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 767381.000000 [18:25:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 767411.000000 [18:25:24] Load avg. on willow is WARNING: WARNING - load average: 17.40, 18.35, 19.96 [18:26:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.148926/1.75, alarm hl:np_load_avg=2.284180/2.0, alarm hl:mem_free=1003.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.148926/1.9, alarm hl:np_load_long=2.490234/2.25, alarm hl:mem_free=1003.000000M/200M, alarm hl:available=1/0 [18:28:23] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.38, 20.27, 20.32 [18:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:38:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:45:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:47:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:49:23] Load avg. on willow is WARNING: WARNING - load average: 16.91, 18.53, 19.82 [18:50:23] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.25, 19.36, 20.00 [18:51:24] Load avg. on willow is WARNING: WARNING - load average: 18.97, 19.07, 19.87 [18:54:55] 3(commented) [TS-1339] Multi-maintainer project request: citegen <10https://jira.toolserver.org/browse/TS-1339> (DaB.) [19:03:54] [[Special:Log/newusers]] create 10 * Schody * (New user account) [19:04:20] [[User talk:Schody]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6964&rcid=9169 * Schody * (+3680) (Created page with "{Najbardziej|W najwiekszym stopniu|W najwyzszym stopniu} oczywistym sposobem {dochrapac sie|dostac|osiagnac|uzyskac|zdobyc} sie {az do|do} ladowania {albo|badz|czy tez|ewentualni...") [19:04:55] [[Special:Log/delete]] delete 10 * Dab * (deleted "[[02User talk:Schody10]]": non-use: giblerisch) [19:06:13] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22963.000000 [19:24:34] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8627.000000 [19:24:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 770980.000000 [19:25:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 771012.000000 [19:26:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.520020/1.75, alarm hl:np_load_avg=3.416016/2.0, alarm hl:mem_free=1479.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.520020/1.9, alarm hl:np_load_long=3.303711/2.25, alarm hl:mem_free=1479.000000M/200M, alarm hl:available=1/0 [19:29:24] Load avg. on willow is CRITICAL: CRITICAL - load average: 14.09, 20.62, 23.85 [19:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:38:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:40:53] 3(created) [ACCAPP-483] Create (or re-activate) account; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-483> (Zack Exley) [19:42:23] Load avg. on willow is WARNING: WARNING - load average: 16.47, 17.13, 19.91 [19:44:13] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.211914/1.10, alarm hl:np_load_long=0.773438/1.55, alarm hl:mem_free=20606.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.211914/1.00, alarm hl:np_load_long=0.773438/1.50, alarm hl:mem_free=20606.000000M/350M, alarm hl:available=1/0 [19:45:14] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [19:45:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:47:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:53:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:03:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.922851/1.75, alarm hl:np_load_avg=1.899414/2.0, alarm hl:mem_free=1083.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.922851/1.9, alarm hl:np_load_long=2.036133/2.25, alarm hl:mem_free=1083.000000M/200M, alarm hl:available=1/0 [20:06:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19718.000000 [20:09:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:24:23] Load avg. on willow is OK: OK - load average: 12.31, 13.73, 14.86 [20:25:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4560.000000 [20:25:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 774641.000000 [20:26:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 774673.000000 [20:35:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:35:33] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3219.000000 [20:36:27] @replag [20:36:28] Dispenser: s1-rr-a: 1w 1d 23h 21m 23s [+1.00 s/s]; s1-rr-a-c: 47m 7s [-0.22 s/s]; s1-user: 1w 1d 23h 21m 23s [+1.00 s/s]; s2-user: 17s [-0.06 s/s]; s2-user-c: 5h 27m 52s [-0.02 s/s]; s3-rr-a: 46s [+0.00 s/s]; s3-user: 46s [+0.00 s/s]; s5-user-c: 5h 27m 52s [-0.02 s/s] [20:37:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:38:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:39:33] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1728.000000 [20:45:24] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.713867/1.10, alarm hl:np_load_long=0.846680/1.55, alarm hl:mem_free=20850.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.713867/1.00, alarm hl:np_load_long=0.846680/1.50, alarm hl:mem_free=20850.000000M/350M, alarm hl:available=1/0 [20:46:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:47:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:47:24] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:02:25] Load avg. on willow is WARNING: WARNING - load average: 17.20, 14.42, 13.53 [21:03:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.994141/1.75, alarm hl:np_load_avg=1.799316/2.0, alarm hl:mem_free=1273.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.994141/1.9, alarm hl:np_load_long=1.693848/2.25, alarm hl:mem_free=1273.000000M/200M, alarm hl:available=1/0 [21:04:24] Load avg. on willow is OK: OK - load average: 13.64, 14.08, 13.52 [21:06:24] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19608.000000 [21:11:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:12:24] Load avg. on willow is WARNING: WARNING - load average: 15.14, 14.28, 13.80 [21:13:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.783691/1.75, alarm hl:np_load_avg=1.767090/2.0, alarm hl:mem_free=1045.000000M/350M, alarm hl:available=1/0 [21:16:56] @replag [21:16:57] matthewrbowker: s1-rr-a: 1w 2d 1m 52s [+1.00 s/s]; s1-user: 1w 2d 1m 52s [+1.00 s/s]; s2-user-c: 5h 27m 51s [-0.00 s/s]; s3-rr-a: 27s [-0.01 s/s]; s3-user: 27s [-0.01 s/s]; s5-user-c: 5h 27m 51s [-0.00 s/s] [21:23:36] @replag [21:23:36] sumanah: s1-rr-a: 1w 2d 8m 31s [+1.00 s/s]; s1-rr-a-c: 3m 22s [-0.93 s/s]; s1-user: 1w 2d 8m 31s [+1.00 s/s]; s2-user-c: 5h 26m 51s [-0.15 s/s]; s3-rr-a: 1m 19s [+0.13 s/s]; s3-user: 1m 19s [+0.13 s/s]; s5-user-c: 5h 26m 51s [-0.15 s/s] [21:26:02] Sumana Harihareswara * Re: [Toolserver-l] S1 replag [21:26:24] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 778274.000000 [21:26:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 778301.000000 [21:29:02] Sumana Harihareswara * Re: [Toolserver-l] S1 replag [21:34:20] nighty~ [21:35:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:37:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:38:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:46:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:47:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:06:26] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18991.000000 [22:23:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:25:02] DaB. * Re: [Toolserver-l] S1 replag [22:25:18] DaBPunkt: https://en.wikipedia.org/wiki/User_talk:Josh_Parris#Toolserver_performance_to_improve_on_Friday.3F [22:26:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 781881.000000 [22:26:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 781900.000000 [22:28:23] Dispenser: ah, so that was the source [22:29:13] Somebody (belatedly) misreport on my report on Asher's guesstimate [22:29:59] Last week Friday become this week Friday as I was AFK over the weekend [22:35:36] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.571289/1.10, alarm hl:np_load_long=0.753906/1.55, alarm hl:mem_free=20656.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.571289/1.00, alarm hl:np_load_long=0.753906/1.50, alarm hl:mem_free=20656.000000M/350M, alarm hl:available=1/0 [22:35:37] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:36:34] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [22:37:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:39:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:46:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:47:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:06:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13939.000000 [23:23:02] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:26:36] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 785483.000000 [23:26:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 785502.000000 [23:35:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:37:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:39:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:46:56] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:47:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:54:56] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3473.000000 [23:57:57] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1373.000000 [23:59:55] nacht ts