[00:07:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:12:22] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [00:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13061 [00:18:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13059.000000 [00:19:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:24:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 301030 MB (5% inode=32%): [00:43:03] /tmp on willow is WARNING: DISK WARNING - free space: / 20710 MB (19% inode=98%): [00:43:03] / on willow is WARNING: DISK WARNING - free space: / 20710 MB (19% inode=98%): [00:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39667 [00:50:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [00:55:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39734.000000 [01:07:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:11:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:12:22] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [01:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13021 [01:19:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13003.000000 [01:19:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:24:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:28:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 302307 MB (5% inode=33%): [01:43:03] /tmp on willow is WARNING: DISK WARNING - free space: / 20654 MB (19% inode=98%): [01:43:03] / on willow is WARNING: DISK WARNING - free space: / 20654 MB (19% inode=98%): [01:49:40] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40127 [01:50:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [01:56:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40100.000000 [02:07:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:10:02] /sql on thyme is WARNING: DISK WARNING - free space: /sql 183507 MB (19% inode=99%): [02:11:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:12:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [02:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12526 [02:20:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12480.000000 [02:20:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:24:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 300515 MB (5% inode=32%): [02:44:02] /tmp on willow is WARNING: DISK WARNING - free space: / 20600 MB (19% inode=98%): [02:44:02] / on willow is WARNING: DISK WARNING - free space: / 20600 MB (19% inode=98%): [02:48:30] zzz [02:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40424 [02:50:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [02:56:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40381.000000 [03:06:05] inductiveload: There seems to a multimaintainer project "dumps" that all users are in [03:07:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:11:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:12:32] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [03:14:42] Dispenser: cheers [03:16:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12836 [03:20:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12884.000000 [03:20:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:24:52] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 298506 MB (5% inode=32%): [03:37:10] i have to head off, but i'll check it out :-) [03:44:03] /tmp on willow is WARNING: DISK WARNING - free space: / 20541 MB (19% inode=98%): [03:44:03] / on willow is WARNING: DISK WARNING - free space: / 20541 MB (19% inode=98%): [03:49:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41117 [03:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [03:56:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41231.000000 [04:07:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:12:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:13:24] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [04:16:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12900 [04:20:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12806.000000 [04:20:33] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:24:52] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:29:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 296267 MB (5% inode=32%): [04:39:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40545 MB (9% inode=99%): [04:44:03] /tmp on willow is WARNING: DISK WARNING - free space: / 20486 MB (19% inode=98%): [04:44:03] / on willow is WARNING: DISK WARNING - free space: / 20486 MB (19% inode=98%): [04:49:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42180 [04:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [04:57:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42277.000000 [05:00:03] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 60117 MB (14% inode=99%): [05:07:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:12:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:13:23] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [05:16:22] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:17:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12107 [05:20:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12134.000000 [05:20:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:23:03] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 63780 MB (5% inode=99%): [05:24:51] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 294661 MB (5% inode=32%): [05:44:03] /tmp on willow is WARNING: DISK WARNING - free space: / 20429 MB (19% inode=98%): [05:44:03] / on willow is WARNING: DISK WARNING - free space: / 20429 MB (19% inode=98%): [05:49:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43160 [05:50:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [05:50:51] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:58:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43363.000000 [06:08:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:12:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:13:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 353308 MB (6% inode=36%): [06:13:03] /tmp on willow is OK: DISK OK - free space: / 22744 MB (21% inode=99%): [06:13:03] / on willow is OK: DISK OK - free space: / 22744 MB (21% inode=99%): [06:13:32] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [06:18:23] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/24:DOWN:1 UP: OK [06:18:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13136 [06:20:13] SMTP on damiana is OK: SMTP OK - 1.714 sec. response time [06:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:21:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13161.000000 [06:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44165 [06:50:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [06:58:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44228.000000 [07:17:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:18:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 322466 MB (6% inode=34%): [07:18:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13714 [07:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:21:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:21:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13730.000000 [07:44:24] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.707031/1.10, alarm hl:np_load_long=0.921875/1.55, alarm hl:mem_free=17066.000000M/500M, alarm hl:tmp_free=14059M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.707031/1.00, alarm hl:np_load_long=0.921875/1.50, alarm hl:mem_free=17066.000000M/600M, alarm hl:tmp_free [07:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44934 [07:50:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [07:59:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44942.000000 [08:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:18:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 321962 MB (6% inode=34%): [08:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:19:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14375 [08:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:21:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14398.000000 [08:41:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:48:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.337891/1.10, alarm hl:np_load_long=1.374024/1.55, alarm hl:mem_free=18095.000000M/500M, alarm hl:tmp_free=14009M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.337891/1.00, alarm hl:np_load_long=1.374024/1.50, alarm hl:mem_free=18095.000000M/600M, alarm hl:tmp_free [08:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44994 [08:50:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [08:56:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:59:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45038.000000 [09:09:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 318242 MB (5% inode=34%): [09:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:19:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14966 [09:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:21:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14988.000000 [09:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45802 [09:50:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [09:59:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45863.000000 [10:09:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317589 MB (5% inode=34%): [10:16:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 329656 MB (6% inode=34%): [10:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:20:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15557 [10:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:21:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15573.000000 [10:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46015 [10:51:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [10:59:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45954.000000 [11:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 332487 MB (6% inode=35%): [11:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:20:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15650 [11:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:21:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15649.000000 [11:31:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.643555/1.10, alarm hl:np_load_long=1.015625/1.55, alarm hl:mem_free=20441.000000M/500M, alarm hl:tmp_free=13743M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.643555/1.00, alarm hl:np_load_long=1.015625/1.50, alarm hl:mem_free=20441.000000M/600M, alarm hl:tmp_free [11:33:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:40:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.555664/1.10, alarm hl:np_load_long=1.191406/1.55, alarm hl:mem_free=20208.000000M/500M, alarm hl:tmp_free=13737M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.555664/1.00, alarm hl:np_load_long=1.191406/1.50, alarm hl:mem_free=20208.000000M/600M, alarm hl:tmp_free [11:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45951 [11:51:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [11:52:55] @replag [11:52:55] Betacommand: s1-rr-a: 4h 18m 58s [+0.07 s/s]; s1-user: 12h 45m 41s [+0.14 s/s]; s3-rr-a: 12s [-0.00 s/s]; s3-user: 12s [-0.00 s/s] [11:57:24] @replag [11:57:24] Betacommand: s1-rr-a: 4h 19m 2s [+0.01 s/s]; s1-user: 12h 45m 13s [-0.10 s/s]; s3-rr-a: 1m 2s [+0.19 s/s]; s3-user: 1m 2s [+0.19 s/s] [11:59:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45910.000000 [12:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 332133 MB (6% inode=35%): [12:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:18:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:21:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15705.000000 [12:21:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15709 [12:25:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.705078/1.10, alarm hl:np_load_long=1.683594/1.55, alarm hl:mem_free=20193.000000M/500M, alarm hl:tmp_free=13690M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.705078/1.00, alarm hl:np_load_long=1.683594/1.50, alarm hl:mem_free=20193.000000M/600M, alarm hl:tmp_free [12:28:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46256 [12:52:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [12:59:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46624.000000 [13:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 331723 MB (6% inode=35%): [13:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:21:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15931.000000 [13:21:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15931 [13:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44591 [13:52:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [13:53:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40386 MB (9% inode=99%): [13:59:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44118.000000 [14:10:34] /sql on thyme is WARNING: DISK WARNING - free space: /sql 183400 MB (19% inode=99%): [14:14:23] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 62962 MB (15% inode=99%): [14:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 331406 MB (6% inode=35%): [14:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:21:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16344.000000 [14:21:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16343 [14:35:25] @replag [14:35:25] Betacommand: s1-rr-a: 4h 32m 48s [+0.09 s/s]; s1-user: 12h 15m 57s [-0.19 s/s]; s2-user: 11s [-] [14:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44212 [14:52:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [14:59:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44287.000000 [15:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 330771 MB (6% inode=35%): [15:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:21:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16420.000000 [15:22:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16426 [15:37:45] @replag [15:37:46] euphoria: s1-rr-a: 4h 33m 7s [+0.01 s/s]; s1-user: 12h 26m 35s [+0.17 s/s]; s2-user: 21s [+0.00 s/s]; s3-rr-a: 1m 2s [-]; s3-user: 1m 2s [-] [15:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44991 [15:52:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [15:57:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38096 MB (9% inode=99%): [15:59:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45083.000000 [16:10:24] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24042 MB (5% inode=99%): [16:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 330456 MB (6% inode=35%): [16:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:21:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16575.000000 [16:22:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16568 [16:48:24] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 49184 MB (12% inode=99%): [16:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45995 [16:52:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [17:00:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46129.000000 [17:16:04] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 330233 MB (6% inode=35%): [17:17:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:21:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:21:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17019.000000 [17:22:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17022 [17:23:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 66761 MB (5% inode=99%): [17:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46897 [17:52:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [18:00:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47043.000000 [18:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 329953 MB (6% inode=35%): [18:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:21:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:21:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17382.000000 [18:22:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17380 [18:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48095 [18:52:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [19:00:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48322.000000 [19:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 329666 MB (6% inode=34%): [19:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:21:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17787.000000 [19:22:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:23:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17796 [19:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49244 [19:53:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [20:00:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49329.000000 [20:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 329408 MB (6% inode=34%): [20:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:21:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18437.000000 [20:22:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:23:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18430 [20:39:16] @replag [20:39:16] Dereckson: s1-rr-a: 5h 6m 3s [+0.11 s/s]; s1-user: 13h 52m 6s [+0.28 s/s]; s2-user: 15s [-0.00 s/s]; s3-rr-a: 32s [-0.00 s/s]; s3-user: 32s [-0.00 s/s] [20:39:29] Hi. [20:40:24] soo.. it's getting worse [20:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50065 [20:51:14] hello all [20:51:17] @replag [20:51:17] DaBPunkt: s1-rr-a: 5h 7m 52s [+0.15 s/s]; s1-user: 13h 54m 29s [+0.20 s/s]; s2-user: 16s [+0.00 s/s]; s3-rr-a: 1m 5s [+0.05 s/s]; s3-user: 1m 5s [+0.05 s/s] [20:53:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [21:00:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50121.000000 [21:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 329082 MB (6% inode=34%): [21:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:18:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:21:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18564.000000 [21:22:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:24:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18550 [21:36:22] @replag [21:36:23] Earwig: s1-rr-a: 5h 8m 32s [+0.01 s/s]; s1-user: 14h 3m 26s [+0.20 s/s]; s2-user: 24s [+0.00 s/s]; s3-rr-a: 1m 2s [-0.00 s/s]; s3-user: 1m 2s [-0.00 s/s] [21:36:34] DaBPunkt: I realize you're very busy, but any info on the s1 replag? [21:37:47] Earwig: rev_sha1 is populated – that will take some time [21:38:02] in the meaning of days [21:38:21] okay, thanks [21:38:50] np [21:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50686 [21:53:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [22:00:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50711.000000 [22:14:22] nacht ts [22:16:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 328904 MB (6% inode=34%): [22:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:22:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18565.000000 [22:23:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:24:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18555 [22:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51265 [22:54:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [23:00:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51270.000000 [23:17:03] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 327527 MB (6% inode=34%): [23:17:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:22:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18829.000000 [23:23:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:24:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18824 [23:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51787 [23:55:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default