[00:00:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [00:01:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73757.000000 [00:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:23:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25731.000000 [00:26:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:32:29] @replag [00:32:29] Earwig: s1-rr-a: 7h 8m 17s [+0.06 s/s]; s1-user: 20h 41m 34s [+0.17 s/s]; s2-user: 10s [-0.00 s/s]; s3-rr-a: 48s [+0.00 s/s]; s3-user: 48s [+0.00 s/s]; s6-rr-a: 12s [+0.00 s/s]; s6-user: 12s [+0.00 s/s] [00:35:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25684 [00:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 74747 [00:50:31] @replag [00:50:32] Earwig: s1-rr-a: 7h 8m 15s [-0.00 s/s]; s1-user: 20h 45m 50s [+0.24 s/s]; s6-rr-a: 11s [-0.00 s/s]; s6-user: 11s [-0.00 s/s] [00:50:39] *panic* [00:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 312251 MB (5% inode=33%): [01:01:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74817.000000 [01:01:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [01:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:18:31] @replag [01:18:31] Brooke: s1-rr-a: 7h 6m 36s [-0.06 s/s]; s1-user: 20h 51m 5s [+0.19 s/s]; s2-user: 20s [+0.00 s/s]; s3-rr-a: 34s [-0.01 s/s]; s3-user: 34s [-0.01 s/s]; s6-rr-a: 19s [+0.00 s/s]; s6-user: 19s [+0.00 s/s] [01:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25591.000000 [01:26:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:35:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25621 [01:50:14] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75283 [01:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 282617 MB (5% inode=31%): [02:01:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75288.000000 [02:01:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [02:10:33] /sql on thyme is WARNING: DISK WARNING - free space: /sql 180737 MB (18% inode=99%): [02:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25666.000000 [02:26:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:35:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25615 [02:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75825 [02:51:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 301241 MB (5% inode=32%): [03:01:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75874.000000 [03:01:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [03:12:24] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.349609/1.10, alarm hl:np_load_long=1.064453/1.55, alarm hl:mem_free=17912.000000M/500M, alarm hl:tmp_free=13956M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.349609/1.00, alarm hl:np_load_long=1.064453/1.50, alarm hl:mem_free=17912.000000M/600M, alarm hl:tmp_free [03:14:24] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:17:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25633.000000 [03:26:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:36:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25617 [03:45:23] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106643 MB (10% inode=98%): [03:47:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106438 MB (10% inode=98%): [03:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 76584 [03:51:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 300779 MB (5% inode=32%): [04:00:23] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 108781 MB (11% inode=98%): [04:00:34] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107804 MB (11% inode=98%): [04:01:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76706.000000 [04:01:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [04:17:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25596.000000 [04:26:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:37:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25643 [04:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 77275 [04:51:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 301029 MB (5% inode=32%): [05:01:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77355.000000 [05:02:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [05:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:23:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 63380 MB (5% inode=99%): [05:23:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26325.000000 [05:27:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:37:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26444 [05:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78020 [05:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 300244 MB (5% inode=32%): [06:01:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78076.000000 [06:03:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [06:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:21:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41547 MB (10% inode=99%): [06:23:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26467.000000 [06:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:37:54] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:38:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26478 [06:42:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 33549 MB (8% inode=99%): [06:43:25] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 60169 MB (14% inode=99%): [06:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78738 [06:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 299559 MB (5% inode=32%): [07:01:24] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78736.000000 [07:03:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [07:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:23:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26336.000000 [07:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:38:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26260 [07:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 79099 [07:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 298537 MB (5% inode=32%): [08:01:24] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79096.000000 [08:04:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [08:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26132.000000 [08:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:39:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26127 [08:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 79673 [08:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 298597 MB (5% inode=32%): [09:02:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79797.000000 [09:05:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [09:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26168.000000 [09:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:39:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26122 [09:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 80978 [09:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 298330 MB (5% inode=32%): [10:02:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81488.000000 [10:05:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [10:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25998.000000 [10:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:39:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25979 [10:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 81975 [10:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297855 MB (5% inode=32%): [11:02:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81968.000000 [11:05:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [11:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26011.000000 [11:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:39:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25953 [11:50:14] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82490 [11:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297515 MB (5% inode=32%): [12:02:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82746.000000 [12:06:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [12:12:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.613281/1.10, alarm hl:np_load_long=1.343750/1.55, alarm hl:mem_free=17671.000000M/500M, alarm hl:tmp_free=13653M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.613281/1.00, alarm hl:np_load_long=1.343750/1.50, alarm hl:mem_free=17671.000000M/600M, alarm hl:tmp_free [12:16:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25889.000000 [12:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:40:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25798 [12:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 83457 [12:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 297182 MB (5% inode=32%): [13:02:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83516.000000 [13:03:06] @replag [13:03:06] Dispenser: s1-rr-a: 7h 11m 14s [+0.01 s/s]; s1-user: 23h 12m 1s [+0.20 s/s] [13:06:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [13:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25861.000000 [13:28:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:40:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25988 [13:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84022 [13:51:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 296771 MB (5% inode=32%): [14:02:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84106.000000 [14:06:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [14:11:33] /sql on thyme is WARNING: DISK WARNING - free space: /sql 179482 MB (18% inode=99%): [14:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26933.000000 [14:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:37:44] hello all [14:41:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27308 [14:50:14] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 83958 [14:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 296512 MB (5% inode=32%): [15:02:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84027.000000 [15:04:24] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40732 MB (10% inode=99%): [15:07:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [15:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28188.000000 [15:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:38:23] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58512 MB (14% inode=99%): [15:42:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28640 [15:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 84737 [15:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 295825 MB (5% inode=32%): [16:02:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84824.000000 [16:05:53] hello DaB, is nightshade fully install yet? I've noticed oursql is missing from python and vim's defaults are different (somewhat expect). [16:06:15] @replag [16:06:15] Dispenser: s1-rr-a: 8h 8m 49s [+0.47 s/s]; s1-user: 23h 34m 38s [+0.14 s/s]; s3-rr-a: 44s [+0.01 s/s]; s3-user: 44s [+0.01 s/s]; s7-rr-a: 18s [+0.00 s/s]; s7-user: 18s [+0.00 s/s] [16:07:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [16:09:47] Dispenser: oursql? [16:10:12] python library for accessing MySQL [16:11:49] AFAIS I installed all python-liberies that were requested. If more are needed, please add them on https://wiki.toolserver.org/view/User:Dab/Debian-Packages [16:17:35] I just added strace to the list [16:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:20:16] We might need to build oursql. AFAIK there's no debian package, but its available using pip [http://packages.python.org/oursql/install.html#using-pip]. River had commented on this before [https://jira.toolserver.org/browse/TS-760] [16:21:30] Dispenser: you can always use a virtualenv [16:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29844.000000 [16:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:29:26] @replag [16:29:26] DarkoNeko: s1-rr-a: 8h 19m 47s [+0.47 s/s]; s1-user: 23h 39m 45s [+0.22 s/s]; s2-user: 2m 23s [+0.00 s/s]; s2-user-c: 2m 11s [+0.00 s/s]; s3-rr-a: 1m 7s [+0.02 s/s]; s3-user: 1m 7s [+0.02 s/s]; s5-user-c: 2m 11s [+0.00 s/s] [16:42:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30274 [16:44:03] rmytop is also missing [16:47:53] Dispenser: I can look if it compile under linux – the problem is that rmytop is a programm by River. [16:50:14] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85383 [16:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 295605 MB (5% inode=32%): [16:57:00] Dispenser: good news: It compiles. I will look to create a package of it [16:57:23] yay! [17:02:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85448.000000 [17:05:23] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106760 MB (10% inode=98%): [17:05:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106759 MB (10% inode=98%): [17:08:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [17:10:33] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107718 MB (11% inode=98%): [17:11:23] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107567 MB (11% inode=98%): [17:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:23:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 63278 MB (5% inode=99%): [17:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31163.000000 [17:28:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:43:32] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31358 [17:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85892 [17:51:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 295323 MB (5% inode=32%): [17:59:33] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:02:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86089.000000 [18:08:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [18:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:23:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31846.000000 [18:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:43:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31977 [18:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87093 [18:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 285976 MB (5% inode=31%): [18:55:23] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106904 MB (10% inode=98%): [18:59:34] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:00:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106840 MB (10% inode=98%): [19:02:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87200.000000 [19:08:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [19:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:24:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32602.000000 [19:28:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:44:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32982 [19:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 87993 [19:51:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 256932 MB (4% inode=29%): [20:00:33] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:02:23] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107462 MB (11% inode=98%): [20:02:33] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107462 MB (11% inode=98%): [20:02:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88053.000000 [20:08:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [20:17:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.291992/1.10, alarm hl:np_load_long=1.141601/1.55, alarm hl:mem_free=18221.000000M/500M, alarm hl:tmp_free=13176M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.291992/1.00, alarm hl:np_load_long=1.141601/1.50, alarm hl:mem_free=18221.000000M/600M, alarm hl:tmp_free [20:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:19:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:24:54] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33535.000000 [20:25:16] @replag [20:25:16] Betacommand: s1-rr-a: 9h 19m 0s [+0.25 s/s]; s1-user: 1d 35m 2s [+0.23 s/s]; s2-user: 13s [-0.01 s/s] [20:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:36:23] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106327 MB (10% inode=98%): [20:36:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106327 MB (10% inode=98%): [20:38:37] what where those .nfs* files again? [20:43:23] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107062 MB (11% inode=98%): [20:43:33] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107062 MB (11% inode=98%): [20:44:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33657 [20:50:14] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89015 [20:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 225001 MB (4% inode=26%): [20:52:23] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106892 MB (10% inode=98%): [20:52:34] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106892 MB (10% inode=98%): [21:00:33] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:02:44] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89081.000000 [21:08:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [21:09:59] what's up with the filesystem? it's so slow [21:14:37] my java programs are having fatal errors: "siginfo:si_signo=SIGBUS: si_errno=Stale NFS file handle" [21:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:18:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:24:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34020.000000 [21:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:36:23] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106791 MB (10% inode=98%): [21:36:34] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106790 MB (10% inode=98%): [21:44:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34333 [21:47:24] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107327 MB (11% inode=98%): [21:47:34] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107326 MB (11% inode=98%): [21:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89714 [21:51:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 276948 MB (5% inode=31%): [21:54:23] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106680 MB (10% inode=98%): [21:54:33] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106678 MB (10% inode=98%): [21:58:24] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41157 MB (10% inode=99%): [22:00:33] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:02:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=18.668945/1.95, alarm hl:tmp_free=49599M/100M, alarm hl:np_load_avg=9.655762/2.0, alarm hl:mem_free=1690.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=18.668945/2.3, alarm hl:np_load_long=5.199707/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=1690.000000M/200 [22:02:33] Load avg. on willow is CRITICAL: CRITICAL - load average: 136.12, 83.07, 44.98 [22:02:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89783.000000 [22:08:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [22:16:23] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107128 MB (11% inode=98%): [22:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:23:23] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106584 MB (10% inode=98%): [22:25:24] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 56293 MB (13% inode=99%): [22:25:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34920.000000 [22:27:44] i cannot submit sge jobs -.- [22:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:28:05] they're infinitely waiting [22:32:23] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107748 MB (11% inode=98%): [22:32:34] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107747 MB (11% inode=98%): [22:39:03] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [22:39:33] Load avg. on willow is WARNING: WARNING - load average: 6.48, 8.89, 19.26 [22:44:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35286 [22:47:33] Load avg. on willow is OK: OK - load average: 8.11, 8.09, 14.55 [22:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90363 [22:51:04] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 276696 MB (5% inode=31%): [23:00:33] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:03:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90437.000000 [23:08:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [23:16:17] huh? I thought you said submit "steve jobs" [23:16:21] I got a wtf moment [23:17:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:18:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:25:53] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35767.000000 [23:28:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:44:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35873 [23:48:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.238281/1.10, alarm hl:np_load_long=0.987305/1.55, alarm hl:mem_free=17411.000000M/500M, alarm hl:tmp_free=13031M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.238281/1.00, alarm hl:np_load_long=0.987305/1.50, alarm hl:mem_free=17411.000000M/600M, alarm hl:tmp_free [23:49:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [23:50:13] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 90828 [23:51:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 274902 MB (5% inode=30%):