[00:01:41] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 412668 MB (7% inode=40%): [00:13:12] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:17:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:19:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:22:02] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:28:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:31:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:37:51] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [00:50:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:59:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42730 MB (10% inode=99%): [01:01:41] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410048 MB (7% inode=40%): [01:13:42] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 66358 MB (16% inode=99%): [01:17:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:20:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:22:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:28:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:31:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:50:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:58:42] /sql on thyme is WARNING: DISK WARNING - free space: /sql 196063 MB (20% inode=99%): [02:01:42] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 411958 MB (7% inode=40%): [02:17:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:20:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:23:01] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:28:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:31:24] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:51:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:01:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410125 MB (7% inode=40%): [03:17:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:20:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:23:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:28:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:32:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:52:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:02:51] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410099 MB (7% inode=40%): [04:03:11] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:17:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:18:25] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.177734/1.10, alarm hl:np_load_long=0.360840/1.55, alarm hl:mem_free=322.000000M/500M, alarm hl:tmp_free=14558M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.177734/1.00, alarm hl:np_load_long=0.360840/1.50, alarm hl:mem_free=322.000000M/600M, alarm hl:tmp_free= [04:20:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:23:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:24:25] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [04:25:16] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41511 MB (10% inode=99%): [04:28:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:31:27] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.292480/1.00, alarm hl:np_load_long=0.309570/1.50, alarm hl:mem_free=561.000000M/600M, alarm hl:tmp_free=14581M/100M, alarm hl:available=1/0 [04:32:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:34:15] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22586 MB (5% inode=99%): [04:42:25] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [04:44:17] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44398 MB (10% inode=99%): [04:45:16] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 73706 MB (18% inode=99%): [04:52:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:03:17] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:03:17] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410052 MB (7% inode=40%): [05:17:56] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:20:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:21:36] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.267090/1.10, alarm hl:np_load_long=0.241699/1.55, alarm hl:mem_free=175.000000M/500M, alarm hl:tmp_free=14559M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.267090/1.00, alarm hl:np_load_long=0.241699/1.50, alarm hl:mem_free=175.000000M/600M, alarm hl:tmp_free= [05:24:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:28:36] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:32:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:38:04] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:38:34] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.148438/1.10, alarm hl:np_load_long=0.208984/1.55, alarm hl:mem_free=111.000000M/500M, alarm hl:tmp_free=14574M/200M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.148438/1.00, alarm hl:np_load_long=0.208984/1.50, alarm hl:mem_free=111.000000M/600M, alarm hl:tmp_free= [05:49:35] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [05:52:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:03:16] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 409943 MB (7% inode=40%): [06:18:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:20:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:24:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:28:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:32:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:52:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:03:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 409805 MB (7% inode=40%): [07:18:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:21:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:24:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:28:36] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:33:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:47:35] Load avg. on adenia is WARNING: WARNING - load average: 17.07, 12.33, 7.30 [07:51:35] Load avg. on adenia is OK: OK - load average: 14.76, 14.38, 9.39 [07:53:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:03:25] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 410259 MB (7% inode=40%): [08:18:25] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:21:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:24:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:28:37] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:33:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:42:24] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106345 MB (10% inode=98%): [08:42:45] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106263 MB (10% inode=98%): [08:44:36] Load avg. on adenia is WARNING: WARNING - load average: 15.32, 12.06, 8.10 [08:45:36] Load avg. on adenia is OK: OK - load average: 9.14, 10.98, 7.97 [08:53:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:03:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 408028 MB (7% inode=40%): [09:08:25] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107395 MB (11% inode=98%): [09:08:45] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107967 MB (11% inode=98%): [09:18:26] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:21:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:24:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:29:36] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:33:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:47:36] Load avg. on adenia is WARNING: WARNING - load average: 18.10, 11.46, 6.94 [09:48:28] could I request a query for commons that checks the pages that both {{tl|information}} and {{tl|location}} appear [09:48:54] it would be nice to know about how that relates to pages with {{LargeImage}} [09:53:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:59:36] Load avg. on adenia is OK: OK - load average: 14.10, 14.97, 11.82 [10:01:36] /sql on cassia is WARNING: DISK WARNING - free space: /sql 84416 MB (7% inode=99%): [10:03:35] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 407521 MB (7% inode=40%): [10:18:36] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:21:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:24:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:28:36] Load avg. on adenia is WARNING: WARNING - load average: 17.70, 11.82, 9.54 [10:29:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:33:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:53:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:55:36] Load avg. on adenia is OK: OK - load average: 12.77, 14.53, 14.98 [11:03:36] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 407449 MB (7% inode=40%): [11:19:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:21:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:24:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:29:57] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:33:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:48:57] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1847.000000 [11:53:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:03:57] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405634 MB (7% inode=40%): [12:19:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:21:06] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:25:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:28:15] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:30:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:33:06] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:38:14] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:46:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3628.000000 [12:54:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:02:25] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42090 MB (10% inode=99%): [13:04:56] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405630 MB (7% inode=40%): [13:11:26] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 21881 MB (5% inode=99%): [13:19:35] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:21:06] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:21:25] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 46911 MB (11% inode=99%): [13:25:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:30:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:33:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:33:44] 3(commented) [DRTRIGON-97] More options for sum_cat_disc <10https://jira.toolserver.org/browse/DRTRIGON-97> (drtrigon) [13:40:25] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36178 MB (8% inode=99%): [13:43:26] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24005 MB (5% inode=99%): [13:47:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5302.000000 [13:50:26] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 25542 MB (6% inode=99%): [13:54:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:59:35] /sql on thyme is WARNING: DISK WARNING - free space: /sql 192081 MB (20% inode=99%): [14:05:54] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405778 MB (7% inode=40%): [14:20:36] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:21:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:25:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:30:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:31:44] 3(deleted) [ACCAPP-520] PendingChanges tool for finnish Wikipedia (Marlen Caemmerer) [14:31:48] 3(resolved) [TS-1369] SSL certificate problem, bad/outdated HTTPS CA <10https://jira.toolserver.org/browse/TS-1369> (Marlen Caemmerer) [14:32:53] no krinkle... [14:33:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:48:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5035.000000 [14:54:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:59:43] 3(resolved) [TS-1405] Are database indexes on s5 missing? <10https://jira.toolserver.org/browse/TS-1405> (Marlen Caemmerer) [15:05:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405353 MB (7% inode=40%): [15:21:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:21:36] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:26:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:30:16] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:30:25] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 7747 MB (1% inode=99%): [15:31:25] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 79244 MB (19% inode=99%): [15:34:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:35:22] [[Special:Log/newusers]] create 10 * Julesli23 * (New user account) [15:48:05] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1957.000000 [15:48:15] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:48:24] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1978 [15:49:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6305.000000 [15:54:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:58:16] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:05:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405175 MB (7% inode=40%): [16:13:16] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:16:06] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3640.000000 [16:16:25] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3658 [16:21:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:21:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:26:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:30:16] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [16:34:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:36:46] 3(commented) [TS-1401] New SSH keypair <10https://jira.toolserver.org/browse/TS-1401> (inductiveload) [16:49:06] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6410.000000 [16:54:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:05:54] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 405019 MB (7% inode=39%): [17:16:15] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7244.000000 [17:16:24] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7258 [17:21:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:22:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:31:17] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:34:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:49:16] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6086.000000 [17:54:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:05:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 404966 MB (7% inode=39%): [18:16:16] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10845.000000 [18:16:25] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10859 [18:21:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:22:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:26:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:27:35] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42731 MB (10% inode=99%): [18:31:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:34:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:41:35] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 65513 MB (16% inode=99%): [18:49:16] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4419.000000 [18:54:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:05:54] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 404850 MB (7% inode=39%): [19:06:16] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3509.000000 [19:13:02] Dr. Trigon * [Toolserver-l] Fwd: Write attempts to wiki API from TS timeout [19:14:43] 3(commented) [DRTRIGON-113] Enable big data substitutions (in MB range) <10https://jira.toolserver.org/browse/DRTRIGON-113> (drtrigon) [19:16:35] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14462 [19:17:15] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14506.000000 [19:21:25] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:23:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:26:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:31:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:34:26] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:47:16] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [19:50:15] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1831.000000 [19:52:16] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1728.000000 [19:54:26] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:05:54] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 404762 MB (7% inode=39%): [20:16:36] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18068 [20:17:16] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18110.000000 [20:22:26] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:23:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:27:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:31:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [20:34:26] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:55:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:04:45] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70659 MB (5% inode=99%): [21:05:44] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71187 MB (6% inode=99%): [21:05:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 404597 MB (7% inode=39%): [21:07:44] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70882 MB (5% inode=99%): [21:16:46] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21671 [21:18:16] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21771.000000 [21:22:36] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:23:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:27:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:30:46] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 70498 MB (5% inode=99%): [21:31:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [21:34:36] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:55:35] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:05:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 404497 MB (7% inode=39%): [22:16:47] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25273 [22:18:25] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25373.000000 [22:22:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:23:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:27:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:31:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [22:34:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:55:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:05:55] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 404430 MB (7% inode=39%): [23:17:46] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28936 [23:18:27] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28974.000000 [23:22:46] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:23:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:27:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:31:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [23:34:46] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:55:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output