[00:02:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:07:22] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [00:08:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:12:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7603.000000 [00:13:22] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:15:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7641 [00:20:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 309369 MB (5% inode=33%): [00:21:32] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:21] NTP on damiana is CRITICAL: NTP CRITICAL: Offset -1.025764 secs [00:33:22] NTP on turnera is CRITICAL: NTP CRITICAL: Offset -1.010249 secs [00:34:12] is there any way i could have my own php.ini? [00:35:02] /tmp on willow is WARNING: DISK WARNING - free space: / 21980 MB (20% inode=99%): [00:35:02] / on willow is WARNING: DISK WARNING - free space: / 21980 MB (20% inode=99%): [00:35:02] NTP on hemlock is CRITICAL: NTP CRITICAL: Offset -1.005322 secs [00:36:21] NTP on damiana is OK: NTP OK: Offset -0.010015 secs [00:39:22] NTP on adenia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:41:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [00:43:03] NTP on yarrow is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:43:22] NTP on turnera is OK: NTP OK: Offset -0.012527 secs [00:45:21] NTP on adenia is OK: NTP OK: Offset 0.012414 secs [00:48:22] NTP on ortelius is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:49:21] NTP on rosemary is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:49:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29418.000000 [00:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29419 [00:50:21] NTP on cassia is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:51:22] NTP on ortelius is OK: NTP OK: Offset 0.021217 secs [00:53:22] NTP on cassia is OK: NTP OK: Offset 0.025006 secs [00:55:02] NTP on hemlock is OK: NTP OK: Offset 0.006603 secs [01:00:22] NTP on thyme is CRITICAL: NTP CRITICAL: Offset -1.009069 secs [01:01:02] NTP on wolfsbane is CRITICAL: NTP CRITICAL: Offset -1.030813 secs [01:01:21] NTP on rosemary is OK: NTP OK: Offset 0.024876 secs [01:02:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:02:22] NTP on nightshade is CRITICAL: NTP CRITICAL: Offset -1.006218 secs [01:02:31] NTP on hyacinth is CRITICAL: NTP CRITICAL: Offset -1.031231 secs [01:07:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [01:07:22] NTP on ptolemy is CRITICAL: NTP CRITICAL: Offset -1.024635 secs [01:08:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:08:21] NTP on daphne is CRITICAL: NTP CRITICAL: Offset -1.027053 secs [01:08:31] NTP on hyacinth is OK: NTP OK: Offset -0.035108 secs [01:11:21] NTP on thyme is OK: NTP OK: Offset 0.009748 secs [01:12:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8023.000000 [01:13:21] NTP on adenia is CRITICAL: NTP CRITICAL: Offset -1.029839 secs [01:13:22] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:14:22] NTP on rosemary is CRITICAL: NTP CRITICAL: Offset -1.028731 secs [01:15:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8039 [01:19:22] NTP on ptolemy is OK: NTP OK: Offset -0.002164 secs [01:21:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 309281 MB (5% inode=33%): [01:21:31] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:22:02] NTP on yarrow is OK: NTP OK: Offset 0.009148 secs [01:26:22] NTP on rosemary is OK: NTP OK: Offset -0.005946 secs [01:31:02] NTP on wolfsbane is OK: NTP OK: Offset 0.006881 secs [01:32:22] NTP on nightshade is OK: NTP OK: Offset 0.006453 secs [01:36:03] /tmp on willow is WARNING: DISK WARNING - free space: / 21937 MB (20% inode=99%): [01:36:03] / on willow is WARNING: DISK WARNING - free space: / 21937 MB (20% inode=99%): [01:36:21] NTP on daphne is OK: NTP OK: Offset 0.007291 secs [01:37:22] NTP on adenia is OK: NTP OK: Offset 0.001235 secs [01:41:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [01:49:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29975.000000 [01:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 29975 [02:02:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:07:22] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [02:08:04] /sql on thyme is WARNING: DISK WARNING - free space: /sql 184727 MB (19% inode=99%): [02:08:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:12:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8315.000000 [02:13:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:15:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8285 [02:21:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316033 MB (5% inode=34%): [02:21:31] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:28:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.166016/1.10, alarm hl:np_load_long=0.782227/1.55, alarm hl:mem_free=17854.000000M/500M, alarm hl:tmp_free=13429M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.166016/1.00, alarm hl:np_load_long=0.782227/1.50, alarm hl:mem_free=17854.000000M/600M, alarm hl:tmp_free [02:33:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:36:04] /tmp on willow is WARNING: DISK WARNING - free space: / 21896 MB (20% inode=99%): [02:36:04] / on willow is WARNING: DISK WARNING - free space: / 21896 MB (20% inode=99%): [02:42:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [02:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30341 [02:50:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30349.000000 [03:02:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:08:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:08:22] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [03:12:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8619.000000 [03:14:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:15:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8583 [03:21:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317614 MB (5% inode=34%): [03:21:33] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:36:04] /tmp on willow is WARNING: DISK WARNING - free space: / 21836 MB (20% inode=99%): [03:36:04] / on willow is WARNING: DISK WARNING - free space: / 21836 MB (20% inode=99%): [03:42:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [03:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31092 [03:50:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31101.000000 [04:02:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:08:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:08:23] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [04:11:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.000488/1.95, alarm hl:tmp_free=21805M/100M, alarm hl:np_load_avg=1.630371/2.0, alarm hl:mem_free=1797.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.000488/2.3, alarm hl:np_load_long=1.220703/2.5, alarm hl:cpu=100.000000/98, alarm hl:mem_free=1797.000000M/200M, [04:12:51] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8888.000000 [04:14:23] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow OK: longrun-sol@willow OK [04:15:22] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8926 [04:22:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317042 MB (5% inode=34%): [04:22:31] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:02] / on willow is WARNING: DISK WARNING - free space: / 21778 MB (20% inode=99%): [04:37:02] /tmp on willow is WARNING: DISK WARNING - free space: / 21778 MB (20% inode=99%): [04:43:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [04:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32110 [04:51:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32126.000000 [05:03:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:09:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:09:22] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [05:13:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9882.000000 [05:15:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9929 [05:21:02] /sql on cassia is WARNING: DISK WARNING - free space: /sql 71006 MB (6% inode=99%): [05:22:03] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 66506 MB (5% inode=99%): [05:22:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316693 MB (5% inode=34%): [05:22:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:02] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36773 MB (9% inode=99%): [05:36:03] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24045 MB (5% inode=99%): [05:37:03] / on willow is WARNING: DISK WARNING - free space: / 21713 MB (20% inode=99%): [05:37:03] /tmp on willow is WARNING: DISK WARNING - free space: / 21713 MB (20% inode=99%): [05:43:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [05:49:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33546 [05:51:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33563.000000 [06:00:03] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 69350 MB (17% inode=99%): [06:03:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:09:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [06:10:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:13:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10444.000000 [06:16:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:16:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10460 [06:22:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316395 MB (5% inode=34%): [06:26:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37441 MB (9% inode=99%): [06:37:03] / on willow is WARNING: DISK WARNING - free space: / 21661 MB (20% inode=99%): [06:37:03] /tmp on willow is WARNING: DISK WARNING - free space: / 21661 MB (20% inode=99%): [06:42:03] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24427 MB (5% inode=99%): [06:43:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [06:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34026 [06:51:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34027.000000 [07:01:21] Load avg. on adenia is WARNING: WARNING - load average: 18.39, 16.77, 12.91 [07:03:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:09:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [07:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:12:02] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 66124 MB (16% inode=99%): [07:13:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10705.000000 [07:16:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10715 [07:17:22] Load avg. on adenia is OK: OK - load average: 8.79, 14.22, 14.78 [07:22:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:23:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 316154 MB (5% inode=34%): [07:37:03] / on willow is WARNING: DISK WARNING - free space: / 21608 MB (20% inode=99%): [07:37:03] /tmp on willow is WARNING: DISK WARNING - free space: / 21608 MB (20% inode=99%): [07:44:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [07:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33893 [07:52:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33882.000000 [08:03:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:09:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [08:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:13:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10686.000000 [08:16:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10675 [08:22:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:23:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 315599 MB (5% inode=34%): [08:38:02] / on willow is WARNING: DISK WARNING - free space: / 21550 MB (20% inode=99%): [08:38:02] /tmp on willow is WARNING: DISK WARNING - free space: / 21550 MB (20% inode=99%): [08:45:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [08:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34060 [08:53:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34091.000000 [09:03:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:09:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [09:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:14:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10591.000000 [09:16:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10593 [09:22:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 315550 MB (5% inode=34%): [09:38:02] / on willow is WARNING: DISK WARNING - free space: / 21489 MB (20% inode=99%): [09:38:03] /tmp on willow is WARNING: DISK WARNING - free space: / 21489 MB (20% inode=99%): [09:46:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [09:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35002 [09:53:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35045.000000 [10:03:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:09:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [10:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:14:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10628.000000 [10:16:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10627 [10:22:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:25:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 315779 MB (5% inode=34%): [10:39:02] /tmp on willow is WARNING: DISK WARNING - free space: / 21380 MB (20% inode=98%): [10:39:03] / on willow is WARNING: DISK WARNING - free space: / 21380 MB (20% inode=98%): [10:46:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [10:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35601 [10:53:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35617.000000 [11:03:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:09:22] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [11:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:14:10] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10734.000000 [11:15:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.336914/1.10, alarm hl:np_load_long=1.022461/1.55, alarm hl:mem_free=20393.000000M/500M, alarm hl:tmp_free=13903M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.336914/1.00, alarm hl:np_load_long=1.022461/1.50, alarm hl:mem_free=20393.000000M/600M, alarm hl:tmp_free [11:16:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10754 [11:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 315157 MB (5% inode=33%): [11:35:01] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:39:03] /tmp on willow is WARNING: DISK WARNING - free space: / 21378 MB (20% inode=98%): [11:39:03] / on willow is WARNING: DISK WARNING - free space: / 21378 MB (20% inode=98%): [11:46:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [11:49:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36096 [11:53:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36142.000000 [12:03:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:10:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [12:14:12] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10650.000000 [12:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10655 [12:17:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:25:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 314478 MB (5% inode=33%): [12:39:03] /tmp on willow is WARNING: DISK WARNING - free space: / 21322 MB (20% inode=98%): [12:39:03] / on willow is WARNING: DISK WARNING - free space: / 21322 MB (20% inode=98%): [12:46:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [12:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36555 [12:53:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36581.000000 [13:00:01] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42924 MB (10% inode=99%): [13:03:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:10:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [13:14:21] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10785.000000 [13:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10774 [13:17:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 313854 MB (5% inode=33%): [13:34:03] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 57137 MB (14% inode=99%): [13:39:02] /tmp on willow is WARNING: DISK WARNING - free space: / 21208 MB (20% inode=98%): [13:39:03] / on willow is WARNING: DISK WARNING - free space: / 21208 MB (20% inode=98%): [13:46:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [13:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37160 [13:54:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37159.000000 [14:03:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:09:03] /sql on thyme is WARNING: DISK WARNING - free space: /sql 183925 MB (19% inode=99%): [14:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:10:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [14:14:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10723.000000 [14:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10705 [14:18:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 313101 MB (5% inode=33%): [14:40:02] /tmp on willow is WARNING: DISK WARNING - free space: / 21174 MB (20% inode=98%): [14:40:02] / on willow is WARNING: DISK WARNING - free space: / 21174 MB (20% inode=98%): [14:47:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [14:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37545 [14:54:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37575.000000 [15:04:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:10:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:10:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [15:14:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10544.000000 [15:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10572 [15:18:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:25:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 312376 MB (5% inode=33%): [15:40:02] /tmp on willow is WARNING: DISK WARNING - free space: / 21147 MB (20% inode=98%): [15:40:02] / on willow is WARNING: DISK WARNING - free space: / 21147 MB (20% inode=98%): [15:47:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [15:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38006 [15:54:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37983.000000 [16:04:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:10:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:10:22] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [16:15:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10692.000000 [16:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10706 [16:18:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:25:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 311629 MB (5% inode=33%): [16:40:01] /tmp on willow is WARNING: DISK WARNING - free space: / 21095 MB (20% inode=98%): [16:40:01] / on willow is WARNING: DISK WARNING - free space: / 21095 MB (20% inode=98%): [16:48:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [16:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37870 [16:54:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37881.000000 [17:04:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:11:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [17:15:21] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11003.000000 [17:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10997 [17:19:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:23:01] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 66063 MB (5% inode=99%): [17:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 310577 MB (5% inode=33%): [17:41:03] /tmp on willow is WARNING: DISK WARNING - free space: / 21040 MB (20% inode=98%): [17:41:03] / on willow is WARNING: DISK WARNING - free space: / 21040 MB (20% inode=98%): [17:48:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [17:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38453 [17:54:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38498.000000 [18:05:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:11:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [18:15:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11143.000000 [18:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11141 [18:19:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:23:42] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:26:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 309943 MB (5% inode=33%): [18:42:02] /tmp on willow is WARNING: DISK WARNING - free space: / 20983 MB (20% inode=98%): [18:42:03] / on willow is WARNING: DISK WARNING - free space: / 20983 MB (20% inode=98%): [18:48:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [18:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38817 [18:54:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38838.000000 [19:06:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:11:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [19:13:26] is there anything anyone can do to address the replag on s1? [19:15:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11588.000000 [19:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11587 [19:19:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:26:01] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 307718 MB (5% inode=33%): [19:43:02] /tmp on willow is WARNING: DISK WARNING - free space: / 20934 MB (19% inode=98%): [19:43:02] / on willow is WARNING: DISK WARNING - free space: / 20934 MB (19% inode=98%): [19:48:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [19:49:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38662 [19:54:02] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.006836/1.00, alarm hl:np_load_long=0.996094/1.50, alarm hl:mem_free=19933.000000M/600M, alarm hl:tmp_free=13377M/100M, alarm hl:available=1/0 [19:55:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38631.000000 [19:55:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:06:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:10:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:11:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [20:16:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12076.000000 [20:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12080 [20:19:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 305597 MB (5% inode=33%): [20:43:02] / on willow is WARNING: DISK WARNING - free space: / 20889 MB (19% inode=98%): [20:43:02] /tmp on willow is WARNING: DISK WARNING - free space: / 20889 MB (19% inode=98%): [20:48:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [20:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38625 [20:55:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38667.000000 [21:06:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:10:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:11:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [21:16:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12065.000000 [21:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12060 [21:19:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 305355 MB (5% inode=33%): [21:43:02] / on willow is WARNING: DISK WARNING - free space: / 20790 MB (19% inode=98%): [21:43:02] /tmp on willow is WARNING: DISK WARNING - free space: / 20790 MB (19% inode=98%): [21:49:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [21:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38958 [21:55:11] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38973.000000 [22:06:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:12:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [22:16:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12587.000000 [22:16:41] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12587 [22:19:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:23:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:28:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 305303 MB (5% inode=33%): [22:43:02] / on willow is WARNING: DISK WARNING - free space: / 20798 MB (19% inode=98%): [22:43:02] /tmp on willow is WARNING: DISK WARNING - free space: / 20798 MB (19% inode=98%): [22:49:41] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38951 [22:50:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [22:55:11] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38938.000000 [23:07:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:10:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:11:21] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:12:21] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [23:16:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12703 [23:17:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12704.000000 [23:19:31] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:24:23] inductiveload: Don't use WikiProxy [23:24:41] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:25:07] It slower than pulling text from across the Atlantic [23:28:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 303949 MB (5% inode=33%): [23:31:12] Dispenser: hi there! thanks for helping. what should i do, query directly? [23:31:40] What's this for anyway? [23:32:04] and what programming language [23:33:04] i was just wondering how best to query page text from the toolserver. language is probably php or python [23:35:55] Generally API. For creating a bot there libraries in Perl, Python, PHP, and more. Most people on IRC are knowledgeable with PHP or Python [23:36:03] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:36:04] Is it a web tool? [23:36:14] yes [23:36:23] what kind? [23:36:25] i already know pywikipediabot [23:36:43] s/know/can use [23:37:07] it is for analysing template parameters on enWS [23:37:30] That's actually on my todo list [23:38:24] best go for a dump and figure out a scheme for incremental updating [23:40:05] ok, so i get one of the XML dumps and feed it to a local install of MW? [23:41:12] Incomplete effort of template parameters: http://toolserver.org/~kolossos/templatetiger/ [23:42:21] https://wiki.toolserver.org/view/User-store <- Info on our local dumps [23:43:03] / on willow is WARNING: DISK WARNING - free space: / 20763 MB (19% inode=98%): [23:43:03] /tmp on willow is WARNING: DISK WARNING - free space: / 20763 MB (19% inode=98%): [23:44:32] You'll have to hack MW to get an useable template information out [23:46:49] thanks, that's very helpful [23:48:43] are you supposed to generate the file for a wiki not on the list yourself, or is there a request process? [23:49:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39127 [23:50:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [23:55:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39088.000000