[00:02:34] @replag [00:02:35] DaBPunkt: s1-sec: 1m 4s [+0.00 s/s]; s2-pri: 16s [+0.00 s/s]; s2/s5-pri-c: 2h 9m 0s [-1.79 s/s]; s3-rr: 2m 24s [+0.00 s/s]; s3-user: 2m 24s [+0.00 s/s]; s4-user: error; s6-rr: 24s [+0.00 s/s]; s6-user: 24s [+0.00 s/s] [00:02:36] DaBPunkt: s7-rr: 3m 11s [+0.01 s/s]; s7-user: 3m 11s [+0.01 s/s] [00:03:01] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.579590/1.75, alarm hl:np_load_avg=0.575195/2.00, alarm hl:mem_free=268.000000M/300M [00:06:00] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:08:20] 3(resolved) [UTRS-55] Backlog <10https://jira.toolserver.org/browse/UTRS-55> (Andrew Pearson) [00:10:21] 3(work logged) [UTRS-55] Backlog <10https://jira.toolserver.org/browse/UTRS-55> (Andrew Pearson) [00:13:00] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.438476/1.75, alarm hl:np_load_avg=0.425293/2.00, alarm hl:mem_free=224.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.438476/1.50, alarm hl:np_load_long=0.396484/1.75, alarm hl:mem_free=224.000000M/250M [00:13:19] Free Memory on damiana.esi is CRITICAL: CRITICAL - 5.7% (238604 kB) free! [00:20:19] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3437.000000 [00:24:20] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1582.000000 [00:24:22] 3(commented) [ACCAPP-453] I plan to use the toolserver for programming of my bot. <10https://jira.toolserver.org/browse/ACCAPP-453> (Cyberpower678) [00:30:19] Free Memory on damiana.esi is WARNING: WARNING - 7.1% (296176 kB) free! [00:38:03] Toolserver down? [00:38:08] Or just for me? [00:38:22] wait a moment [00:38:51] ok [00:41:54] Here we go again [00:44:03] jem-? [00:44:20] * mmovchin is sad because toolserver is down :( [00:47:26] Sun Grid Engine execd on nightshade is UNKNOWN: Invalid host name nightshade [00:47:26] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [00:48:05] Free Memory on damiana.esi is OK: OK - 50.4% (2111068 kB) free. [00:48:05] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41056 MB (4% inode=99%): [00:48:05] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:48:15] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [00:48:15] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [00:48:21] Thanks, toolserver up again [00:49:09] Nightshade's still not accepting my private key. [00:49:29] Oop, there it goes. It just lagged for about two minutes. :x [00:49:39] @replag [00:49:40] MZMcBride: s1-pri: 10m 40s [-]; s1-sec: 7m 49s [-]; s2-pri: 2m 20s [-]; s3-rr: 4m 21s [-]; s3-user: 4m 21s [-]; s4-user: error; s6-rr: 2m 37s [-]; s6-user: 2m 37s [-] [00:49:41] MZMcBride: s7-rr: 3m 54s [-]; s7-user: 3m 54s [-] [00:49:54] I am switching a service over, I could hang for a moment [00:50:05] NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.005366 secs [00:50:15] NTP on damiana.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.003734 secs [00:51:07] I have a PHP script that died due to a connection but won't Ctrl+C. o.O [00:51:20] And I can't seem to see it on ps aux. Is there a way to just kill all processes running in a given screen? [00:51:37] due to a connection timeout* [00:52:15] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [00:52:15] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [00:52:29] (Got it.) [00:59:13] @replag [00:59:13] mmovchin: s3-rr: 10s [-0.44 s/s]; s3-user: 10s [-0.44 s/s]; s4-user: error [01:00:15] Cluster on damiana.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, ldap OFFLINE, check nfs-hasp, ds--global-misc-ldap OFFLINE, [01:00:15] Cluster on turnera.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, ldap OFFLINE, check nfs-hasp, ds--global-misc-ldap OFFLINE, [01:01:17] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [01:01:17] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [01:01:32] oh no :( [01:02:15] Sun Grid Engine execd on willow is CRITICAL: all.q@willow in error state: QERROR as result of job 1608069s failure: longrun@willow in error state: QERROR as result of job 1608069s failure [01:03:05] NTP on turnera.esi is OK: NTP OK: Offset -0.005686 secs [01:03:15] NTP on damiana.esi is OK: NTP OK: Offset 0.005248 secs [01:05:15] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:05:27] 3(created) [MNT-1186] Add more nameserver to /etc/resolv.conf; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1186> (DaB.) [01:05:28] 3(resolved) [MNT-1186] Add more nameserver to /etc/resolv.conf <10https://jira.toolserver.org/browse/MNT-1186> (DaB.) [01:09:23] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [01:09:30] Free Memory on damiana.esi is OK: OK - 50.5% (2115140 kB) free. [01:09:40] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:09:40] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41031 MB (4% inode=99%): [01:10:10] Cluster on turnera.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, ldap OFFLINE, check nfs-hasp, ds--global-misc-ldap OFFLINE, [01:11:09] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [01:12:10] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [01:13:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:44:18] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, check nagios, [01:44:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40945 MB (4% inode=99%): [01:45:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:45:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:49:10] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [01:49:17] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [01:49:17] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40934 MB (4% inode=99%): [01:49:17] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [01:49:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:50:18] / on adenia is CRITICAL: NRPE: Command check_root not defined [01:50:27] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [01:50:27] Cluster on rosemary is CRITICAL: NRPE: Command check_scstat not defined [01:50:27] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:50:27] Cluster on willow is CRITICAL: NRPE: Command check_scstat not defined [01:50:37] / on cassia is CRITICAL: NRPE: Command check_root not defined [01:50:37] Cluster on daphne is CRITICAL: NRPE: Command check_scstat not defined [01:50:37] Cluster on adenia is CRITICAL: NRPE: Command check_scstat not defined [01:50:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:50:57] Cluster on cassia is CRITICAL: NRPE: Command check_scstat not defined [01:50:57] Cluster on hemlock is CRITICAL: NRPE: Command check_scstat not defined [01:51:07] / on hyacinth is CRITICAL: NRPE: Command check_root not defined [01:51:07] / on thyme is CRITICAL: NRPE: Command check_root not defined [01:51:07] / on rosemary is CRITICAL: NRPE: Command check_root not defined [01:51:15] good night [01:51:17] / on daphne is CRITICAL: NRPE: Command check_root not defined [01:51:27] Cluster on hyacinth is CRITICAL: NRPE: Command check_scstat not defined [01:51:27] Cluster on thyme is CRITICAL: NRPE: Command check_scstat not defined [01:53:21] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [01:53:28] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [01:53:28] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40929 MB (4% inode=99%): [01:53:28] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [01:53:38] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [01:53:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:55:04] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [01:55:04] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40925 MB (4% inode=99%): [01:55:04] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [01:55:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:55:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:56:03] / on adenia is CRITICAL: NRPE: Command check_root not defined [01:56:14] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [01:56:14] Cluster on rosemary is CRITICAL: NRPE: Command check_scstat not defined [01:56:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:56:14] Cluster on willow is CRITICAL: NRPE: Command check_scstat not defined [01:56:23] / on cassia is CRITICAL: NRPE: Command check_root not defined [01:56:24] Cluster on daphne is CRITICAL: NRPE: Command check_scstat not defined [01:56:24] Cluster on adenia is CRITICAL: NRPE: Command check_scstat not defined [01:56:44] Cluster on cassia is CRITICAL: NRPE: Command check_scstat not defined [01:56:44] Cluster on hemlock is CRITICAL: NRPE: Command check_scstat not defined [01:56:54] / on hyacinth is CRITICAL: NRPE: Command check_root not defined [01:56:54] / on thyme is CRITICAL: NRPE: Command check_root not defined [01:56:54] / on rosemary is CRITICAL: NRPE: Command check_root not defined [01:57:59] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, check nagios, [01:57:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:58:06] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40918 MB (4% inode=99%): [01:59:06] / on adenia is CRITICAL: NRPE: Command check_root not defined [01:59:16] / on hyacinth is CRITICAL: NRPE: Command check_root not defined [01:59:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:59:45] Cluster on willow is CRITICAL: NRPE: Command check_scstat not defined [01:59:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:59:46] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [01:59:56] / on rosemary is CRITICAL: NRPE: Command check_root not defined [01:59:56] / on thyme is CRITICAL: NRPE: Command check_root not defined [02:00:06] / on daphne is CRITICAL: NRPE: Command check_root not defined [02:00:06] / on cassia is CRITICAL: NRPE: Command check_root not defined [02:02:21] 3(created) [MNT-1187] Re-Configure nagios; Maintenance; Minor work <10https://jira.toolserver.org/browse/MNT-1187> (DaB.) [02:05:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:05:01] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [02:05:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 40892 MB (4% inode=99%): [02:05:21] Cluster on willow is CRITICAL: NRPE: Command check_scstat not defined [02:05:41] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:06:01] / on adenia is CRITICAL: NRPE: Command check_root not defined [02:06:21] MySQL on hyacinth is CRITICAL: Cant connect to MySQL server on hyacinth (146) [02:06:21] /sql on daphne is CRITICAL: NRPE: Command check_sql not defined [02:06:21] MySQL slave on hyacinth is CRITICAL: Cant connect to MySQL server on hyacinth (146) [02:06:21] / on cassia is CRITICAL: NRPE: Command check_root not defined [02:06:21] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [02:06:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:06:31] / on hyacinth is CRITICAL: NRPE: Command check_root not defined [02:06:41] / on thyme is CRITICAL: NRPE: Command check_root not defined [02:06:41] MySQL slave on adenia is WARNING: No slaves defined [02:06:51] / on rosemary is CRITICAL: NRPE: Command check_root not defined [02:07:00] / on daphne is CRITICAL: NRPE: Command check_root not defined [02:09:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41804 MB (4% inode=99%): [02:09:14] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [02:09:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:09:22] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [02:09:32] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [02:10:11] Cluster on willow is CRITICAL: NRPE: Command check_scstat not defined [02:10:21] / on adenia is CRITICAL: NRPE: Command check_root not defined [02:10:32] /sql on daphne is CRITICAL: NRPE: Command check_sql not defined [02:10:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:10:42] / on cassia is CRITICAL: NRPE: Command check_root not defined [02:10:52] / on thyme is CRITICAL: NRPE: Command check_root not defined [02:11:02] / on hyacinth is CRITICAL: NRPE: Command check_root not defined [02:11:02] / on rosemary is CRITICAL: NRPE: Command check_root not defined [02:11:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:11:12] MySQL slave on adenia is WARNING: No slaves defined [02:11:22] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [02:11:22] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:11:22] / on daphne is CRITICAL: NRPE: Command check_root not defined [02:25:13] @replag all [02:25:13] jase99: s1-pri: 1s [-0.11 s/s]; s1-sec: 1s [-0.08 s/s]; s1-sec-c: 3s [-]; s2-pri: 18s [-0.02 s/s]; s2/s5-pri-c: 3s [-]; s3-rr: 3m 11s [+0.04 s/s]; s3-user: 3m 11s [+0.04 s/s]; s4-rr: 3s [-] [02:25:14] jase99: s4-user: error; s5-rr: 10s [-]; s5-user: 10s [-]; s6-rr: 5s [-0.03 s/s]; s6-user: 5s [-0.03 s/s]; s7-rr: 11s [-0.04 s/s]; s7-user: 11s [-0.04 s/s] [02:33:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:33:19] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, check nagios, [02:33:19] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:33:26] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:33:36] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [02:34:27] / on adenia is CRITICAL: NRPE: Command check_root not defined [02:34:36] /sql on daphne is CRITICAL: NRPE: Command check_sql not defined [02:34:36] / on thyme is CRITICAL: NRPE: Command check_root not defined [02:34:36] NTP on z-dat-s6-a is CRITICAL: NTP CRITICAL: No response from NTP server [02:34:46] / on z-dat-s3-a is CRITICAL: NRPE: Command check_root not defined [02:34:46] NTP on z-dat-s7-a is CRITICAL: NTP CRITICAL: No response from NTP server [02:34:46] / on cassia is CRITICAL: NRPE: Command check_root not defined [02:34:57] / on rosemary is CRITICAL: NRPE: Command check_root not defined [02:34:57] / on z-dat-s4-a is CRITICAL: NRPE: Command check_root not defined [02:34:57] / on hyacinth is CRITICAL: NRPE: Command check_root not defined [02:35:06] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:35:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:35:07] Cluster on willow is CRITICAL: NRPE: Command check_scstat not defined [02:35:07] Load avg. on z-dat-s3-a is CRITICAL: NRPE: Command check_load not defined [02:35:07] / on z-dat-s6-a is CRITICAL: NRPE: Command check_root not defined [02:35:16] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [02:35:16] Load avg. on z-dat-s4-a is CRITICAL: NRPE: Command check_load not defined [02:35:16] / on z-dat-s7-a is CRITICAL: NRPE: Command check_root not defined [02:35:16] MySQL slave on adenia is WARNING: No slaves defined [02:35:27] NTP on z-dat-s3-a is CRITICAL: NTP CRITICAL: No response from NTP server [02:35:27] Load avg. on z-dat-s6-a is CRITICAL: NRPE: Command check_load not defined [02:35:36] / on daphne is CRITICAL: NRPE: Command check_root not defined [02:35:36] NTP on z-dat-s4-a is CRITICAL: NTP CRITICAL: No response from NTP server [02:35:36] Load avg. on z-dat-s7-a is CRITICAL: NRPE: Command check_load not defined [02:35:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:36:45] / on cassia is OK: DISK OK - free space: / 12166 MB (60% inode=92%): [02:40:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:40:24] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, check nagios, [02:40:24] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:40:30] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:40:30] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, check nagios, [02:41:30] / on adenia is CRITICAL: NRPE: Command check_root not defined [02:41:30] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 14923 MB (1% inode=99%): [02:41:40] / on thyme is CRITICAL: NRPE: Command check_root not defined [02:41:40] NTP on z-dat-s6-a is CRITICAL: NTP CRITICAL: No response from NTP server [02:41:50] / on z-dat-s3-a is CRITICAL: NRPE: Command check_root not defined [02:41:50] NTP on z-dat-s7-a is CRITICAL: NTP CRITICAL: No response from NTP server [02:42:00] / on rosemary is CRITICAL: NRPE: Command check_root not defined [02:42:00] / on z-dat-s4-a is CRITICAL: NRPE: Command check_root not defined [02:42:00] / on hyacinth is CRITICAL: NRPE: Command check_root not defined [02:42:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:42:00] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:42:09] Cluster on willow is CRITICAL: NRPE: Command check_scstat not defined [02:42:09] Load avg. on z-dat-s3-a is CRITICAL: NRPE: Command check_load not defined [02:42:10] / on z-dat-s6-a is CRITICAL: NRPE: Command check_root not defined [02:42:19] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [02:42:19] Load avg. on z-dat-s4-a is CRITICAL: NRPE: Command check_load not defined [02:42:20] MySQL slave on adenia is WARNING: No slaves defined [02:42:29] NTP on z-dat-s3-a is CRITICAL: NTP CRITICAL: No response from NTP server [02:42:29] Load avg. on z-dat-s6-a is CRITICAL: NRPE: Command check_load not defined [02:42:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:42:41] NTP on z-dat-s4-a is CRITICAL: NTP CRITICAL: No response from NTP server [02:45:21] wilow needs a reboot [02:45:25] willow [02:47:24] and back [02:50:16] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:50:16] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, check nagios, [02:50:16] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:50:22] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, check nagios, [02:51:02] Load avg. on z-dat-s4-a is CRITICAL: NRPE: Command check_load not defined [02:51:12] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [02:51:22] / on adenia is CRITICAL: NRPE: Command check_root not defined [02:51:22] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 14919 MB (1% inode=99%): [02:51:32] / on thyme is CRITICAL: NRPE: Command check_root not defined [02:51:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/application/sge/execd:toolserver [02:51:42] / on z-dat-s3-a is CRITICAL: NRPE: Command check_root not defined [02:51:42] / on z-dat-s4-a is CRITICAL: NRPE: Command check_root not defined [02:51:42] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [02:51:52] / on rosemary is CRITICAL: NRPE: Command check_root not defined [02:51:52] / on z-dat-s6-a is CRITICAL: NRPE: Command check_root not defined [02:51:52] / on hyacinth is CRITICAL: NRPE: Command check_root not defined [02:51:52] Sun Grid Engine execd on willow is CRITICAL: all.q@willow in unknown state: longrun@willow in unknown state [02:52:02] Cluster on willow is CRITICAL: NRPE: Command check_scstat not defined [02:52:02] Load avg. on z-dat-s3-a is CRITICAL: NRPE: Command check_load not defined [02:52:11] MySQL slave on adenia is WARNING: No slaves defined [02:52:12] Load avg. on z-dat-s6-a is CRITICAL: NRPE: Command check_load not defined [02:52:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:54:02] Load avg. on z-dat-s4-a is OK: OK - load average: 2.47, 1.95, 1.98 [02:54:42] / on z-dat-s4-a is OK: DISK OK - free space: / 11754 MB (39% inode=87%): [02:54:52] / on hyacinth is OK: DISK OK - free space: / 11754 MB (39% inode=87%): [03:05:22] 3(commented) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (DaB.) [03:05:38] @replag [03:05:38] DaBPunkt: s1-pri: 11s [+0.00 s/s]; s4-user: error [03:08:52] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [03:09:32] SMF on willow is OK: OK - all services online [03:12:01] Load avg. on z-dat-s3-a is OK: OK - load average: 2.41, 1.75, 1.69 [03:12:42] / on z-dat-s3-a is OK: DISK OK - free space: / 11759 MB (39% inode=87%): [03:13:02] Cluster on willow is OK: CLUSTER OK ! [03:17:20] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, check nagios, [03:17:20] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [03:17:27] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, check nagios, [03:18:27] / on adenia is CRITICAL: NRPE: Command check_root not defined [03:18:37] / on thyme is CRITICAL: NRPE: Command check_root not defined [03:18:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [03:18:47] / on rosemary is CRITICAL: NRPE: Command check_root not defined [03:18:47] / on z-dat-s6-a is CRITICAL: NRPE: Command check_root not defined [03:19:07] Cluster on nightshade is CRITICAL: NRPE: Command check_scstat not defined [03:19:07] Load avg. on z-dat-s6-a is CRITICAL: NRPE: Command check_load not defined [03:19:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:19:27] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 14905 MB (1% inode=99%): [03:26:46] / on z-dat-s6-a is OK: DISK OK - free space: / 11759 MB (39% inode=87%): [03:27:06] Load avg. on z-dat-s6-a is OK: OK - load average: 2.05, 1.72, 1.68 [03:29:06] Cluster on nightshade is OK: CLUSTER OK ! [03:31:27] / on adenia is OK: DISK OK - free space: / 8862 MB (44% inode=94%): [03:35:47] / on rosemary is OK: DISK OK - free space: / 10438 MB (52% inode=92%): [03:42:21] 3(commented) [MNT-1187] Re-Configure nagios <10https://jira.toolserver.org/browse/MNT-1187> (DaB.) [03:48:13] nacht ts [03:49:37] / on thyme is OK: DISK OK - free space: / 10239 MB (51% inode=92%): [04:11:17] Load avg. on nightshade is WARNING: WARNING - load average: 28.79, 18.55, 14.02 [04:15:17] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 35.23, 24.92, 17.59 [04:16:47] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:16:57] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [04:16:57] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:17:07] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:17:17] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [04:17:17] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [04:18:37] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [04:19:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:20:17] Load avg. on nightshade is WARNING: WARNING - load average: 25.68, 24.90, 19.78 [04:33:04] aude * [Toolserver-l] Wikimania scholarship deadline: Feb 16 [05:16:47] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:17:07] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [05:17:07] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:17:17] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [05:17:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:17:17] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [05:18:47] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [05:19:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:16:47] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:17:07] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [06:17:07] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:17:17] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [06:17:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:17:17] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [06:18:47] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [06:19:27] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:42:17] Load avg. on nightshade is WARNING: WARNING - load average: 21.33, 14.71, 11.56 [06:45:17] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 32.48, 20.41, 14.23 [06:46:17] Load avg. on nightshade is WARNING: WARNING - load average: 26.32, 20.85, 14.79 [07:16:56] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:17:27] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:17:27] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [07:17:27] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [07:17:27] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [07:17:27] Load avg. on nightshade is OK: OK - load average: 8.27, 11.33, 14.77 [07:17:27] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:18:56] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [07:20:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:16:56] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:17:26] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [08:17:26] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:18:26] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [08:18:27] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [08:18:27] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:18:56] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [08:20:26] Load avg. on nightshade is WARNING: WARNING - load average: 27.50, 16.92, 12.58 [08:20:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:30:27] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 31.48, 21.82, 16.94 [08:31:26] Load avg. on nightshade is WARNING: WARNING - load average: 28.63, 22.95, 17.67 [08:51:27] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 25.89, 22.98, 20.18 [09:04:27] Load avg. on nightshade is WARNING: WARNING - load average: 11.02, 17.73, 19.66 [09:14:27] Load avg. on nightshade is OK: OK - load average: 7.51, 10.10, 14.62 [09:16:56] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:17:27] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [09:17:27] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:18:26] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [09:18:26] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [09:18:27] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:18:56] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [09:21:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:41:04] Dr. Trigon * Re: [Toolserver-l] Job stuck in error state [10:17:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:17:26] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [10:17:27] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:18:26] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [10:18:26] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:18:56] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [10:19:26] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [10:21:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:34:29] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1920.000000 [11:41:23] 3(commented) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (DaB.) [11:42:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41498 MB (4% inode=99%): [12:02:28] MySQL on z-dat-s4-a is OK: Uptime: 508689 Threads: 3 Questions: 13712654 Slow queries: 45781 Opens: 955 Flush tables: 1 Open tables: 285 Queries per second avg: 26.956 [12:02:29] MySQL slave on z-dat-s4-a is OK: Uptime: 508690 Threads: 3 Questions: 13712670 Slow queries: 45781 Opens: 955 Flush tables: 1 Open tables: 285 Queries per second avg: 26.956 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [12:03:00] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1.000000 [12:03:25] 3(resolved) [TS-1277] Re-Setup z-dat-s4-a <10https://jira.toolserver.org/browse/TS-1277> (DaB.) [12:04:29] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [12:05:24] 3(created) [TS-1289] Give user sgeadmin acces to sql-mapnik; Toolserver: Databases, SGE; Task <10https://jira.toolserver.org/browse/TS-1289> (merl) [12:17:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:17:29] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [12:18:48] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:19:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:21:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:49:51] [[Special:Log/newusers]] create 10 * Aravind V R * (New user account) [13:04:29] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6863.000000 [13:05:04] José Emilio Mori Recio * Re: [Toolserver-l] What happened to pagecounts stats? [13:05:29] :) [13:17:28] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [13:18:09] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:18:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:19:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:22:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:59:23] 3(commented) [OSM-4] Fan Error in ptolemy <10https://jira.toolserver.org/browse/OSM-4> (Marlen Caemmerer) [14:04:38] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9212.000000 [14:17:26] 3(commented) [TS-1285] Retrieve data from expired user account <10https://jira.toolserver.org/browse/TS-1285> (Marlen Caemmerer) [14:17:38] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [14:18:17] apmon: are you there? [14:18:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:19:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:19:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:20:02] Happy Valentines Day! :) [14:23:00] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:23:50] nosy: ping [14:24:00] apmon: pong [14:24:19] apmon: i want to restart postgress without ld_preload [14:24:26] apmon: thoughts? [14:24:46] adjust the shared_buffers before? [14:24:58] Is a definite possibility of where the segfaults come from [14:25:13] although it does appear that the issues started later than that [14:25:32] my first nagios message about this was from jan 10 [14:26:08] Ah, OK. In that case that does sound about the time we switched the ld_preload [14:26:38] Is it possible to change two more postgres parameters? [14:27:00] apmon: ill just try that now [14:27:01] fsync seems to be turned off, which could lead to corruption if the server crashes [14:27:27] and it might be sensible to turn autovacuum on, which also appears to be disabled [14:27:31] yes its off [14:27:44] nosy: Sounds good, to try run it without ld_preload [14:28:19] enabled autovacuum and fsync. shared_buffers back to 7GB? [14:28:31] I think you can leave them at 512Mb [14:28:45] it did not seem to make any difference between 512 and 7Gb [14:28:47] ok [14:28:55] performancewise? [14:29:28] At least from the munin graphs (which do have a lot of variance) I could not identify any obvious improvements [14:29:38] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1820.000000 [14:34:13] apmon: the restart is done [14:34:19] I will have to regularly restart tirex in that case to free the postgres memory, but that should still be better than regular crashes [14:34:22] nosy: thanks [14:36:06] nosy: In order to try and figure out why tirex-master takes so much kernel CPU time, would it be possible to profile it? [14:36:59] apmon: isn't it writte in perl? You can just run it with nytprof, know how? [14:37:08] apmon: its probably more a performance question [14:37:56] if you look at prstat output it speeds nearly all of its time in the kernel and not in userspace [14:38:09] s/speeds/spends [14:39:05] avar: how do you run it with nytprof? [14:40:13] not sure if we have this software on our very special os anyway [14:40:53] lockstat appears to be on ptolemy, which might give some hints where it spends its time [14:41:01] avar: do you mean http://search.cpan.org/~timb/Devel-NYTProf-4.06/lib/Devel/NYTProf.pm ? [14:42:51] apmon: ok. here are a lot of parameters [14:42:57] do you want the overview? [14:43:12] of lockstat? [14:44:24] apmon: yes [14:45:20] I don't know for sure, but a bit of googling indicates that "lockstat -I -i 977 -s10 -h sleep 5" might give me what I am looking for [14:46:47] and tirex-master instead of sleep 5 [14:47:02] apmon: i mailed you the usage info it gives [14:48:11] apmon: should i simply kill tirex-master and start it in the profiler? [14:48:27] nosy: thanks. To my gmail address? [14:48:35] apmon: yes [14:49:41] to start it with the nytprof? [14:49:57] One problem is that it appears to get worse over time [14:50:07] possibly connected with the memory leak? [14:51:45] nosy: To restart tirex-master, you need to first use the ~osm/bin/stop-tirex-master script, as a straight killing will cause shared memory to lockup [14:52:08] apmon: have you looked at http://munin.toolserver.org/OSM/ptolemy/tirex_status_queued_requests.html [14:52:09] ? [14:52:16] is it informative to you? [14:52:32] i dont know what queued in dirty exactly means [14:52:34] I haven't actually gotten your email yet [14:52:59] then i probably anoy the irc server... [14:53:08] nosy: pastbin? [14:53:27] queued in dirty all all the tiles that exist, but are outdated. [14:53:42] and are tried to be rerendered [14:54:09] The queue is growing so much as I disabled rendering to run the vaccuum. [14:54:13] mail came back [14:54:42] thats odd, any reason for that? [14:54:49] forgot the r in the mail address [14:54:57] your name is krueger [14:55:00] i think :) [14:55:16] :-) [14:55:52] I got one now. Those are the options of lockstat? [14:56:15] apmon: yes that should be [14:56:21] Did you also send me one with the "lockstat -I -i 977 -s10 -h sleep 5"? [14:56:39] nope hang on [14:56:46] will use pastebin [14:56:59] OK [14:58:47] apmon: http://pastebin.com/inA4GFBy [14:58:55] apmon: i need to go in 5 min [14:58:56] sorry [14:59:04] OK, thanks for the help [14:59:09] when will you be online again? [14:59:31] I'll probably be online most of the day today. [14:59:42] Otherwise tomorrow at a similar time? [14:59:57] better i try to come in this evenign [15:00:12] I'll see if I can make anything out of the profiling data, or try and if I can run tirex with the perl profiler [15:00:19] if evering goes well (not yet sure) i will travel to the data center tomorrow [15:00:38] the profiling data is afais for sleep -5 [15:03:39] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [15:04:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11730.000000 [15:17:38] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [15:18:47] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:19:58] /sql on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 13219 MB (1% inode=99%): [15:19:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:19:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:23:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:29:47] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2126.000000 [15:41:56] DabPunkt? [15:42:44] any toolserver admin here? [15:43:00] I've accidentally removed the permissions for a folder. [15:44:15] /home/mmovchin/public_html/paste/admin/ [16:03:39] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [16:04:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12301.000000 [16:17:39] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [16:18:58] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:19:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:19:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:23:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:29:47] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3017.000000 [16:34:27] 3(commented) [TS-1285] Retrieve data from expired user account <10https://jira.toolserver.org/browse/TS-1285> (Soxred93) [17:03:50] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [17:04:58] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12241.000000 [17:17:49] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [17:19:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:20:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:20:08] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:24:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:29:48] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2916.000000 [17:59:48] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3623.000000 [18:03:48] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [18:06:08] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13116.000000 [18:17:49] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [18:20:08] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:20:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:20:17] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:23:12] Any admin here? [18:24:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:45:58] Load avg. on nightshade is WARNING: WARNING - load average: 20.02, 15.32, 12.29 [18:48:57] Load avg. on nightshade is OK: OK - load average: 13.05, 14.62, 12.59 [18:52:53] Any admin here? [18:53:13] * Danny_B|backup points to nosy  [18:57:17] nosy :) [18:57:47] (19:57:38) nosy : I'm not here right now :( [18:59:48] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5640.000000 [19:03:49] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [19:04:23] mmovchin: you could also just ask your question. someone else might be able to help you [19:06:18] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14634.000000 [19:17:49] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [19:20:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:20:18] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:20:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:24:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:25] [[Special:Log/newusers]] create 10 * Fcarcena01 * (New user account) [19:51:50] Is it possible to pretect user tables or databases from other toolserver users? [19:55:25] mmovchin: I don't think that's a verb. [20:00:00] @replag [20:00:01] Joan: s2-pri: 18s [-]; s2/s5-pri-c: 4h 37m 32s [+0.64 s/s]; s3-rr: 4h 58m 45s [+1.00 s/s]; s3-user: 4h 58m 45s [+1.00 s/s]; s5-user: 2h 16m 18s [+0.68 s/s] [20:00:49] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8208.000000 [20:02:58] Load avg. on nightshade is WARNING: WARNING - load average: 16.28, 16.25, 14.29 [20:03:47] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [20:06:18] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16902.000000 [20:06:57] Load avg. on nightshade is OK: OK - load average: 11.77, 14.42, 14.02 [20:09:48] Joan I don't understand what you mean? [20:09:57] Pretect? [20:10:01] Is that a word? [20:13:26] *protect [20:13:48] What do you mean "protect"? [20:14:03] Do you understand user tables? [20:14:10] And their associated permissions? [20:14:19] If you have u_mmovchin, you have read/write access and nobody else does. [20:14:31] If you have u_mmovchin_p, you have read/write access and everybody else has read access. [20:14:37] What do you want to protect? [20:16:01] u_mmovchin_p f.e. [20:16:14] We just covered that case. [20:16:32] Also, don't use "f.e.", people will think you're stupid. Use "e.g." [20:16:52] ok :) [20:16:56] thanks [20:17:20] The opposite is actually the difficult case. That is, getting other TS users access to your DB. [20:17:37] That requires a multi-maintainer account, I think. [20:17:40] ok thank you [20:17:44] No problem. [20:17:58] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [20:20:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:20:38] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:21:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:24:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:37:49] Nyeh? [20:37:54] madman@nightshade:~$ svn co https://svn.toolserver.org/svnroot/madman madmanbot [20:37:55] svn: Could not open the requested SVN filesystem [20:45:26] mm [20:45:30] let me look [20:46:04] AMadman: I see no svn-reposity of you [20:46:52] I looked at https://wiki.toolserver.org/view/Subversion and didn't see any creation process. I saw one to add repositories to Fisheye. [20:47:00] Same process? [20:48:20] Most of the article "The toolserver provides Subversion hosting," "by default, only you have access to commit..." seemed to imply everyone had one already. I may have misinterpreted. :x [20:48:46] no, you don't misinterpret it. It is just wrong (or very outdated) [20:49:11] AMadman: request a reposity at jira in the TS-queue [20:49:13] Oh, okay. [20:49:15] Will do. [20:49:21] I can do it now, if you do [20:49:59] Load avg. on nightshade is WARNING: WARNING - load average: 16.85, 15.17, 13.42 [20:50:19] https://jira.toolserver.org/browse/TS-1291 [20:50:21] :) [20:50:22] 3(created) [TS-1291] New SVN repository: madman; Toolserver: Subversion; Minor Task <10https://jira.toolserver.org/browse/TS-1291> (madman) [20:53:23] 3(resolved) [TS-1291] New SVN repository: madman <10https://jira.toolserver.org/browse/TS-1291> (DaB.) [20:53:44] Thanks! :D [20:53:46] [[Subversion]] 10https://wiki.toolserver.org/w/index.php?diff=6665&oldid=5534&rcid=8798 * Dab * (+66) () [20:53:58] Load avg. on nightshade is OK: OK - load average: 10.67, 14.15, 13.55 [20:54:01] np [21:00:58] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10269.000000 [21:01:58] Load avg. on nightshade is WARNING: WARNING - load average: 22.99, 18.30, 15.31 [21:03:49] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [21:04:39] zzz =_= [21:05:08] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [21:05:20] 3(created) [MNT-1188] Switch master for s7; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1188> (DaB.) [21:06:27] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19070.000000 [21:09:08] MySQL slave on z-dat-s7-a is OK: Uptime: 470189 Threads: 5 Questions: 43492483 Slow queries: 18085 Opens: 248466 Flush tables: 1 Open tables: 3381 Queries per second avg: 92.500 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [21:09:58] Load avg. on nightshade is OK: OK - load average: 11.00, 14.41, 14.68 [21:10:23] 3(updated) [MNT-1188] Switch master for s7 <10https://jira.toolserver.org/browse/MNT-1188> (DaB.) [21:12:20] @replag [21:12:20] DaBPunkt: s1-sec: 1m 12s [+0.00 s/s]; s2/s5-pri-c: 5h 20m 38s [+0.60 s/s]; s3-rr: 6h 11m 5s [+1.00 s/s]; s3-user: 6h 11m 5s [+1.00 s/s]; s5-user: 2h 55m 17s [+0.54 s/s] [21:17:43] Oh dear. Should I have done a separate issue to add the madman repository to Fisheye? :x [21:17:58] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [21:20:07] AMadman: just re-open the request [21:20:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:20:38] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:21:23] 3(reopened) [TS-1291] New SVN repository: madman <10https://jira.toolserver.org/browse/TS-1291> (madman) [21:21:28] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:25:20] 3(created) [MNT-1189] Replication of s3 stopped arround 6h ago; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1189> (DaB.) [21:25:21] 3(updated) [MNT-1189] Replication of s3 stopped arround 6h ago <10https://jira.toolserver.org/browse/MNT-1189> (DaB.) [21:25:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:41:03] Frédéric Schütz * Re: [Toolserver-l] What happened to pagecounts stats? [21:44:35] @replag [21:44:36] DaBPunkt: s2-pri: 9m 46s [+0.09 s/s]; s2/s5-pri-c: 5h 39m 18s [+0.58 s/s]; s3-rr: 3h 18m 40s [-5.35 s/s]; s3-user: 3h 18m 40s [-5.35 s/s]; s4-rr: 5h 39m 18s [+0.29 s/s]; s5-user: 3h 12m 15s [+0.53 s/s]; s6-rr: 10m 49s [+0.01 s/s]; s6-user: 10m 49s [+0.01 s/s] [21:51:48] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [21:55:16] @replag [21:55:16] DaBPunkt: s1-sec-c: 7m 50s [+0.01 s/s]; s2-pri: 11m 43s [+0.18 s/s]; s2/s5-pri-c: 5h 32m 44s [-0.61 s/s]; s3-rr: 2h 25s [-7.33 s/s]; s3-user: 2h 25s [-7.33 s/s]; s4-rr: 7m 50s [-31.04 s/s]; s4-user: 7m 50s [-]; s5-user: 2h 58m 30s [-1.29 s/s] [21:55:17] DaBPunkt: s6-rr: 21m 30s [+1.00 s/s]; s6-user: 21m 30s [+1.00 s/s] [21:58:18] Load avg. on nightshade is WARNING: WARNING - load average: 11.77, 14.46, 15.21 [22:00:34] @replag [22:00:35] DaBPunkt: s1-pri: 51s [+0.00 s/s]; s1-sec: 51s [-0.01 s/s]; s1-sec-c: 13m 8s [+1.00 s/s]; s2-pri: 13m 32s [+0.34 s/s]; s2/s5-pri-c: 5h 34m 38s [+0.36 s/s]; s3-rr: 53m 13s [-12.66 s/s]; s3-user: 53m 13s [-12.66 s/s]; s4-rr: 5h 34m 38s [+61.57 s/s] [22:00:36] DaBPunkt: s4-user: 13m 8s [+1.00 s/s]; s5-user: 2h 38m 9s [-3.83 s/s]; s6-rr: 26m 48s [+1.00 s/s]; s6-user: 26m 48s [+1.00 s/s] [22:00:48] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2936 [22:01:08] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9258.000000 [22:01:50] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [22:02:19] @replag [22:02:19] DaBPunkt: s1-pri: 46s [-0.05 s/s]; s1-sec: 46s [-0.05 s/s]; s1-sec-c: 14m 53s [+1.01 s/s]; s2-pri: 13m 48s [+0.15 s/s]; s2/s5-pri-c: 5h 36m 14s [+0.92 s/s]; s3-rr: 34m 25s [-10.80 s/s]; s3-user: 34m 25s [-10.80 s/s]; s4-rr: 5h 36m 14s [+0.92 s/s] [22:02:20] DaBPunkt: s4-user: 14m 53s [+1.00 s/s]; s5-rr: 46s [+0.00 s/s]; s5-user: 2h 30m 26s [-4.43 s/s]; s6-rr: 28m 33s [+1.00 s/s]; s6-user: 28m 33s [+1.00 s/s]; s7-rr: 47s [+0.00 s/s]; s7-user: 47s [+0.00 s/s] [22:05:22] 3(created) [MNT-1190] Changed master of s2; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1190> (DaB.) [22:05:24] 3(updated) [MNT-1190] Changed master of s2 <10https://jira.toolserver.org/browse/MNT-1190> (DaB.) [22:06:49] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20428.000000 [22:07:20] 3(created) [MNT-1191] Change master of s4; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1191> (DaB.) [22:10:08] Load avg. on nightshade is OK: OK - load average: 10.58, 13.27, 14.81 [22:11:08] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3057.000000 [22:14:08] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1365.000000 [22:14:57] MySQL slave on z-dat-s3-a is OK: Uptime: 349054 Threads: 25 Questions: 278529323 Slow queries: 17580 Opens: 1033436 Flush tables: 1 Open tables: 16377 Queries per second avg: 797.954 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 525 [22:15:26] @replag [22:15:26] DaBPunkt: s1-sec-c: 28m 0s [+1.00 s/s]; s2-pri: 13s [-1.04 s/s]; s2/s5-pri-c: 5h 47m 25s [+0.85 s/s]; s3-rr: 7m 14s [-2.07 s/s]; s3-user: 7m 14s [-2.07 s/s]; s4-rr: 5h 47m 25s [+0.85 s/s]; s4-user: 28m 0s [+1.00 s/s]; s5-user: 9m 52s [-10.72 s/s] [22:15:27] DaBPunkt: s6-rr: 5m 25s [-1.76 s/s]; s6-user: 5m 25s [-1.76 s/s] [22:16:03] @replag [22:16:04] DaBPunkt: s1-sec-c: 28m 37s [+0.99 s/s]; s2/s5-pri-c: 5h 47m 41s [+0.43 s/s]; s3-rr: 7m 11s [-0.08 s/s]; s3-user: 7m 11s [-0.08 s/s]; s4-rr: 28m 37s [-509.24 s/s]; s4-user: 28m 37s [+0.99 s/s]; s5-user: 3m 38s [-9.96 s/s]; s6-rr: 2m 57s [-3.94 s/s] [22:16:05] DaBPunkt: s6-user: 2m 58s [-3.90 s/s] [22:16:38] great, it is increasing instead of decreasing :( [22:18:08] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [22:19:49] s4 replag on z-dat-s4-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1932.000000 [22:20:03] @replag [22:20:05] DaBPunkt: s1-pri: 45s [-0.00 s/s]; s1-sec: 45s [-0.00 s/s]; s1-sec-c: 32m 37s [+1.00 s/s]; s2-pri: 47s [+0.12 s/s]; s2/s5-pri-c: 5h 49m 34s [+0.47 s/s]; s3-rr: 8m 27s [+0.32 s/s]; s3-user: 8m 27s [+0.32 s/s]; s4-rr: 32m 39s [+1.00 s/s] [22:20:06] DaBPunkt: s4-user: 32m 39s [+1.00 s/s]; s5-rr: 49s [+0.00 s/s]; s6-rr: 47s [-0.54 s/s]; s6-user: 47s [-0.54 s/s]; s7-rr: 48s [+0.00 s/s]; s7-user: 48s [+0.00 s/s] [22:20:27] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1979.000000 [22:20:48] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:20:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:21:20] 3(commented) [MNT-1191] Change master of s4 <10https://jira.toolserver.org/browse/MNT-1191> (DaB.) [22:21:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:25:08] Load avg. on nightshade is WARNING: WARNING - load average: 16.24, 15.76, 15.19 [22:25:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:28:28] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1765.000000 [22:33:58] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2173 [22:34:24] 3(commented) [MNT-1191] Change master of s4 <10https://jira.toolserver.org/browse/MNT-1191> (DaB.) [22:34:48] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 1499.000000 [22:34:57] MySQL slave on z-dat-s4-a is OK: Uptime: 546640 Threads: 10 Questions: 15026166 Slow queries: 46274 Opens: 1464 Flush tables: 1 Open tables: 340 Queries per second avg: 27.488 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1333 [22:59:50] @replag [22:59:51] DaBPunkt: s2/s5-pri-c: 2h 25m 36s [-5.13 s/s]; s3-rr: 19m 50s [+0.29 s/s]; s3-user: 19m 50s [+0.29 s/s] [23:06:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3951.000000 [23:10:11] @replag [23:10:12] DaBPunkt: s2/s5-pri-c: 1h 3m 1s [-7.98 s/s]; s3-rr: 19m 26s [-0.04 s/s]; s3-user: 19m 26s [-0.04 s/s] [23:14:18] Load avg. on nightshade is OK: OK - load average: 11.29, 12.71, 14.93 [23:18:28] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [23:20:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:20:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:21:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:25:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:29:17] Load avg. on nightshade is WARNING: WARNING - load average: 20.92, 16.61, 15.11 [23:39:38] nacht ts [23:42:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41324 MB (4% inode=99%): [23:57:28] Load avg. on nightshade is OK: OK - load average: 11.42, 13.30, 14.87