[00:05:38] Cluster on turnera.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:05:38] Cluster on damiana.esi is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:06:08] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [00:06:08] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [00:13:00] Sun Grid Engine execd on wolfsbane is CRITICAL: short@wolfsbane in error state: QERROR as result of job 1600963s failure: all.q@wolfsbane in error state: QERROR as result of job 1600963s failure [00:13:00] Sun Grid Engine execd on willow is CRITICAL: all.q@willow in error state: QERROR as result of job 1600970s failure: longrun@willow in error state: QERROR as result of job 1600970s failure [00:13:07] Sun Grid Engine execd on ortelius is CRITICAL: short@ortelius in error state: QERROR as result of job 1600965s failure: all.q@ortelius in error state: QERROR as result of job 1600965s failure [00:23:58] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [00:23:58] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [00:23:58] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:24:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42828 MB (4% inode=99%): [00:24:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:24:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:30:08] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [00:33:58] Sun Grid Engine execd on wolfsbane is OK: short@wolfsbane OK: all.q@wolfsbane OK [00:33:58] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:34:07] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [01:06:40] ok, automic reboot works [01:07:50] Free Memory on damiana.esi is OK: OK - 85.9% (3596744 kB) free. [01:07:51] MySQL on ha-sql.esi is CRITICAL: Cant connect to MySQL server on ha-sql.esi (146) [01:08:21] Cluster on turnera.esi is CRITICAL: damiana:nge0-turnera:nge0 waiting, mysql OFFLINE, check nfs-hasp, nfs-home Online, [01:09:01] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1974.000000 [01:10:21] Cluster on damiana.esi is CRITICAL: mysql OFFLINE, check nfs-hasp, [01:10:51] MySQL on ha-sql.esi is OK: Uptime: 6 Threads: 3 Questions: 40 Slow queries: 0 Opens: 18 Flush tables: 1 Open tables: 11 Queries per second avg: 6.666 [01:11:22] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [01:11:22] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [01:24:15] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [01:24:15] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [01:24:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:25:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42658 MB (4% inode=99%): [01:25:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:25:15] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:30:22] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [01:37:01] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3654.000000 [01:45:22] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.381836/1.75, alarm hl:np_load_avg=0.415039/2.00, alarm hl:mem_free=251.000000M/300M [01:50:14] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3578.000000 [01:50:23] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:55:54] @replag [01:55:54] DaBPunkt: s2-pri: 12s [-]; s2/s5-pri-c: 41m 21s [-]; s3-rr: 31s [-]; s3-user: 31s [-]; s4-rr: 41m 21s [-]; s4-user: error [01:56:02] @replag [01:56:03] DaBPunkt: s2/s5-pri-c: 41m 20s [-0.12 s/s]; s3-rr: 39s [+0.98 s/s]; s3-user: 39s [+0.98 s/s]; s4-rr: 41m 20s [-0.12 s/s]; s4-user: error [01:59:15] @replag [01:59:16] DaBPunkt: s1-sec-c: 14s [-]; s2/s5-pri-c: 34m 51s [-2.01 s/s]; s3-rr: 12s [-0.14 s/s]; s3-user: 12s [-0.14 s/s]; s4-rr: 34m 51s [-2.01 s/s]; s4-user: error [01:59:40] nacht ts [02:03:15] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1687.000000 [02:03:21] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.582031/1.75, alarm hl:np_load_avg=0.687012/2.00, alarm hl:mem_free=267.000000M/300M [02:25:15] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42521 MB (4% inode=99%): [02:25:15] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [02:25:15] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [02:25:15] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:25:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:25:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:30:32] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [03:25:14] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [03:25:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42339 MB (4% inode=99%): [03:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [03:25:16] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:25:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:25:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:30:33] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [03:40:13] Free Memory on turnera.esi is WARNING: WARNING - 6.3% (264760 kB) free! [03:54:13] Free Memory on turnera.esi is OK: OK - 10.2% (428816 kB) free. [04:13:32] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.419922/1.75, alarm hl:np_load_avg=0.386230/2.00, alarm hl:mem_free=276.000000M/300M [04:14:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:15:24] Free Memory on turnera.esi is WARNING: WARNING - 9.6% (402440 kB) free! [04:18:32] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.369141/1.75, alarm hl:np_load_avg=0.368164/2.00, alarm hl:mem_free=218.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.369141/1.50, alarm hl:np_load_long=0.365723/1.75, alarm hl:mem_free=218.000000M/250M [04:23:24] Free Memory on turnera.esi is CRITICAL: CRITICAL - 5.1% (212520 kB) free! [04:25:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42181 MB (4% inode=99%): [04:25:24] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [04:25:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:25:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:25:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:26:12] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [04:30:43] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [04:31:24] Free Memory on turnera.esi is WARNING: WARNING - 6.9% (286896 kB) free! [04:43:50] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 waiting, check nfs-hasp, [04:45:51] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [04:47:51] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [05:04:01] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.368652/1.75, alarm hl:np_load_avg=0.408691/2.00, alarm hl:mem_free=219.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.368652/1.50, alarm hl:np_load_long=0.376953/1.75, alarm hl:mem_free=219.000000M/250M [05:12:38] [[Suggestions]] 10https://wiki.toolserver.org/w/index.php?diff=6655&oldid=6342&rcid=8781 * Pathoschild * (+940) (/* Framework for scripts */ re: [[m:User:Pathoschild/ToyEngine|ToyEngine]] PHP framework in progress) [05:12:51] Free Memory on damiana.esi is CRITICAL: CRITICAL - 4.6% (193684 kB) free! [05:14:01] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:26:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/gdevsync:default svc:/milestone/multi-user:default svc:/system/cluster/cl-svc-enable:default svc:/milestone/multi-user-server:default svc:/system/dumpadm:default svc:/system/fmd:default svc:/system/basicreg:default svc:/system/cluster/ql_upgrade:default svc:/system/cluster/mountgfs:default svc:/system/cluster/sc_zones:default svc:/system/zones:default svc:/sys [05:26:00] SMF on damiana.esi is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:26:00] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [05:26:40] Free Memory on damiana.esi is OK: OK - 43.6% (1824440 kB) free. [05:26:52] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:26:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42136 MB (4% inode=99%): [05:26:53] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [05:28:00] / on willow is WARNING: DISK WARNING - free space: / 23061 MB (20% inode=99%): [05:28:09] Cluster on damiana.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, nfs-home Online, [05:28:19] Cluster on turnera.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, nfs-home Online, [05:29:20] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [05:29:20] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [05:30:10] Cluster on damiana.esi is CRITICAL: ldap OFFLINE, check nfs-hasp, ds--global-misc-ldap OFFLINE, [05:30:20] Cluster on turnera.esi is CRITICAL: ldap OFFLINE, check nfs-hasp, ds--global-misc-ldap OFFLINE, [05:31:30] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [05:43:55] nighty [05:54:40] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1820 [06:13:00] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.619141/1.75, alarm hl:np_load_avg=0.648926/2.00, alarm hl:mem_free=193.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.619141/1.50, alarm hl:np_load_long=0.538086/1.75, alarm hl:mem_free=193.000000M/250M [06:14:40] MySQL slave on z-dat-s6-a is OK: Uptime: 304099 Threads: 7 Questions: 27299456 Slow queries: 19680 Opens: 197813 Flush tables: 1 Open tables: 1841 Queries per second avg: 89.771 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1769 [06:16:00] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:26:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:26:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:26:59] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [06:27:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:27:50] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42625 MB (4% inode=99%): [06:27:50] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [06:28:00] / on willow is WARNING: DISK WARNING - free space: / 23017 MB (20% inode=99%): [06:30:21] 3(created) [TS-1286] Please reactivate my account; Toolserver; Minor Task <10https://jira.toolserver.org/browse/TS-1286> (Daniel Paul Money) [06:31:30] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [07:12:59] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.511231/1.75, alarm hl:np_load_avg=0.491699/2.00, alarm hl:mem_free=196.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.511231/1.50, alarm hl:np_load_long=0.530762/1.75, alarm hl:mem_free=196.000000M/250M [07:15:00] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:26:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:26:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:27:00] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [07:27:50] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [07:28:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:28:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42745 MB (4% inode=99%): [07:28:00] / on willow is WARNING: DISK WARNING - free space: / 22978 MB (20% inode=99%): [07:31:30] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [08:20:20] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.082519/1.75, alarm hl:np_load_avg=0.114746/2.00, alarm hl:mem_free=272.000000M/300M [08:24:30] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 16490 MB (1% inode=99%): [08:26:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:27:20] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [08:27:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:27:20] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:27:59] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [08:28:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42636 MB (4% inode=99%): [08:28:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:28:20] / on willow is WARNING: DISK WARNING - free space: / 22946 MB (20% inode=99%): [08:31:29] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [08:33:20] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.143066/1.75, alarm hl:np_load_avg=0.099121/2.00, alarm hl:mem_free=230.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.143066/1.50, alarm hl:np_load_long=0.138184/1.75, alarm hl:mem_free=230.000000M/250M [09:18:20] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.092773/1.75, alarm hl:np_load_avg=0.085449/2.00, alarm hl:mem_free=261.000000M/300M [09:19:20] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:26:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:27:19] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [09:27:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:28:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42586 MB (4% inode=99%): [09:28:00] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [09:28:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:28:20] / on willow is WARNING: DISK WARNING - free space: / 22932 MB (20% inode=99%): [09:31:30] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [10:18:05] jem-, by the way, you're not a TS admin ? [10:26:30] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:27:21] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [10:27:21] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:27:59] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [10:28:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42439 MB (4% inode=99%): [10:28:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:28:20] / on willow is WARNING: DISK WARNING - free space: / 22894 MB (20% inode=99%): [10:31:30] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [11:09:39] Free Memory on damiana.esi is WARNING: WARNING - 9.5% (397300 kB) free! [11:12:40] Free Memory on damiana.esi is CRITICAL: CRITICAL - 4.0% (166964 kB) free! [11:17:58] MySQL on ha-sql.esi is CRITICAL: Cant connect to MySQL server on ha-sql.esi (146) [11:17:58] Cluster on turnera.esi is CRITICAL: damiana FAILED, damiana:nge1-turnera:nge1 faulted, damiana:nge0-turnera:nge0 faulted, vote damiana Offline, mysql OFFLINE, check nfs-hasp, nfs-home Online, check sge-qmaster, [11:18:08] Free Memory on damiana.esi is OK: OK - 77.1% (3225724 kB) free. [11:20:49] Cluster on damiana.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, mysql OFFLINE, check nfs-hasp, [11:20:59] MySQL on ha-sql.esi is OK: Uptime: 8 Threads: 2 Questions: 20 Slow queries: 0 Opens: 16 Flush tables: 1 Open tables: 9 Queries per second avg: 2.500 [11:22:00] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [11:27:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:27:50] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [11:28:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42352 MB (4% inode=99%): [11:28:09] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [11:28:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:28:28] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:28:28] / on willow is WARNING: DISK WARNING - free space: / 22804 MB (20% inode=99%): [11:32:18] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [11:36:00] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [11:41:59] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1928.000000 [11:48:01] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1980.000000 [12:06:19] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.127441/1.75, alarm hl:np_load_avg=0.153320/2.00, alarm hl:mem_free=242.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.127441/1.50, alarm hl:np_load_long=0.166504/1.75, alarm hl:mem_free=242.000000M/250M [12:10:00] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3608.000000 [12:12:19] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:13:12] [[Special:Log/newusers]] create 10 * Reza luke * (New user account) [12:16:01] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3660.000000 [12:19:30] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.174316/1.75, alarm hl:np_load_avg=0.200195/2.00, alarm hl:mem_free=253.000000M/300M [12:27:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:28:00] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [12:28:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42203 MB (4% inode=99%): [12:28:09] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [12:28:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:28:29] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:28:29] / on willow is WARNING: DISK WARNING - free space: / 22769 MB (20% inode=99%): [12:32:19] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [12:47:03] Dr. Trigon * [Toolserver-l] Job stuck in error state [13:02:29] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.138184/1.75, alarm hl:np_load_avg=0.156250/2.00, alarm hl:mem_free=272.000000M/300M [13:10:09] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7214.000000 [13:11:35] Quentinv57: No, I'm a simple user from Spain :) [13:14:38] !replag [13:14:46] hmm [13:14:47] @replag [13:14:48] guillom: s2/s5-pri-c: 2h 0s [-]; s3-rr: 18s [-]; s3-user: 18s [-]; s4-rr: 2h 0s [-]; s4-user: error; s5-user: 2h 5m 2s [-] [13:14:51] thx [13:16:59] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7317.000000 [13:18:03] Joan, are you around, by any chance? [13:19:29] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [13:27:59] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [13:28:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:29:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42048 MB (4% inode=99%): [13:29:09] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [13:29:29] / on willow is WARNING: DISK WARNING - free space: / 22733 MB (20% inode=99%): [13:29:29] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:29:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:32:20] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [13:41:38] @replag all [13:41:39] nosy1: s1-pri: 0s [-]; s1-sec: 0s [-]; s1-sec-c: 1s [-]; s2-pri: 2s [-]; s2/s5-pri-c: 2h 10s [+0.01 s/s]; s3-rr: 1m 10s [+0.03 s/s]; s3-user: 1m 10s [+0.03 s/s]; s4-rr: 2h 9s [+0.01 s/s] [13:41:40] nosy1: s4-user: error; s5-rr: 3s [-]; s5-user: 1h 54m 51s [-0.38 s/s]; s6-rr: 1s [-]; s6-user: 1s [-]; s7-rr: 4s [-]; s7-user: 4s [-] [13:45:06] @replag all [13:45:07] nosy1: s1-pri: 0s [-]; s1-sec: 0s [-]; s1-sec-c: 1s [-]; s2-pri: 15s [+0.06 s/s]; s2/s5-pri-c: 2h 57s [+0.23 s/s]; s3-rr: 3s [-0.32 s/s]; s3-user: 3s [-0.32 s/s]; s4-rr: 2h 57s [+0.23 s/s] [13:45:08] nosy1: s4-user: error; s5-rr: 1s [-0.01 s/s]; s5-user: 1h 54m 10s [-0.20 s/s]; s6-rr: 4s [+0.01 s/s]; s6-user: 4s [+0.01 s/s]; s7-rr: 1s [-0.01 s/s]; s7-user: 1s [-0.01 s/s] [13:45:09] Free Memory on turnera.esi is WARNING: WARNING - 9.7% (406472 kB) free! [13:47:39] / on willow is OK: DISK OK - free space: / 24106 MB (21% inode=99%): [13:48:38] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.267578/1.75, alarm hl:np_load_avg=0.221191/2.00, alarm hl:mem_free=253.000000M/300M [13:49:30] Hello all [13:50:47] hello DaBPunkt [13:52:09] Free Memory on turnera.esi is OK: OK - 10.1% (422320 kB) free. [13:54:15] Hi, DaBPunkt [13:54:35] Is "the problem" near from a solution? [13:56:42] jem-: no, we haven't found it until now. [13:58:11] Free Memory on turnera.esi is WARNING: WARNING - 8.8% (369744 kB) free! [14:03:34] hello DaBPunkt [14:03:46] I have coded something for you :P [14:04:37] for me? [14:04:43] yes I do [14:05:01] for databases that need to be created, deleted or marked as locked [14:05:41] sent link in PM [14:08:54] Ok, DaBPunkt, hopefully it can be soon... [14:09:33] jem-: we insert a ugly hack, that keeps the impact for the users to a minimum [14:10:10] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7055.000000 [14:13:38] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [14:15:32] @replag [14:15:32] DaBPunkt: s2/s5-pri-c: 2h 6m 21s [+0.18 s/s]; s3-rr: 21s [+0.01 s/s]; s3-user: 21s [+0.01 s/s]; s4-rr: 2h 6m 21s [+0.18 s/s]; s4-user: error; s5-user: 1h 53m 5s [-0.04 s/s] [14:15:43] @replag [14:15:43] DaBPunkt: s2/s5-pri-c: 2h 6m 23s [+0.19 s/s]; s4-rr: 2h 6m 23s [+0.19 s/s]; s4-user: error; s5-user: 1h 52m 53s [-1.14 s/s] [14:16:09] Ok, DaBPunkt, thanks and keep up the good work [14:16:57] we try :) [14:17:08] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7557.000000 [14:19:09] Free Memory on turnera.esi is WARNING: WARNING - 8.0% (333224 kB) free! [14:27:03] Platonides * Re: [Toolserver-l] Job stuck in error state [14:28:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:28:57] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [14:29:19] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41902 MB (4% inode=99%): [14:29:19] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@turnera-bge0.esi.toolserver.org (using password: YES) [14:29:50] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:29:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:32:10] Free Memory on turnera.esi is OK: OK - 10.4% (434224 kB) free. [14:32:30] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [14:34:59] could someone make URL rewrite at http://toolserver.org/ ? http://toolserver.org/~soxred93/pcount/… should become http://toolserver.org/~tparis/pcount/… [14:38:01] Cluster on damiana.esi is CRITICAL: turnera FAILED, damiana:nge1-turnera:nge1 faulted, damiana:nge0-turnera:nge0 faulted, vote turnera Offline, check nfs-hasp, nfs-home Online, [14:39:00] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [14:40:40] NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.008364 secs [14:41:01] Cluster on turnera.esi is WARNING: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, [14:52:31] Sun Grid Engine execd on willow is CRITICAL: all.q@willow in error state: QERROR as result of job 1602968s failure: longrun@willow in error state: QERROR as result of job 1602968s failure [14:53:41] NTP on turnera.esi is OK: NTP OK: Offset -0.015231 secs [14:54:01] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [14:54:01] Cluster on turnera.esi is OK: CLUSTER OK ! check nfs-hasp, [14:55:00] s2 replag on daphne is CRITICAL: QUERY CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [14:55:21] MySQL on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [14:55:31] MySQL slave on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [14:58:01] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 34.000000 [14:58:20] MySQL on daphne is OK: Uptime: 6624055 Threads: 23 Questions: 5434328500 Slow queries: 882758 Opens: 35509931 Flush tables: 3 Open tables: 16372 Queries per second avg: 820.393 [14:58:30] MySQL slave on daphne is OK: Uptime: 6624063 Threads: 24 Questions: 5434330694 Slow queries: 882759 Opens: 35509931 Flush tables: 3 Open tables: 16372 Queries per second avg: 820.392 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43 [15:04:02] / on ortelius is WARNING: DISK WARNING - free space: / 5384 MB (17% inode=93%): [15:05:01] / on ortelius is OK: DISK OK - free space: / 9435 MB (31% inode=93%): [15:09:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 3483 MB (11% inode=93%): [15:10:21] s5 replag on daphne is CRITICAL: (Service Check Timed Out) [15:10:51] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2698 MB (9% inode=93%): [15:12:30] MySQL slave on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [15:12:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 5408 MB (18% inode=93%): [15:13:00] s2 replag on daphne is CRITICAL: QUERY CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [15:13:11] Sun Grid Engine execd on ortelius is CRITICAL: short@ortelius in error state: QERROR as result of job 1603019s failure: all.q@ortelius in error state: QERROR as result of job 1603019s failure [15:13:21] MySQL on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [15:17:11] s4 replag on daphne is CRITICAL: QUERY CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [15:28:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:29:00] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [15:29:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42197 MB (4% inode=99%): [15:29:31] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [15:30:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:30:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:33:01] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [15:49:31] MySQL on daphne is OK: Uptime: 6627119 Threads: 44 Questions: 5434755823 Slow queries: 883152 Opens: 35510103 Flush tables: 3 Open tables: 16375 Queries per second avg: 820.78 [15:49:31] MySQL slave on daphne is OK: Uptime: 6627123 Threads: 43 Questions: 5434756336 Slow queries: 883152 Opens: 35510103 Flush tables: 3 Open tables: 16375 Queries per second avg: 820.77 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [15:52:31] Sun Grid Engine execd on willow is CRITICAL: all.q@willow in error state: QERROR as result of job 1602968s failure: longrun@willow in error state: QERROR as result of job 1602968s failure [15:53:01] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 3.000000 [15:54:30] MySQL on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [15:54:31] MySQL slave on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [15:56:01] s2 replag on daphne is CRITICAL: QUERY CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [15:59:00] s2 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 0.000000 [16:02:30] MySQL on daphne is OK: Uptime: 6627898 Threads: 42 Questions: 5434837735 Slow queries: 883298 Opens: 35510144 Flush tables: 3 Open tables: 16378 Queries per second avg: 819.994 [16:02:31] MySQL slave on daphne is OK: Uptime: 6627903 Threads: 43 Questions: 5434838687 Slow queries: 883299 Opens: 35510144 Flush tables: 3 Open tables: 16378 Queries per second avg: 819.993 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [16:10:40] s5 replag on daphne is CRITICAL: (Service Check Timed Out) [16:13:11] Sun Grid Engine execd on ortelius is CRITICAL: short@ortelius in error state: QERROR as result of job 1603019s failure: all.q@ortelius in error state: QERROR as result of job 1603019s failure [16:17:11] s4 replag on daphne is CRITICAL: QUERY CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [16:28:01] s2 replag on daphne is CRITICAL: QUERY CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [16:28:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:29:00] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [16:29:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42537 MB (4% inode=99%): [16:29:31] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [16:30:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:30:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:31:31] MySQL on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [16:31:31] MySQL slave on daphne is CRITICAL: User nagios has exceeded the max_user_connections resource (current value: 15) [16:33:01] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [16:41:22] 3(resolved) [UTRS-42] Getting database error when attempting to enter appeals <10https://jira.toolserver.org/browse/UTRS-42> (Hersfold) [16:49:21] 3(work started) [UTRS-53] Emails invalid <10https://jira.toolserver.org/browse/UTRS-53> (Hersfold) [16:53:05] Sun Grid Engine execd on willow is CRITICAL: all.q@willow in error state: QERROR as result of job 1602968s failure: longrun@willow in error state: QERROR as result of job 1602968s failure [17:00:52] guillom: I'm here now. [17:02:30] Joan, it's ok; I was a bit scared to use the Global delivery system, but I launched it and it went prefectly. So, thanks for making it possible :) [17:02:39] :-) [17:02:44] I saw your delivery. [17:03:01] Anything I should do better next time? [17:08:46] No, it looked fine. [17:10:56] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11416.000000 [17:11:42] ok, thanks [17:14:05] Sun Grid Engine execd on ortelius is CRITICAL: short@ortelius in error state: QERROR as result of job 1603019s failure: all.q@ortelius in error state: QERROR as result of job 1603019s failure [17:17:20] 3(commented) [UTRS-53] Emails invalid <10https://jira.toolserver.org/browse/UTRS-53> (Hersfold) [17:17:56] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14079.000000 [17:20:18] "Unable to run job: error: no suitable queues." on launching a new crontab job [17:20:57] nosy, DaBPunkt, are you aware of this? [17:21:16] wait a moment [17:21:50] solved [17:22:06] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:22:06] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [17:22:31] Indeed :) Thanks, DaBPunkt [17:28:15] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 waiting, check nfs-hasp, [17:29:16] Cluster on damiana.esi is OK: CLUSTER OK ! check nfs-hasp, [17:29:55] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [17:29:56] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [17:30:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:30:17] DaBPunkt: regarding SGE, I've got a few jobs which I'm trying to delete, and they're stuck deleting (state "dt"). any ideas? [17:30:36] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42455 MB (4% inode=99%): [17:30:40] Merlissimo_: are you online? [17:30:53] ... and I've got another job in the same point as stwalkerster :) [17:30:55] I beleieve DrTrigon is having a similar problem, from the looks of the mailing list [17:30:56] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:30:56] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:32:45] DaBPunkt, hello, on which server should I connect if I want to read things from the database abwiki_p ? [17:33:00] abwiki_p.db.toolserver.org ? [17:33:25] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [17:34:03] Quentinv57: I believe it's abwiki-p.db.toolserver.org [17:34:21] (with a - not a _) [17:34:26] Quentinv57: abwiki-p.rrdb.toolserver.org if you want to read, abwiki-p.userdb.toolserver.org if you need a writable user-database there [17:35:15] stwalkerster: how is your login-name? [17:36:03] oh, apologies, it's on the "acc" multi-maintainer account :P my personal username is "stwalkerster" [17:37:21] DaBPunkt, are you sure ? it does not work [17:38:01] Quentinv57: yes, I am sure [17:41:46] stwalkerster: jem-: (drTrigon): fixed [17:42:06] tyvm :D [17:42:46] DaBPunkt: or nosy around? [17:43:11] Danny_B|webchat: whats up? [17:44:28] DaBPunkt, yes, thanks [17:46:17] I confirm also, thanks :) [18:11:06] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7501.000000 [18:12:03] is there any know issue on willow server? [18:12:14] my cron job didn't run [18:15:16] Free Memory on damiana.esi is WARNING: WARNING - 9.6% (401308 kB) free! [18:18:06] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11351.000000 [18:19:23] 3(commented) [OSM-3] Ptolemy Postgres crashed twice <10https://jira.toolserver.org/browse/OSM-3> (Kai Krueger) [18:28:16] Free Memory on damiana.esi is CRITICAL: CRITICAL - 4.1% (172732 kB) free! [18:29:55] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [18:29:56] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [18:30:15] Free Memory on damiana.esi is WARNING: WARNING - 6.1% (255216 kB) free! [18:30:17] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:31:06] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:31:16] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:31:26] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42291 MB (4% inode=99%): [18:33:36] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [18:36:24] [[Get Top RC Pages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6656&oldid=4910&rcid=8783 * 68.233.255.129 * (-648) (xYTLUF I must admit, the webmaster is a cool guy..!) [18:37:00] [[Get Top RC Pages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6657&oldid=6656&rcid=8784 * 207.245.176.1 * (+648) (Undo revision 6656 by [[Special:Contributions/68.233.255.129|68.233.255.129]] ([[User talk:68.233.255.129|talk]])) [18:38:27] [[Get Top RC Users]] ! 10https://wiki.toolserver.org/w/index.php?diff=6658&oldid=4912&rcid=8785 * 68.233.255.129 * (-741) (Fzrlys Hello! Read the pages not for the first day. Yes, the connection speed is not good. How can I subscribe? I would like to read you in the future!...) [18:39:43] [[Get Top RC Pages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6659&oldid=6657&rcid=8786 * 68.233.255.129 * (-650) (ixIGob Hooray! the one who wrote is a cool guy..!) [18:42:05] [[Get Top RC Users]] ! 10https://wiki.toolserver.org/w/index.php?diff=6660&oldid=6658&rcid=8787 * 68.233.255.129 * (-103) (xKrLCA I must admit, the webmaster is a cool guy..!) [18:42:27] [[Special:Log/block]] block 10 * Valhallasw * (blocked [[02User:68.233.255.12910]] with an expiry time of infinite (anonymous users only, account creation disabled): Inserting nonsense/gibberish into pages) [18:42:40] [[Get Top RC Pages]] M 10https://wiki.toolserver.org/w/index.php?diff=6661&oldid=6659&rcid=8789 * Valhallasw * (+650) (Reverted edits by [[Special:Contributions/68.233.255.129|68.233.255.129]] ([[User talk:68.233.255.129|talk]]) to last revision by [[User:207.245.176.1|207.245.176.1]]) [18:42:41] [[Get Top RC Users]] M 10https://wiki.toolserver.org/w/index.php?diff=6662&oldid=6660&rcid=8790 * Valhallasw * (+844) (Reverted edits by [[Special:Contributions/68.233.255.129|68.233.255.129]] ([[User talk:68.233.255.129|talk]]) to last revision by [[User:Krinkle|Krinkle]]) [19:00:19] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.262695/1.75, alarm hl:np_load_avg=0.276367/2.00, alarm hl:mem_free=269.000000M/300M [19:06:19] Free Memory on damiana.esi is WARNING: WARNING - 10.0% (419396 kB) free! [19:08:19] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [19:11:08] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7507.000000 [19:14:19] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.284180/1.75, alarm hl:np_load_avg=0.303223/2.00, alarm hl:mem_free=297.000000M/300M [19:16:18] Free Memory on damiana.esi is CRITICAL: CRITICAL - 5.4% (224308 kB) free! [19:16:20] 3(commented) [UTRS-53] Emails invalid <10https://jira.toolserver.org/browse/UTRS-53> (Hersfold) [19:17:19] Free Memory on damiana.esi is WARNING: WARNING - 6.1% (256448 kB) free! [19:18:09] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11888.000000 [19:18:20] 3(resolved) [UTRS-53] Emails invalid <10https://jira.toolserver.org/browse/UTRS-53> (Hersfold) [19:22:46] DaBPunkt / nosy, wouldn't it make sense to create a project for soxreds tools instead of /another/ user have them in their local namespace..? [19:25:34] (might also be a good time to remind people to create projects for popular tools...) [19:27:19] qsub is not able to read the script available in my home directory [19:27:33] on willow [19:27:40] should i try from another host? [19:28:19] Free Memory on damiana.esi is CRITICAL: CRITICAL - 5.9% (245700 kB) free! [19:29:59] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [19:29:59] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [19:30:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:31:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:31:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:31:29] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42107 MB (4% inode=99%): [19:33:59] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [19:39:15] shantanoo: what is the error-mesage? [19:40:02] DaBPunkt: *** Unable to read script file because of error: error opening *** [19:40:36] mm, that's a strange errror [19:49:14] Cluster on damiana.esi is CRITICAL: turnera FAILED, damiana:nge1-turnera:nge1 faulted, damiana:nge0-turnera:nge0 faulted, vote turnera Offline, check nfs-hasp, [19:50:15] Cluster on damiana.esi is WARNING: damiana:nge0-turnera:nge0 waiting, check nfs-hasp, [19:51:15] Cluster on damiana.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, nfs OFFLINE, check nfs-hasp, nfs-lh OFFLINE, nfs-home OFFLINE, [19:52:15] Cluster on turnera.esi is CRITICAL: damiana:nge0-turnera:nge0 faulted, check nfs-hasp, nfs-home OFFLINE, [19:52:22] shantanoo: please try again [19:52:25] NTP on turnera.esi is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.00695 secs [19:52:33] (I guess teh script was working before?) [19:52:45] DaBPunkt: still same issue... [19:53:58] shantanoo: where is the script? [19:54:13] DaBPunkt: $HOME/mediawiki/scripts/send_welcome_msg.py [19:55:09] and you started it how? [19:57:58] [[Special:Log/newusers]] create 10 * BALDY131 * (New user account) [19:58:08] I think I see the problem [19:58:53] DaBPunkt: qsub -m bae -l h_rt=0:5:00 -j y -o $HOME/mytool.out \$HOME/mediawiki/scripts/send_welcome_msg.py [19:59:56] try now [20:00:54] AFAIS it runs now [20:03:04] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.406738/1.75, alarm hl:np_load_avg=0.439453/2.00, alarm hl:mem_free=149.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.406738/1.50, alarm hl:np_load_long=0.397461/1.75, alarm hl:mem_free=149.000000M/250M [20:05:26] NTP on turnera.esi is OK: NTP OK: Offset -0.006313 secs [20:11:25] s5 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6811.000000 [20:12:05] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [20:18:56] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11833.000000 [20:21:05] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.429688/1.75, alarm hl:np_load_avg=0.421875/2.00, alarm hl:mem_free=297.000000M/300M [20:25:26] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 15760 MB (1% inode=99%): [20:30:56] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [20:30:56] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [20:31:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:31:16] s5 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3528.000000 [20:31:46] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41996 MB (4% inode=99%): [20:31:46] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:31:55] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:34:38] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [20:36:16] s5 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 945.000000 [20:51:17] @replag [20:51:17] DaBPunkt: s2-pri: 23s [+0.00 s/s]; s2/s5-pri-c: 1h 45m 36s [-0.59 s/s]; s3-rr: 60s [+0.00 s/s]; s3-user: 60s [+0.00 s/s]; s4-rr: 1h 45m 36s [-0.59 s/s]; s4-user: error [20:51:25] Cluster on damiana.esi is CRITICAL: check nfs-hasp, nfs-home OFFLINE, [20:52:26] Cluster on turnera.esi is CRITICAL: check nfs-hasp, nfs-home OFFLINE, [21:10:55] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3540.000000 [21:20:29] zzz =_= [21:22:56] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1759.000000 [21:29:55] Free Memory on damiana.esi is WARNING: WARNING - 9.7% (406992 kB) free! [21:30:56] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [21:30:56] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [21:31:26] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:31:55] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:32:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:32:45] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41800 MB (4% inode=99%): [21:34:56] Free Memory on damiana.esi is OK: OK - 10.6% (444964 kB) free. [21:35:35] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [21:36:13] Hi, Ber :) [21:36:42] Hi jem- ;) [21:36:45] hi all [21:36:57] @replag [21:36:58] techman224: s3-rr: 19s [-0.01 s/s]; s3-user: 19s [-0.01 s/s]; s4-user: error [21:51:26] Cluster on damiana.esi is CRITICAL: check nfs-hasp, nfs-home OFFLINE, [21:51:56] Free Memory on damiana.esi is WARNING: WARNING - 9.8% (408768 kB) free! [21:52:28] Cluster on turnera.esi is CRITICAL: check nfs-hasp, nfs-home OFFLINE, [22:07:08] @replag [22:07:09] Dereckson: s2/s5-pri-c: 1m 25s [-1.37 s/s]; s4-rr: 1m 25s [-1.37 s/s]; s4-user: error [22:22:22] 3(commented) [UTRS-49] Reservation is until assignee navigates away from ticket, not for a specified time <10https://jira.toolserver.org/browse/UTRS-49> (AGK) [22:24:22] 3(resolved) [UTRS-49] Reservation is until assignee navigates away from ticket, not for a specified time <10https://jira.toolserver.org/browse/UTRS-49> (AGK) [22:27:56] Free Memory on damiana.esi is CRITICAL: CRITICAL - 5.0% (211344 kB) free! [22:29:55] Free Memory on damiana.esi is WARNING: WARNING - 7.5% (314056 kB) free! [22:31:05] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [22:31:05] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [22:31:26] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:31:57] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:32:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 41656 MB (4% inode=99%): [22:33:25] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:35:46] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [22:50:23] 3(created) [ACCAPP-453] I plan to use the toolserver for programming of my bot.; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-453> (Symon Klyde) [22:51:45] Cluster on damiana.esi is CRITICAL: check nfs-hasp, nfs-home OFFLINE, [22:52:47] Cluster on turnera.esi is CRITICAL: check nfs-hasp, nfs-home OFFLINE, [23:22:06] Hello [23:22:10] nacht ts [23:22:13] anyone out here? [23:22:20] Ah, DaB. [23:22:37] Ich habe eine kurze Frage [23:22:45] Kannst du noch kurz dableiben? [23:22:48] aber ganz kurz [23:22:56] Ich möchte mich mit deiner USerDB durch mein Script verbinden [23:23:05] PW ist klar, steht in my.cnf [23:23:10] Username ist mmovchin [23:23:20] db name wie ich ihn angelegt habe [23:23:21] und host? [23:23:41] das kommt drauf an, wo du deine db angelegt hast [23:24:09] via phpmyadmin [23:24:13] unter "toolserver" [23:24:19] dann _u_mmovchin_paste [23:24:41] das kann beides nicht sein [23:24:50] wieso? [23:25:18] weil du nur Datenbanken anlegen kanst, die mit "u_mmovchin" anfangen [23:25:39] ja [23:25:43] habe ich ja [23:25:49] "u_mmovchin_paste" [23:25:58] "_u_mmovchin_paste" != "u_mmovchin_paste" [23:26:07] tschuldigung, habe mich vertippt [23:26:33] und was ist dabei der host? [23:26:36] deine Datenbank liegt auf sql.toolserver.org [23:26:44] ok sec [23:26:58] connecting to db FAILED [23:27:18] wieso zeigt es mir nicht an [23:27:25] aber gut, dann mache mich mal debug an [23:27:31] aber ok, ich denke, ich komme zurecht [23:27:36] root@hemlock:~# mysql -h sql.toolserver.org u_mmovchin_paste [23:27:37] danke [23:27:38] geht [23:27:45] also liegt der Fehler irgendwo bei dir :-) [23:27:49] ok gut [23:27:57] Free Memory on damiana.esi is OK: OK - 14.2% (595424 kB) free. [23:27:57] auf gehts PHP code durchforsten :) [23:28:00] viel erfolg [23:28:09] Danke aber für die Auskunft über den host [23:28:11] Gute Nacht dir [23:28:21] gn8 [23:31:06] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:31:06] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:31:56] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:32:25] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:32:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 42416 MB (4% inode=99%): [23:33:26] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:34:44] anybody still here? [23:34:57] Do I have only read access to my own userdb? [23:35:21] Or any DB ending in _p. [23:35:38] enwiki_p, etc. Plus user databases that end in _p. E.g., u_mzmcbride_p; [23:35:46] Sun Grid Engine execd on nightshade is WARNING: NRPE: Unable to read output [23:35:52] You can see which databases you have access to by running... "show databases;" [23:35:56] Creatively. [23:37:42] Why I cannot write to my own? [23:37:49] u_mmovchin_paste [23:41:22] what error do you get? [23:42:22] I just cannot write to my db from my script [23:42:27] in phpmyadmin it works [23:42:32] I don't understand it [23:42:45] again, what error do you get [23:42:58] use mysql_error (in php) [23:43:14] and/or set error_reporting(-1) [23:43:26] (I seldomly use php) [23:43:34] ok [23:43:35] sec [23:48:19] mh, it's empty [23:48:20] http://pastebin.com/eLS0Nzcr [23:49:40] now I have it [23:49:41] Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) connecting to db [23:50:02] without the "connecting to..." [23:50:10] it's: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) [23:51:06] This means SQL is not running? [23:51:10] But this cannot be [23:51:57] Cluster on damiana.esi is CRITICAL: check nfs-hasp, nfs-home OFFLINE, [23:52:44] mmovchin: don't use @mysql_connect [23:52:50] it won't show errors [23:52:56] Cluster on turnera.esi is CRITICAL: check nfs-hasp, nfs-home OFFLINE, [23:53:01] and I'm guessing you are using $dbhost=localhost [23:53:05] which is wrong [23:53:36] no, sql.toolserver.org [23:54:24] I'm using sql.toolserver.org [23:54:26] are you sure? the error notes 'Can't connect to *local* MySQL server through socket (...)' [23:54:52] so check what is actually in $dbhost [23:55:22] oh damn [23:55:31] dbhost is empty [23:55:36] I made a small mistake [23:55:40] I'm so stupid.... [23:55:43] But now: [23:55:45] Access denied for user 'mmovchin'@'wolfsbane.toolserver.org' (using password: NO) [23:55:47] mh [23:56:04] oh wait [23:57:42] I'm off to bed-good luck wiht this [23:57:51] ok thanks [23:57:55] it works now [23:58:01] I left all the parameters out