[00:03:33] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.735351/1.50, alarm hl:mem_free=181.000000M/100M: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.276855/2.00, alarm hl:mem_free=181.000000M/250M [00:13:34] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:20:14] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.025391/1.00, alarm hl:mem_free=16257.000000M/100M [00:21:13] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [00:26:13] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:26:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:26:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 250776 MB (4% inode=30%): [00:27:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:27:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67603 MB (6% inode=99%): [00:33:33] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.159668/2.00, alarm hl:mem_free=173.000000M/250M [00:38:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:03:33] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.591309/1.50, alarm hl:mem_free=284.000000M/100M [01:06:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:19:34] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.420899/2.00, alarm hl:mem_free=214.000000M/250M [01:26:13] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:26:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:26:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 253541 MB (4% inode=31%): [01:27:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:27:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68385 MB (7% inode=99%): [01:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:03:34] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.521973/1.50, alarm hl:mem_free=300.000000M/100M [02:04:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:07:43] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=2.137695/1.50, alarm hl:mem_free=1193.000000M/100M [02:07:53] Load avg. on nightshade is WARNING: WARNING - load average: 17.81, 16.16, 11.70 [02:08:54] Load avg. on nightshade is OK: OK - load average: 11.72, 14.74, 11.48 [02:18:43] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [02:21:43] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.937012/1.50, alarm hl:mem_free=1082.000000M/100M [02:26:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:26:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:26:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 253487 MB (4% inode=31%): [02:27:22] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:27:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:27:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68127 MB (6% inode=99%): [02:35:34] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.242188/2.00, alarm hl:mem_free=154.000000M/250M [02:37:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:37:43] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [02:43:12] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 273 bytes in 18.163 second response time [02:44:03] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 9.113 second response time [02:45:34] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.180664/2.00, alarm hl:mem_free=148.000000M/250M [02:47:43] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.856934/1.50, alarm hl:mem_free=774.000000M/100M [02:49:53] Load avg. on nightshade is WARNING: WARNING - load average: 21.84, 19.93, 15.66 [02:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:53:54] Load avg. on nightshade is OK: OK - load average: 8.20, 14.41, 14.43 [02:58:43] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [03:18:24] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:26:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:26:54] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 231573 MB (4% inode=29%): [03:27:23] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:27:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:27:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67928 MB (6% inode=99%): [03:28:12] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 14.445 second response time [03:31:23] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:26:44] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:27:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 231557 MB (4% inode=29%): [04:27:43] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:28:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67725 MB (6% inode=99%): [04:28:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:03:54] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.154785/2.00, alarm hl:mem_free=223.000000M/250M [05:04:54] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:11:53] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.183105/2.00, alarm hl:mem_free=154.000000M/250M [05:23:53] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.583984/1.50, alarm hl:mem_free=1837.000000M/100M [05:26:03] Load avg. on nightshade is WARNING: WARNING - load average: 15.98, 14.53, 11.66 [05:26:54] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:27:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 231365 MB (4% inode=29%): [05:27:53] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:28:02] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67457 MB (6% inode=99%): [05:29:03] Load avg. on nightshade is OK: OK - load average: 10.64, 13.94, 12.02 [05:29:23] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:33:03] Load avg. on nightshade is WARNING: WARNING - load average: 12.06, 16.46, 13.77 [05:36:55] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [05:47:53] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.633789/1.50, alarm hl:mem_free=2113.000000M/100M [05:49:53] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [05:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:13:03] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.638184/1.50, alarm hl:mem_free=2082.000000M/100M [06:13:03] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.684082/1.50, alarm hl:mem_free=309.000000M/100M [06:13:13] Load avg. on willow is WARNING: WARNING - load average: 18.90, 14.58, 12.32 [06:15:03] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [06:20:13] Load avg. on willow is OK: OK - load average: 9.78, 14.16, 13.46 [06:24:03] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:27:03] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:27:13] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 239889 MB (4% inode=29%): [06:28:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:28:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67477 MB (6% inode=99%): [06:29:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:30:04] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.901367/1.50, alarm hl:mem_free=449.000000M/100M [06:30:14] Load avg. on willow is WARNING: WARNING - load average: 16.91, 15.14, 13.68 [06:33:02] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.949219/1.50, alarm hl:mem_free=1993.000000M/100M [06:39:03] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [06:43:43] Sun Grid Engine execd on wolfsbane is WARNING: all.q@wolfsbane exceedes load threshold: alarm hl:np_load_avg=1.114258/1.00, alarm hl:mem_free=1066.000000M/100M [06:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:55:42] Sun Grid Engine execd on wolfsbane is OK: all.q@wolfsbane OK [07:09:14] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 84884 MB (8% inode=99%): [07:27:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:27:13] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 239615 MB (4% inode=29%): [07:28:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68302 MB (7% inode=99%): [07:28:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:29:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:30:14] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.734375/1.50, alarm hl:mem_free=230.000000M/100M: longrun@willow exceedes load threshold: alarm hl:np_load_long=2.127930/2.00, alarm hl:mem_free=230.000000M/250M [07:51:24] Load avg. on willow is WARNING: WARNING - load average: 12.55, 16.90, 17.22 [07:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:55:27] 3(created) [MAGNUS-286] Please, add option to skip last steep and just add {{NowCommons}}; Magnus' tools; New Feature <10https://jira.toolserver.org/browse/MAGNUS-286> (bulwersator) [07:59:23] Load avg. on willow is OK: OK - load average: 8.84, 11.76, 14.67 [08:00:13] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:02:23] Load avg. on willow is WARNING: WARNING - load average: 14.25, 13.96, 15.08 [08:03:13] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.735351/1.50, alarm hl:mem_free=415.000000M/100M [08:03:23] Load avg. on willow is OK: OK - load average: 13.16, 13.64, 14.89 [08:09:13] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:27:13] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 237630 MB (4% inode=29%): [08:27:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:28:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 68169 MB (6% inode=99%): [08:28:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:29:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:53:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:11:23] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.236816/2.00, alarm hl:mem_free=195.000000M/250M [09:12:24] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:16:23] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.269531/2.00, alarm hl:mem_free=222.000000M/250M [09:27:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 239387 MB (4% inode=29%): [09:27:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:28:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67937 MB (6% inode=99%): [09:28:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:29:54] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:04:03] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.938476/1.00, alarm hl:mem_free=14794.000000M/100M [10:10:03] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [10:27:23] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 247496 MB (4% inode=30%): [10:27:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:28:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67737 MB (6% inode=99%): [10:28:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:30:03] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:38:04] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.090820/1.00, alarm hl:mem_free=15350.000000M/100M [10:44:23] Load avg. on nightshade is WARNING: WARNING - load average: 17.53, 12.60, 8.40 [10:46:22] Load avg. on nightshade is OK: OK - load average: 12.80, 12.70, 8.97 [10:46:33] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.597168/1.50, alarm hl:mem_free=1931.000000M/100M [10:47:33] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [10:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:06:03] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [11:23:02] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.228516/1.00, alarm hl:mem_free=15300.000000M/100M [11:27:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 247344 MB (4% inode=30%): [11:27:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:28:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67512 MB (6% inode=99%): [11:28:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:30:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:52:34] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.533203/1.50, alarm hl:mem_free=614.000000M/100M [11:53:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:53:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:13:36] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.225586/2.00, alarm hl:mem_free=230.000000M/250M [12:15:36] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [12:17:16] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [12:27:16] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.038086/1.00, alarm hl:mem_free=15036.000000M/100M [12:27:35] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 247190 MB (4% inode=30%): [12:27:35] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:28:35] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67442 MB (6% inode=99%): [12:28:36] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:30:15] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:49:16] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [13:01:16] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.092774/1.00, alarm hl:mem_free=15180.000000M/100M [13:02:35] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.545899/1.50, alarm hl:mem_free=279.000000M/100M [13:07:35] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [13:10:28] 3(created) [TS-1248] Expose aft_* tables; Toolserver: Databases; Task <10https://jira.toolserver.org/browse/TS-1248> (Roan Kattouw) [13:10:35] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.388184/2.00, alarm hl:mem_free=211.000000M/250M [13:17:16] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [13:20:16] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.000976/1.00, alarm hl:mem_free=15134.000000M/100M [13:22:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:27:35] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:27:35] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 247001 MB (4% inode=30%): [13:28:35] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67161 MB (6% inode=99%): [13:28:36] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:30:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:52:45] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.542480/1.50, alarm hl:mem_free=811.000000M/100M [13:53:45] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [14:07:45] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.611328/1.50, alarm hl:mem_free=930.000000M/100M [14:15:45] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [14:22:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:27:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 246869 MB (4% inode=30%): [14:27:46] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:28:34] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67035 MB (6% inode=99%): [14:28:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:30:26] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:36:56] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.736328/1.50, alarm hl:mem_free=572.000000M/100M [14:47:56] Load avg. on willow is WARNING: WARNING - load average: 16.30, 15.23, 13.46 [14:48:55] Load avg. on willow is OK: OK - load average: 13.61, 14.66, 13.37 [14:52:55] Load avg. on willow is WARNING: WARNING - load average: 14.20, 15.11, 13.90 [15:22:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:27:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 250462 MB (4% inode=30%): [15:27:55] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:28:45] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66906 MB (6% inode=99%): [15:28:55] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:31:26] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:35:50] hello all [15:35:54] @replag [15:35:55] DaBPunkt: s2-pri: 11s [+0.00 s/s]; s3-rr: 1m 39s [+0.00 s/s]; s3-user: 1m 39s [+0.00 s/s] [15:37:55] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.859863/1.50, alarm hl:mem_free=212.000000M/100M: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.835938/2.00, alarm hl:mem_free=212.000000M/250M [15:41:20] 3(assigned) [TS-1248] Expose aft_* tables <10https://jira.toolserver.org/browse/TS-1248> (DaB.) [15:41:24] 3(work started) [TS-1248] Expose aft_* tables <10https://jira.toolserver.org/browse/TS-1248> (DaB.) [16:08:25] 3(resolved) [TS-1248] Expose aft_* tables <10https://jira.toolserver.org/browse/TS-1248> (DaB.) [16:10:21] 3(commented) [TS-1248] Expose aft_* tables <10https://jira.toolserver.org/browse/TS-1248> (DaB.) [16:12:22] 3(commented) [TS-1248] Expose aft_* tables <10https://jira.toolserver.org/browse/TS-1248> (Roan Kattouw) [16:17:17] anyone have a quick minute to help with a javascript bookmarklet? [16:17:22] 3(commented) [TS-1248] Expose aft_* tables <10https://jira.toolserver.org/browse/TS-1248> (DaB.) [16:22:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:27:46] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 248854 MB (4% inode=30%): [16:27:55] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:28:45] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67719 MB (6% inode=99%): [16:28:55] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:29:55] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [16:32:26] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:32:55] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.609863/1.50, alarm hl:mem_free=263.000000M/100M [16:44:54] Load avg. on willow is WARNING: WARNING - load average: 17.59, 15.54, 14.30 [16:52:55] Load avg. on willow is OK: OK - load average: 12.77, 14.99, 14.79 [17:02:56] Load avg. on willow is WARNING: WARNING - load average: 15.48, 16.12, 15.20 [17:22:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:24:35] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.020508/1.00, alarm hl:mem_free=14137.000000M/100M [17:25:36] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [17:27:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 246124 MB (4% inode=30%): [17:28:04] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:28:45] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67580 MB (6% inode=99%): [17:29:05] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:32:36] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:33:15] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.609375/1.50, alarm hl:mem_free=602.000000M/100M [17:37:15] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [17:45:46] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.301758/1.00, alarm hl:mem_free=16255.000000M/100M [17:47:16] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.570312/1.50, alarm hl:mem_free=432.000000M/100M [17:47:45] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [18:01:05] Load avg. on willow is WARNING: WARNING - load average: 28.82, 19.30, 15.47 [18:05:05] Load avg. on willow is OK: OK - load average: 9.70, 14.51, 14.42 [18:18:23] 3(commented) [ACCAPP-423] Studying the relationship between box office revenues of Hollywood films and the statistics on their Wikipedia articles <10https://jira.toolserver.org/browse/ACCAPP-423> [18:22:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:27:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 245132 MB (4% inode=30%): [18:28:05] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:28:44] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67440 MB (6% inode=99%): [18:29:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:33:35] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:40:15] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [18:43:16] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.502930/1.50, alarm hl:mem_free=651.000000M/100M [18:44:56] I have a problem again with access rights as lalm: it looks like I have no rights to select from planet_ways in osm_mapnik [18:46:48] while planet_line has an access entry for osm, planet_ways (to me) doesn't list any privileges using \z planet_ways [18:48:05] Load avg. on willow is WARNING: WARNING - load average: 15.82, 15.47, 14.10 [18:49:04] Load avg. on willow is OK: OK - load average: 13.73, 14.96, 14.00 [18:49:49] same for the other raw tables... [18:50:41] DaBPunkt: would it be possible to give me the select privilege on planet_nodes, planet_rels and planet_ways? [19:08:20] 3(commented) [TS-1246] MMP for the Look-and-Listen-Map <10https://jira.toolserver.org/browse/TS-1246> (Peter Wendorff) [19:09:55] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 85448 MB (8% inode=99%): [19:22:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:27:55] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 245068 MB (4% inode=30%): [19:28:15] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:28:15] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [19:28:55] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67296 MB (6% inode=99%): [19:29:15] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:33:36] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:36:27] jongleur: ask apmon, I have no idea about postgres :) [19:36:35] ;) ok [19:36:43] :/ another round of waiting... [19:36:48] thanks [19:38:11] DaBPunkt: can I reopen the Ticket and assign it to apmon myself? or is another way preferred? [19:38:25] please create a new one [19:38:34] and assign it to him [19:38:35] ok [19:41:50] who is apmon as a user? [19:42:32] Kai Krueger [19:43:57] thanks ;) [19:44:37] np [19:48:15] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.569824/1.50, alarm hl:mem_free=439.000000M/100M [19:50:22] 3(created) [TS-1249] SELECT rights on "raw" tables of osm_mapnik; Toolserver: Databases; Blocker Task <10https://jira.toolserver.org/browse/TS-1249> (Peter Wendorff) [19:59:15] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [20:03:15] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.749024/1.50, alarm hl:mem_free=178.000000M/100M: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.637207/2.00, alarm hl:mem_free=178.000000M/250M [20:18:20] hi apmon :D [20:21:57] hi [20:22:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:23:01] I write directly because I'm not sure if it's possible... opened a ticket a few minutes ago about SELECT-privileges on the mapnik database [20:23:11] https://jira.toolserver.org/browse/TS-1249 [20:25:31] Got an email alert about it [20:25:45] ;) I know, assigned you. [20:25:52] I'lll see if I can deal with it later today [20:25:57] I am in a meeting at the moment [20:26:03] great, thanks [20:27:55] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 244325 MB (4% inode=30%): [20:28:15] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:28:55] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67154 MB (6% inode=99%): [20:29:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:33:46] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:46:18] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.556152/1.50, alarm hl:mem_free=177.000000M/100M: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.447754/2.00, alarm hl:mem_free=177.000000M/250M [20:49:21] 3(commented) [ACCAPP-423] Studying the relationship between box office revenues of Hollywood films and the statistics on their Wikipedia articles <10https://jira.toolserver.org/browse/ACCAPP-423> (DaB.) [20:50:18] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [21:08:18] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=1.345215/2.00, alarm hl:mem_free=190.000000M/250M [21:09:18] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [21:22:59] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [21:28:08] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 242669 MB (4% inode=30%): [21:28:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:28:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 67007 MB (6% inode=99%): [21:29:18] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:33:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:01:58] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1929 [22:15:49] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1915 [22:22:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [22:28:08] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 240867 MB (4% inode=29%): [22:28:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:29:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66860 MB (6% inode=99%): [22:29:17] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:34:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:35:58] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3634 [23:00:48] MySQL slave on z-dat-s7-a is OK: Uptime: 2074593 Threads: 9 Questions: 451515724 Slow queries: 50552 Opens: 5070105 Flush tables: 1 Open tables: 6372 Queries per second avg: 217.640 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1585 [23:00:58] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3578 [23:19:59] MySQL slave on z-dat-s3-a is OK: Uptime: 808828 Threads: 15 Questions: 771531171 Slow queries: 105615 Opens: 11734427 Flush tables: 2 Open tables: 16368 Queries per second avg: 953.887 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1689 [23:22:58] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:28:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 227956 MB (4% inode=28%): [23:28:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:29:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 66727 MB (6% inode=99%): [23:29:18] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:32:21] 3(created) [ACCAPP-427] Analysis for creating successful administrator model, ad hoc reporting by request; Account Approval; Minor New Account <10https://jira.toolserver.org/browse/ACCAPP-427> (Bob Kaplan) [23:34:18] SSH on nightshade.mgmt is CRITICAL: Server answer: