[00:00:17] nacht ts [00:04:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [00:22:16] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.169922/1.00, alarm hl:mem_free=12392.000000M/100M [00:31:18] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:32:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:38:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:40:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59887 MB (6% inode=99%): [00:42:04] zzz =_= [00:58:17] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [01:08:19] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.651367/1.00, alarm hl:mem_free=10887.000000M/100M [01:13:18] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [01:16:48] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Proxy Error - 150 bytes in 2.588 second response time [01:18:58] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 8.888 second response time [01:32:17] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:33:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:34:31] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:38:21] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:40:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59757 MB (6% inode=99%): [01:50:18] [[Special:Log/newusers]] create 10 * UnQuébécois * (New user account) [02:32:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:33:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:38:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:40:10] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59630 MB (6% inode=99%): [03:04:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:32:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:33:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:39:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:40:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59529 MB (6% inode=99%): [04:29:51] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [04:29:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:30:11] MySQL slave on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [04:32:51] MySQL on thyme is OK: Uptime: 145 Threads: 5 Questions: 4 Slow queries: 0 Opens: 18 Flush tables: 1 Open tables: 9 Queries per second avg: 0.27 [04:33:00] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 309.000000 [04:33:12] MySQL slave on thyme is OK: Uptime: 167 Threads: 8 Questions: 1982 Slow queries: 4 Opens: 151 Flush tables: 1 Open tables: 141 Queries per second avg: 11.868 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 315 [04:33:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:33:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:34:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:39:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:40:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60268 MB (6% inode=99%): [05:34:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:34:30] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:34:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:39:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:41:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60029 MB (6% inode=99%): [06:34:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:34:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:34:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:40:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:41:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60067 MB (6% inode=99%): [07:18:11] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.556641/1.50, alarm hl:mem_free=1098.000000M/100M [07:19:10] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:23:11] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.638184/1.50, alarm hl:mem_free=926.000000M/100M [07:27:21] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 56655 MB (5% inode=99%): [07:34:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:34:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:41:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59952 MB (6% inode=99%): [07:41:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:04:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [08:34:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:35:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:41:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59837 MB (6% inode=99%): [08:41:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:04:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:12:39] @replag [09:12:40] matthewrbowker: s2-pri: 1m 6s [+0.00 s/s]; s2/s5-pri-c: 10s [-0.12 s/s]; s3-rr: 1m 15s [+0.00 s/s]; s3-user: 1m 15s [+0.00 s/s] [09:34:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:35:34] anybody here who knows about Java deployment on the Toolserver? [09:35:40] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:36:47] I deployed my lalm.war file now and the catalina.err log says "Deploying" without printing any errors, but I have no idea how to access the application now [09:41:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59773 MB (6% inode=99%): [09:41:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:04:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:35:41] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:35:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:41:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59650 MB (6% inode=99%): [10:41:51] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:34:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:35:54] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:35:54] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:41:54] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59490 MB (6% inode=99%): [11:41:54] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:02:31] zzz [12:34:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:36:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:37:03] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:42:03] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:42:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57298 MB (5% inode=99%): [13:34:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:36:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:38:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:42:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:42:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 27063 MB (2% inode=99%): [14:34:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:37:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:38:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:42:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:42:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59975 MB (6% inode=99%): [15:34:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:38:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:38:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:42:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59827 MB (6% inode=99%): [15:43:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:38:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:38:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:43:21] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:43:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59655 MB (6% inode=99%): [17:04:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:31:18] @replag all [17:31:19] nosy: s1-pri: 1s [-0.00 s/s]; s1-sec: 0s [-0.00 s/s]; s1-sec-c: 4s [-0.05 s/s]; s2-pri: 5s [-0.00 s/s]; s2/s5-pri-c: 4s [-0.00 s/s]; s3-rr: 20s [-0.00 s/s]; s3-user: 20s [-0.00 s/s]; s4-rr: 4s [-0.05 s/s] [17:31:20] nosy: s4-user: 4s [-0.04 s/s]; s5-rr: 1s [-]; s5-user: 1s [-]; s6-rr: 2s [-]; s6-user: 2s [-]; s7-rr: 1s [-0.00 s/s]; s7-user: 1s [-0.00 s/s] [17:38:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:38:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:43:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:43:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59503 MB (6% inode=99%): [18:04:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:38:24] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:38:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:43:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:44:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59329 MB (6% inode=99%): [19:06:51] Hi. Anybody here who is able to help with Tomcat/Java software on the Toolserver? [19:25:54] or does anybody know when DaBPunkt is here again/currently? [19:27:23] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 58967 MB (6% inode=99%): [19:34:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:37:57] jongleur: He's usually here Monday through Friday. [19:38:02] Sometimes on the weekends, too. [19:38:12] He's in Germany. [19:38:19] I know, I'm too. [19:38:26] (from germany) [19:39:02] He helped me a lot end of december, but I wasn't at home and was not able to finally test my project locally after some recoding. [19:39:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:39:31] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:40:19] Now I deployed a .war-file (java/servlet application) to the Toolserver and the logs don't print errors any more as far as I can see, but I have no idea how to access the website according to the war nor how to debug further if necessary (database access might still be buggy) [19:43:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:44:24] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59163 MB (6% inode=99%): [20:34:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:39:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:39:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:43:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:45:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59933 MB (6% inode=99%): [21:34:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [21:39:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:39:41] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:43:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:45:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59773 MB (6% inode=99%): [22:13:23] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:14:00] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [22:18:39] Hi, I have a db problem on the toolserver that is currently affecting the Valued Images Project on Commons badly. [22:19:05] Th main query of the vic bot throws mysql error [22:19:06] Error 1159: [22:19:08] Got timeout reading communication packets [22:19:37] what is the deal with that? my other bots all run fine, just this one acts up [22:21:10] Do you know which query it's running, on which DB and which server? [22:22:19] one sec, yes [22:22:28] commonswiki_p [22:22:39] sql-s3 [22:22:56] select /* SLOW_OK */ page_title, GROUP_CONCAT( DISTINCT cl_to SEPARATOR '|') from revision, page left join categorylinks on page_id = cl_from where page_latest=rev_id and page_title like 'Valued_image_candidates/%' and page_namespace=4 and ( TO_DAYS(CURRENT_DATE) - TO_DAYS(rev_timestamp) ) < 5 group by page_id [22:27:26] trying commonswiki-p.rrdb.toolserver.org instead now [22:28:06] s3 is picked by a procedure that determines the least replagged commons server. It is not written by me and might be out of date. [22:30:37] sigh [22:30:42] yep, that was it [22:31:01] thanks! ;-) [22:31:06] :-) [22:34:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [22:34:43] Mh. Getting "HTTPError: 504 Gateway Time-out" in pywikipedia on commons again [22:35:08] not the first time [22:39:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:39:42] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:40:01] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:53] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [22:43:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:46:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59614 MB (6% inode=99%): [23:34:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:39:45] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:39:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:43:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:46:20] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59480 MB (6% inode=99%): [23:48:33] zzz