[00:22:04] Platonides * Re: [Toolserver-l] SQL-S1 maintenance [00:29:37] nacht ts [00:32:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:34:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:39:20] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:33:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:34:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:16:56] @replag [02:16:57] legoktm: s1-rr-a: error; s1-rr-a-wd: error; s1-user-wd: 29s [-0.47 s/s]; s2-user-wd: 29s [-0.41 s/s]; s3-rr-a: 30s [-0.00 s/s]; s3-user: 30s [-0.00 s/s]; s4-user-wd: 29s [-0.51 s/s]; s5-user-wd: 29s [-0.41 s/s] [02:33:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:34:41] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:39:21] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:12:34] Hey, what's apmon_ ? (jamaican accent) [03:12:49] hey dschwen [03:12:51] did you get around to reimport the coastline data? [03:13:03] Oh, I forgot about that, sorry [03:13:10] no problem [03:13:15] I should try and do that now [03:13:19] you did ask me to remind you [03:13:30] ok, great [03:13:32] I probably should also write a script for doing this. [03:13:44] I have already forgotten again how to do the coast line import :-S [03:14:26] Btw, does the KothicJS work for you now? [03:32:52] nope, not in chrome [03:33:02] cleared my cache and force reloaded [03:33:11] still the same access-control error [03:33:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:34:55] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:35:10] Is it still trying to load tiles with zoom level NaN? [03:39:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:44:06] dschwen: I have now either deleted the entire database, or updated the coast lines. Hopefully the latter... ;-) [03:45:34] Could you please check if everything works again for you? [04:33:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:34:56] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:39:27] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:26:59] apmon_, darn it Dubai is still under water [05:27:24] I don't understand this. The cached JSON tiles do conatin landmass data [05:27:46] but when I purge the cache the new tiles don't have land [05:28:02] and only in that region (as far as I can see) [05:28:20] I even found the boundary to where land is working [05:28:41] well, thanks anyways [05:28:50] I'll have to go to bed now [05:29:11] Well, if there is anything else I can do let me know [05:30:34] thanks, will do [05:31:12] I plan on dumping the DB contents (with reduced resolution) to see if the data is just not there (and if anything else is missing) [05:31:40] Well, at least you are only missing coastlines in dubai... The new set of json tiles don't yet have any coastlines it seems :-S [05:31:44] I cannot think of any reason why this should have suddenly stopped working [05:31:55] I did not change the extraction scripts [05:32:22] ok, bye now [05:32:25] Is there any visualisation of the coastlines alone? [05:32:30] hm? [05:32:42] not on my part [05:32:59] not yet at least [05:33:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:33:59] Looking at the tile rendering which now also has updated coastlines, Dubai seems fine [05:34:23] hm [05:34:37] ok, thanks, that already helps [05:34:46] will investigate tomorrow [05:34:48] bye [05:35:55] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:40:24] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:34:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:36:54] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:41:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:35:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:36:53] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:41:24] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:35:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:36:53] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:41:24] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:18:14] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:23] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:23] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:19:33] RAID on hyacinth is CRITICAL: Connection refused by host [09:19:43] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [09:20:02] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:20:14] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:20:14] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:20:43] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:35:42] MySQL on z-dat-s3-a is OK: Uptime: 957 Threads: 12 Questions: 687 Slow queries: 1 Opens: 247 Flush tables: 1 Open tables: 239 Queries per second avg: 0.717 [09:36:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:36:54] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:42:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:36:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:36:54] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:42:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:36:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:37:03] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:42:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:26:01] @replag [12:26:01] Merlissimo: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 37s [+0.00 s/s]; s1-user-wd: 1m 19s [+0.00 s/s]; s2-user: 15s [+0.00 s/s]; s2-user-wd: 1m 19s [+0.00 s/s]; s4-user-wd: 1m 19s [+0.00 s/s]; s5-user-wd: 1m 19s [+0.00 s/s] [12:36:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:38:03] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:42:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:36:23] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:38:02] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:42:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:49:25] Hi. Anyone having an idea why I cannot connect to the osm_mapnik database on sql-mapnik? [13:53:03] I remember that I could read the catalina.out logfile of tomcat, but now I get a permission denied [13:54:21] so everything I know is, that connection = DriverManager.getConnection(getConnectionUrl(), user, password); returns null, where getConnectionURL results in jdbc:postgresql://sql-mapnik/osm_mapnik and user and password are empty [13:55:03] resulting in this postgresql-connection string: jdbc:postgresql://sql-mapnik/osm_mapnik [14:05:46] jongleur: there was a mail about a removed libary on tomcat. I do not rembeber details (because it did not affect me), but perhaps you should search for this mail [14:05:54] yes, I know [14:06:11] ;) That was my ticket that lead to it. [14:07:01] therefore I added the postgresql-driver to my war file but it doesn't work. Not sure if I miss to register the driver at the drivermanager or if that's not possible to use the driverManager now... [14:08:37] I think, I somehow have to extend jdbc.drivers... [14:26:34] RAID on thyme is CRITICAL: Connection refused by host [14:26:34] SSH on thyme is CRITICAL: Connection refused [14:27:03] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [14:36:54] Environment IPMI on thyme is CRITICAL: Connection refused by host [14:38:54] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:42:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:52:37] Does anyone know if es:Special:UncategorizedPages is uploaded from Toolserver? [14:54:02] MySQL on thyme is OK: Uptime: 2058 Threads: 2 Questions: 495 Slow queries: 0 Opens: 50 Flush tables: 1 Open tables: 43 Queries per second avg: 0.240 [15:26:16] Magnus Manske * [Toolserver-l] 502 [15:27:32] RAID on thyme is CRITICAL: Connection refused by host [15:27:34] SSH on thyme is CRITICAL: Connection refused [15:31:16] Dennis Tobar * Re: [Toolserver-l] 502 [15:32:16] Jeremy Baron * Re: [Toolserver-l] 502 [15:33:16] Sumurai8 (DD) * Re: [Toolserver-l] 502 [15:34:16] Simon Walker * Re: [Toolserver-l] 502 [15:36:54] Environment IPMI on thyme is CRITICAL: Connection refused by host [15:38:55] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:40:16] Mardetanha * Re: [Toolserver-l] 502 [15:42:09] MrAjedrez: the content of original special pages is never created at toolserver. [15:42:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:42:47] Then, Merlissimo, do you know where is that db generated? [15:43:53] the scripts are using the same database slave as watchlists [15:44:06] on the wmf cluster [15:46:10] Ok, thanks [15:52:16] Magnus Manske * Re: [Toolserver-l] 502 [15:56:53] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [16:09:56] isn't anyone available to find out what happened to /mnt/user-store? [16:10:51] apmon_ it seems the coastline data is there, and it is present in the JSON tile data. I'm still investigating why it is not drawn, and why only there [16:11:08] Ah, good to know [16:11:50] It probably was good to do an update of the coastline data though anyway. Once every month doesn't seem like an unreasonable update cycle [16:17:28] I don't know what I'm talking about :-( [16:17:45] Nor I'm stepping through my code and the ocean data is not there [16:18:03] I saw it in the network tab of the webkit debugger [16:18:06] grrr [16:27:33] RAID on thyme is CRITICAL: Connection refused by host [16:27:34] SSH on thyme is CRITICAL: Connection refused [16:28:31] hey [16:36:54] Environment IPMI on thyme is CRITICAL: Connection refused by host [16:37:56] Is there an issue with NFS? The /tiles mount on ptolemy seems to have gotten dropped. [16:39:02] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:42:22] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:56:54] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [17:27:33] RAID on thyme is CRITICAL: Connection refused by host [17:27:33] SSH on thyme is CRITICAL: Connection refused [17:36:53] Environment IPMI on thyme is CRITICAL: Connection refused by host [17:39:03] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:39:35] any admin here? I can't get a shell [17:42:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:55:02] @replag [17:55:03] Merlissimo: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 6m 5s [+0.02 s/s]; s1-user-wd: 3h 38m 50s [+0.66 s/s]; s2-user-wd: 3h 38m 50s [+0.66 s/s]; s4-user-wd: 3h 38m 53s [+0.66 s/s]; s5-user-wd: 3h 38m 50s [+0.66 s/s]; s7-rr-a: error [17:55:04] Merlissimo: s7-user: error [17:56:53] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:25:53] MySQL on z-dat-s7-a is OK: Uptime: 64 Threads: 2 Questions: 4 Slow queries: 0 Opens: 17 Flush tables: 1 Open tables: 10 Queries per second avg: 0.62 [18:27:33] RAID on thyme is CRITICAL: Connection refused by host [18:27:33] SSH on thyme is CRITICAL: Connection refused [18:32:54] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [18:36:53] Environment IPMI on thyme is CRITICAL: Connection refused by host [18:40:03] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:42:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:03:32] hello, I want help [19:27:34] RAID on thyme is CRITICAL: Connection refused by host [19:27:34] SSH on thyme is CRITICAL: Connection refused [19:32:53] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [19:34:50] helllo all [19:36:54] Environment IPMI on thyme is CRITICAL: Connection refused by host [19:38:14] Any one can help me? [19:40:02] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:42:23] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:44:43] shuaib_: help with what? [19:45:12] in toolserver when we create crontab in putty [20:07:53] MySQL on z-dat-s7-a is OK: Uptime: 41 Threads: 5 Questions: 321 Slow queries: 0 Opens: 97 Flush tables: 1 Open tables: 90 Queries per second avg: 7.829 [20:16:03] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:18:54] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.005 second response time [20:20:51] apmon_, I found the issue with the Dubai coastline! [20:21:10] Oh, good. What was it? [20:21:31] After the DB corruption incident I added ST_IsValid to all my queries [20:21:43] the dubai coastline is not valid [20:21:59] if I remove that test it outputs the correct coastline [20:22:25] Ah, someone should fix the coast lines then... ;-) [20:22:38] should I risk removing the test for the coastline data only? [20:23:01] I wonder why the tiles are rendered correctly [20:23:05] Well, as long as you don't crash postgresql every couple of seconds again... ;-) [20:23:15] well... let's test [20:24:18] Mapnik doesn't include the ST_IsValid test. The issue is that invalid geometries break further processing in postgis e.g. ST_Intersect. Mapnik does that processing it self, so doesn't suffer from the bug in postgis [20:24:42] dschwen: Now is a good time to test, as things are broken on ptolemy anyway at the moment [20:25:46] The correct fix though is indeed to fix the coast line of dubai. Osm2pgsql (which I guess doesn't effect coastlines, as they are imported as separate shape files) does now exclude all invalid geometries in the diff processing [20:27:33] RAID on thyme is CRITICAL: Connection refused by host [20:27:33] SSH on thyme is CRITICAL: Connection refused [20:36:55] Environment IPMI on thyme is CRITICAL: Connection refused by host [20:38:24] seems to work ok [20:39:17] hmm, the coastlien is fixable in josm? [20:39:50] Good, as ptolemy is now also back up and running properly after a reboot due to a failed NFS mount [20:40:03] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:40:12] yes, either in josm or potlatch2 [20:42:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:27:34] RAID on thyme is CRITICAL: Connection refused by host [21:27:35] SSH on thyme is CRITICAL: Connection refused [21:34:16] DaB. * Re: [Toolserver-announce] [Toolserver-l] SQL-S1 maintenance (was: Re: sql-s1-rr will be switched this evening) [21:37:04] Environment IPMI on thyme is CRITICAL: Connection refused by host [21:39:16] DaB. * Re: [Toolserver-announce] [Toolserver-l] SQL-S1 maintenance [21:40:02] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:42:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:28:33] RAID on thyme is CRITICAL: Connection refused by host [22:28:34] SSH on thyme is CRITICAL: Connection refused [22:38:03] Environment IPMI on thyme is CRITICAL: Connection refused by host [22:39:54] MySQL on z-dat-s7-a is CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [22:40:03] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:42:34] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:45:53] MySQL on z-dat-s7-a is OK: Uptime: 771 Threads: 3 Questions: 935 Slow queries: 0 Opens: 151 Flush tables: 1 Open tables: 141 Queries per second avg: 1.212 [23:02:23] @replag [23:02:24] DaBPunkt: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 13m 57s [+0.03 s/s]; s1-user-wd: 8h 45m 12s [+1.00 s/s]; s2-user-wd: 8h 44m 48s [+1.00 s/s]; s4-user-wd: 8h 45m 7s [+1.00 s/s]; s5-user-wd: 8h 44m 49s [+1.00 s/s] [23:03:48] @replag [23:03:49] DaBPunkt: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 14m 57s [+0.71 s/s]; s1-user-wd: 8h 42m 8s [-2.18 s/s]; s2-user-wd: 8h 40m 15s [-3.23 s/s]; s4-user-wd: 8h 40m 38s [-3.18 s/s]; s5-user-wd: 8h 40m 15s [-3.24 s/s] [23:14:14] DaBPunkt: I can't SSH (at all) into nightshade, but willow works [23:14:46] Dispenser: yes, same here. I wait for puppet to fix thisd [23:15:15] Ok, thanks... [23:22:16] Andre Koopal * Re: [Toolserver-l] mdbtools [23:29:33] RAID on thyme is CRITICAL: Connection refused by host [23:29:33] SSH on thyme is CRITICAL: Connection refused [23:38:03] Environment IPMI on thyme is CRITICAL: Connection refused by host [23:40:03] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:43:33] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)