sv74 Posted March 19, 2014 Report Share Posted March 19, 2014 (edited) Hello, comrades! I have a tough problem to solve Seeking your advice on geoprocessing VERY large volumes of vector geodata. Do not mix geoprecessing (buffer, clip, etc.) with converting between data formats - shp, osm, etc. For example of "large" geodataset you may consider OpenstreetMap Data for entire planet - http://wiki.openstreetmap.org/wiki/Planet.osm "Planet.osm is the OpenStreetMap data in one file: all the nodes, ways and relations that make up our map. A new version is released every week. It's a big file (XML variant over 400GB uncompressed, 29GB compressed)." On these large datasets I need to utilize standard geoprocessing functions (Clip, Buffer, Erase, Select, Find Centroid, Polygon to Polyline). I have server-level hardware (both physical and virtual). The problem is that I did not find any software that is able to process such large volumes of data. All software I tried simply freezes or quits. Splitting entire dataset into smaller chunks that can be processed is highly undesirable. The solution that seems to be in Manifold 8 x64. Manifold 8 x64 (no cure can be found ) utilizes multi-threading technology, so one can use it to geoprocess very large volumes of geodata. If I am not mistaken, this is the only commercial GIS that almost fully utilizes multithreading. Global Mapper 15 x64 is nice but still not enough. Regarding x64 and multithreading ESRI products seems to be almost total failure for many years. A lot of complaints about lack of multithreading on Esri forums. However, "industry leader" is rather slow in reacting to them I suspect that geoprocessing data on ArcGIS Server can be a solution? May be Autodesk or Erdas products can be help too? I believe, that many other forum participants waste a lot of precious time trying to find solution to process large datasets faster and without the need for "split-geoprocess-merge" approach. There is a similar need to process very large volumes of raster datasets and any help will be highly appreciated as well! Regads! Edited March 19, 2014 by sv74 Quote Link to comment Share on other sites More sharing options...
qed Posted March 19, 2014 Report Share Posted March 19, 2014 do try FME safe. they have a 64-bit version in order to make use of max ram. 1 Quote Link to comment Share on other sites More sharing options...
sv74 Posted March 19, 2014 Author Report Share Posted March 19, 2014 Thanks for responding. However, I need not to convert between different geodata formats, but to geoprocess the data itself (clip, buffer, proximity, etc.). Quote Link to comment Share on other sites More sharing options...
qed Posted March 19, 2014 Report Share Posted March 19, 2014 fme does has these functions. do read up on it's transformers. it able to work with large data cause it's pipe in. another solution is QGIS. it's free and read that it's proccesing more efficient than arcgis. 1 Quote Link to comment Share on other sites More sharing options...
sv74 Posted March 19, 2014 Author Report Share Posted March 19, 2014 (edited) fme does has these functions. do read up on it's transformers. it able to work with large data cause it's pipe in. another solution is QGIS. it's free and read that it's proccesing more efficient than arcgis. Thank you for your kind clarification about FME Safe transformers. I will definitely try it. http://fmepedia.safe.com/articles/Samples_and_Demos/Parallel-Processing says: "For 2012, FME has been updated to take advantage of multiple-core processors, which improve parallelization of computations (doing multiple tasks at once) on modern PCs. It also makes use of hyper-threading, a technology used to make each physical core appear as two logical processors to the host operating system. " I agree that QGIS is very good GIS software. Developer version of 2.2 is already x64. Regards! Edited March 19, 2014 by sv74 Quote Link to comment Share on other sites More sharing options...
darksabersan Posted March 19, 2014 Report Share Posted March 19, 2014 Thank you for your kind clarification about FME Safe transformers. I will definitely try it. http://fmepedia.safe.com/articles/Samples_and_Demos/Parallel-Processing says: "For 2012, FME has been updated to take advantage of multiple-core processors, which improve parallelization of computations (doing multiple tasks at once) on modern PCs. It also makes use of hyper-threading, a technology used to make each physical core appear as two logical processors to the host operating system. " I agree that QGIS is very good GIS software. Developer version of 2.2 is already x64. Regards! Agree with you concerning QGIS. In various aspect is much better than ArcGIS. darksabersan. 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.