Search the Community

Hello, comrades! I have a tough problem to solve Seeking your advice on geoprocessing VERY large volumes of vector geodata. Do not mix geoprecessing (buffer, clip, etc.) with converting between data formats - shp, osm, etc. For example of "large" geodataset you may consider OpenstreetMap Data for entire planet - http://wiki.openstreetmap.org/wiki/Planet.osm "Planet.osm is the OpenStreetMap data in one file: all the nodes, ways and relations that make up our map. A new version is released every week. It's a big file (XML variant over 400GB uncompressed, 29GB compressed)." On these large datasets I need to utilize standard geoprocessing functions (Clip, Buffer, Erase, Select, Find Centroid, Polygon to Polyline). I have server-level hardware (both physical and virtual). The problem is that I did not find any software that is able to process such large volumes of data. All software I tried simply freezes or quits. Splitting entire dataset into smaller chunks that can be processed is highly undesirable. The solution that seems to be in Manifold 8 x64. Manifold 8 x64 (no cure can be found ) utilizes multi-threading technology, so one can use it to geoprocess very large volumes of geodata. If I am not mistaken, this is the only commercial GIS that almost fully utilizes multithreading. Global Mapper 15 x64 is nice but still not enough. Regarding x64 and multithreading ESRI products seems to be almost total failure for many years. A lot of complaints about lack of multithreading on Esri forums. However, "industry leader" is rather slow in reacting to them I suspect that geoprocessing data on ArcGIS Server can be a solution? May be Autodesk or Erdas products can be help too? I believe, that many other forum participants waste a lot of precious time trying to find solution to process large datasets faster and without the need for "split-geoprocess-merge" approach. There is a similar need to process very large volumes of raster datasets and any help will be highly appreciated as well! Regads!