Batch area calculation Zone-wise

[badar]

Hi,

I have daily landuse maps from which I've extracted vegetation in GeoTiff format of my study area. Each raster file (.tif) is saved with date as file name e.g. 21_12_1937.tif, 22_12_1937.tif and so on. Total Dataset exceeds 10,000 tif files.

Attribute table of any particular day is like that

OID Value Count

0  1    2756

Here Value 1 represents the vegetation which is the only class in all rasters now.

This study area is divided into 10 administrative zones. I want to calculate vegetation area per defined zone. I can do this for a single date by applying "Zonal Statistics as Table" tool in spatial analyst and I get result like this

Rowid Value Count Area  Min Max . . .

0    1    124  12400 1  1  . . .

1    2    981  98100 1  1  . . .

.    .    .    .    .  .  . . .

.    .    .    .    .  .  . . .

.    .    .    .    .  .  . . .

Here Value in this table represents each zone from 1 to 10.

I can use batch processing in arcgis or arcobjects or python scripting to create such tables (INFO tables) for each date which are also saved on disk with dates as their file names. So far, so good.

Now I created a new blank INFO table (named as "OUTPUT"), populated the zones values such that table looks like this

Rowid Value

0    1

1    2

2    3

.    .

.    .

.    .

9    10

I then tried to batch join all tables to "OUTPUT" table and joined only Area Field.

Result was like this

Rowid Value AREA_1 .  .  .  AREA_1_10 .  .  .

0    1    12400  .  .  .  23400    .  .  .

.    .    .      .  .  .  .        .  .  .

.    .    .      .  .  .  .        .  .  .

Here there are total 10 rows with "Value" ranging from 1 to 10 and though I batch joined all tables, the resultant table joined only 511 Area columns (I ran the process many times but each time no area column for 512th or more was joined) so from observation I came to know that columns in INFO table can't be greater than 512 in number. So, I had to break my batch join process into the chunks of 500 tables per run (For each run I got 1 separate output table). I then exported each output table into excel. Replaced AREA_1 .... series with corresponding dates and then joined and transposed each output table in excel. Also handling such huge number of columns in excel is a big issue for me. This is too much time taking and cumbersome. I now have to repeat this process for same number of files for different study area. I don't want to go through all this again. I tried to search forums and googled to get any help on automation of this process but to no avail. Can anybody guide me to the right direction?

Is there any better option in arcgis which I can use to calculate zonal area of some feature (Vegetation in my case) other than "Zonal Statistics as Table"? If no then how can I calculate area of each zone such that zones become field and corresponding areas for each day of each zone are calculated as records/rows?

I am using ArcGIS Desktop v9.3. Any help would be highly appreciated

[Lurker]

yeah, INFO is based on coverage file type, IMHO, it has limitation like old DOS file

so, honestly I never use this tool, but is there any change to use another different type of table? like DBF or others maybe, it will solve the problem

regards,

[badar]

zzzzz. Thank you for the response and Sorry for the late reply. I was working on your suggestion.I created a dbf table and then tried to join all my tables to it. but strangely the process ended "SUCCESSFULLY" after joining 1021 tables and number of tables to be joined were >10,000. When I open the output table, first time opening the table crashed the arcgis. and next time when I open the table, it has only OID field. Not any Value or joined AREA fields. It works for a small number though. I tried this 2 times but same results. Any suggestion?

[Lurker]

woah, that was huge file,

As far as I know, version 9.x have a problem for processing large number of data, I have the experience of it,

Hem, any chance to divide your data? force them to 1 huge table data will almost result in error

hem, what exact version of your arcgis? do you apply latest update for arcgis, it should be arcgis 9.3.1 service pack 2 or arcgis 9.3 service pack 1

regards,

[badar]

I don't know how to get the version of ArcGIS I am using. In "About ArcMap" the version I found was ArcGIS 9.3 build 1770.

Yes, I can break my data into 500 tiles chunks and if my total number of tiles are 10,000, I'll have to divide it into 20 chunks of 500 tiles so that each table out of 20 output tables contains areas of 500 tiles. This will increase the laborious/manual work involved. I can get the results and I GOT the results while doing this for one particular study area. but as study area changes, I've to run the process again and do this laborious work again. And although my tiles are named date-wise, the areas of each tile I get after joining are like AREA_1, AREA_1_1, AREA_1_2 and so on. I have to export each table into excel, see the sequence of tile names in which I joined the tables and replace the "AREA_" series with the corresponding dates accordingly.

So, I am looking for an Automated technique to join areas of all my input tiles the way I want.

Please let me elaborate further. I've all attribute tables of tiles in following  format.

OID Value Area

0      1      123

1      2      234

2      3      345

3      4      456

4      5      567

5      6      678

6      7      789

7      8      890

8      9      900

9      10    100

Say these are finally the areas zone-wise of my study area for 1987_27_03.dbf

The option I could think for automation was that If I could somehow join the areas in the following way, it would solve my Problem

XYZ_Field        Value_1 Value_2 Value_3 Value_4 Value_5 Value_6 Value_7 Value_8 Value_9 Value_10  ------> (Change records of "Value" field into different fields)

1987_27_03    123        234        345      456        567        678        789        890      900        100     

and tables for rest of the tiles are joined such that areas for each new date are added as a new record. I have also been searching about this and found that Pivot Table tool in ArcGIS somewhat does what I want to accomplish (Theoretically) but when I applied this tool for one tile, results were not like what I expected.

Is there a way to achieve above mentioned objective by manipulating "PivotTable_management" in python? I was (may be) not able to understand how to apply Pivot Table tool but even if something like this

1987_27_03.dbf

OID        Value_1 Value_2 Value_3 Value_4 Value_5 Value_6 Value_7 Value_8 Value_9 Value_10 

0            123        234        345      456        567        678        789        890      900        100     

can be achieved using this tool then may be I can look into excel to find a way to open these dbf tables as worksheets in one excel workbook. If this is done then I'll try to find/modify some VBA code to replace 0 in OID Field with the name of corresponding worksheet (which would be like 1987_27_03.dbf) and add new records from other worksheets (with "0" in OID replaced with corresponding name of worksheet)

[Lurker]

okay before we go to the technical stage, I want you to update your arcgis to service pack 1 which is build 1850

download here

[hide]

http://gisupdates.esri.com/93sp1a/ArcGISDesktop93sp1.msp

[/hide]

and re-try your step 

[badar]

Aaaahhhhh. First of all, Thank you for the link. Finally, I understood what the Pivot Tool was all about. Then I manipulated this tool along with append, Add Field, Calculate Field tools in python and at last was able to get my desired output  :cool: . I didn't even have to export my tables into excel or access. My (Only 1) output table now contains all my zones as fields and areas for each zone for each particular date as record (Date is added in Date Field for each record). So, now the output table has limited number of Fields (<15) and large number of records (>10,000) which is far easy and convenient to handle as compared to >10000 fields and <15 records.

I can now play warcraft while my developed tool prepares my table  :cheesy: . Thank you very much Lurker for all your help. This thread can now kindly be closed if needed. 

[Lurker]

congrats, believe me, on this case, you far away above me, ahahahaha,

and nope, this threads remain open forever, so it will help other people, and better, you can share your solution here, if you have spare time

congrats,