2
\$\begingroup\$

I have the following function and code snippet to extract cell values for multiple years, format it, and save to a list. Each raster has 365 bands — one for each day. A separate operation is performed on each flattened list pr_flt which contains daily cell values of XY for 1991-2015. The cursor contains XY coordinates for each cell. The following code works but it takes hours to finish due to the huge amount of data and possible use of lists instead of multidimensional arrays. I am wondering if there are any suggestions to reduce the runtime of the code, if possible.

import arcpy def row_to_values(row): values = [] for col in row: if isinstance(col, unicode) and col != u'f': # split and convert all entries to float values += (float(v) for v in col.split(',')) else: values.append(col) return values with arcpy.da.SearchCursor(fc, fields) as cursor: for idx, row in enumerate(cursor): pt_annual_lists = [] pt_loc = str(row[0]) + str(" ") + str(row[1]) for y in xrange(1991,2016,1): pt_year_list = [] result = arcpy.GetCellValue_management(in_raster = "D:/temp/ras" + str(y) + ".tiff", location_point = pt_loc, band_index="") cell_value = result.getOutput(0) pt_year_list.append(cell_value) cell_value = [s.replace('\\n', ',') for s in pt_year_list] pt_annual_lists.append(row_to_values(cell_value)) # flatten list of lists pr_str = [val for sublist in pt_annual_lists for val in sublist] pr_flt = [float(i) for i in pr_str] 
\$\endgroup\$
5
  • \$\begingroup\$If you're truly interested in performance don't code in python, use C++ or C# if you find C++ too daunting. If the total size of the raster is smaller than your available memory you can read the whole raster into an array (RasterToNumpyArray resources.arcgis.com/en/help/main/10.2/index.html#//…) then index the array rather than trying to read a single cell at a time; if your raster is compressed the GetCellValue needs to work very hard to get a single cell value.\$\endgroup\$
    – Michael Stimson
    CommentedMay 19, 2019 at 23:11
  • 1
    \$\begingroup\$You would be more likely to get an answer if you included enough information to actually run the code.\$\endgroup\$CommentedMay 20, 2019 at 4:17
  • 1
    \$\begingroup\$Michael, that's not really true. Numpy is typically about as fast as handwritten c code due to being a heavily optimized library. Numpy is slow when people think it is C and do loops\$\endgroup\$CommentedMay 20, 2019 at 5:28
  • \$\begingroup\$@200_success I have yearly NetCDF files with 365 bands. Please suggest how to include that information here for anyone to try and run my code.\$\endgroup\$
    – Ibe
    CommentedMay 20, 2019 at 8:41
  • \$\begingroup\$Use of coordinates from *.csv file instead of using arcpy.da.SearchCursor has reduced run time by 150%. Other bottleneck is arcpy.GetCellValue_management which is taking 95% of the processing time in each iteration. Please suggest any open source library to extract pixel values faster.\$\endgroup\$
    – Ibe
    CommentedMay 20, 2019 at 18:18

0

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.