Improving the speed of creation for three Perlin Noise Maps in Python?

Question

I am interested in learning how I can improve the speed of the code in this pygame file. I iterate over 6400 * 1800 * 3 or 34,560,000 elements of various numpy arrays here to apply noise values to them. The noise library I'm using can be found on GitHub here.

I am calling static variables from a class called ST here. ST.MAP_WIDTH = 6400 and ST.MAP_HEIGHT = 1800. All other ST attributes called here are assigned in the code. They are the noise-maps I'm making.

from __future__ import division from singleton import ST import numpy as np import noise import timeit import random import math def __noise(noise_x, noise_y, octaves=1, persistence=0.5, lacunarity=2.0): """ Generates and returns a noise value. :param noise_x: The noise value of x :param noise_y: The noise value of y :return: numpy.float32 """ value = noise.pnoise2(noise_x, noise_y, octaves, persistence, lacunarity, random.randint(1, 9999)) return np.float32(value) def __elevation_mapper(noise_x, noise_y): """ Finds and returns the elevation noise for the given noise_x and noise_y parameters. :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer :return: float """ return __noise(noise_x, noise_y, 8, 0.9) def __climate_mapper(y, noise_x, noise_y): """ Finds and returns the climate noise for the given noise_x and noise_y parameters. :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer :return: float """ # find distance from bottom of map and normalize to range [0, 1] distance = math.sqrt((y - (ST.MAP_HEIGHT >> 1))**2) / ST.MAP_HEIGHT value = __noise(noise_x, noise_y, 8, 0.7) return (1 + value - distance) / 2 def __rainfall_mapper(noise_x, noise_y): """ Finds and returns the rainfall noise for the given noise_x and noise_y parameters. :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer :return: float """ return __noise(noise_x, noise_y, 4, 0.65, 2.5) def create_map_arr(): """ This function creates the elevation, climate, and rainfall noise maps, normalizes them to the range [0, 1], and then assigns them to their appropriate attributes in the singleton ST. """ start = timeit.default_timer() elevation_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32) climate_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32) rainfall_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32) randomizer = random.uniform(0.0001, 0.9999) # assign noise map values for y in range(ST.MAP_HEIGHT): for x in range(ST.MAP_WIDTH): noise_x = x / ST.MAP_WIDTH - randomizer noise_y = y / ST.MAP_HEIGHT - randomizer elevation_arr[y][x] = __elevation_mapper(noise_x, noise_y) climate_arr[y][x] = __climate_mapper(y, noise_x, noise_y) rainfall_arr[y][x] = __rainfall_mapper(noise_x, noise_y) # normalize to range [0, 1] and assign to relevant ST attributes ST.ELEVATIONS = (elevation_arr - elevation_arr.min()) / \ (elevation_arr.max() - elevation_arr.min()) ST.CLIMATES = (climate_arr - climate_arr.min()) / \ (climate_arr.max() - climate_arr.min()) ST.RAINFALLS = (rainfall_arr - rainfall_arr.min()) / \ (rainfall_arr.max() - rainfall_arr.min()) stop = timeit.default_timer() print("GENERATION TIME: " + str(stop - start)) 

Answer 1

Losing your Loops

Austin Hastings' comment gives you a good hint where to look at. The main takeaway for you should be:

(Most) loops are damn slow in Python. Especially multiple nested loops.

NumPy can help to vectorize your code, i.e. in this case that more of the looping is done in the C backend instead of in the Python interpreter. I would highly recommend to have a listen to the talk Losing your Loops: Fast Numerical Computing with NumPy by Jake VanderPlas. Although primarily tailored towards data science, it gives a good overview on the topic.

I did some slight modifications to your original script to include some of the vectorization ideas while still using your chosen Perlin noise library. (Sidenote: I changed the __ prefix to a single _, because that is the convention most Python programmers use for internal functions. See PEP8 style guide.)

# -*- coding: utf-8 -*- from __future__ import division, print_function import numpy as np import noise import timeit class ST(object): MAP_HEIGHT = 1800 MAP_WIDTH = 6400 def _noise(noise_x, noise_y, octaves=1, persistence=0.5, lacunarity=2.0): """ Generates and returns a noise value. :param noise_x: The noise value of x :param noise_y: The noise value of y :return: numpy.float32 """ if isinstance(noise_x, np.ndarray): #rand_seed = np.random.randint(1, 9999, noise_x.size) rand_seed = np.ones((noise_x.size, )) # just for comparison value = np.array([noise.pnoise2(x, y, octaves, persistence, lacunarity, r) for x, y, r in zip(noise_x.flat, noise_y.flat, rand_seed)]) return value.reshape(noise_x.shape) else: value = noise.pnoise2(noise_x, noise_y, octaves, persistence, lacunarity, 1.0) # just for comparison #np.random.randint(1, 9999)) return np.float32(value) def _elevation_mapper(noise_x, noise_y): """ Finds and returns the elevation noise for the given noise_x and noise_y parameters. :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer :return: float """ return _noise(noise_x, noise_y, 8, 0.9) def _climate_mapper(y, noise_x, noise_y): """ Finds and returns the climate noise for the given noise_x and noise_y parameters. :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer :return: float """ # find distance from bottom of map and normalize to range [0, 1] distance = np.sqrt((y - (ST.MAP_HEIGHT >> 1))**2) / ST.MAP_HEIGHT value = _noise(noise_x, noise_y, 8, 0.7) return (1.0 + value - distance) / 2.0 def _rainfall_mapper(noise_x, noise_y): """ Finds and returns the rainfall noise for the given noise_x and noise_y parameters. :param noise_x: noise_x = x / ST.MAP_WIDTH - randomizer :param noise_y: noise_y = y / ST.MAP_HEIGHT - randomizer :return: float """ return _noise(noise_x, noise_y, 4, 0.65, 2.5) def create_map_arr(): """ This function creates the elevation, climate, and rainfall noise maps, normalizes them to the range [0, 1], and then assigns them to their appropriate attributes in the singleton ST. """ # assign noise map values randomizer = np.random.uniform(0.0001, 0.9999) start_arr = timeit.default_timer() X, Y = np.mgrid[0:ST.MAP_WIDTH, 0:ST.MAP_HEIGHT] noise_x = X / ST.MAP_WIDTH - randomizer noise_y = Y / ST.MAP_HEIGHT - randomizer elevation_arr_np = _elevation_mapper(noise_x, noise_y) climate_arr_np = _climate_mapper(Y, noise_x, noise_y) rainfall_arr_np = _rainfall_mapper(noise_x, noise_y) duration_arr = timeit.default_timer() - start_arr start_loop = timeit.default_timer() elevation_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32) climate_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32) rainfall_arr = np.zeros([ST.MAP_HEIGHT, ST.MAP_WIDTH], np.float32) for y in range(ST.MAP_HEIGHT): for x in range(ST.MAP_WIDTH): noise_x = x / ST.MAP_WIDTH - randomizer noise_y = y / ST.MAP_HEIGHT - randomizer elevation_arr[y, x] = _elevation_mapper(noise_x, noise_y) climate_arr[y, x] = _climate_mapper(y, noise_x, noise_y) rainfall_arr[y, x] = _rainfall_mapper(noise_x, noise_y) duration_loop = timeit.default_timer() - start_loop print(np.allclose(elevation_arr, elevation_arr_np.T)) print(np.allclose(climate_arr, climate_arr_np.T)) print(np.allclose(rainfall_arr, rainfall_arr_np.T)) print("GENERATION TIME: loop: {:.6f}, array: {:.6f}".format(duration_loop, duration_arr)) if __name__ == "__main__": create_map_arr() 

The bottleneck is still in

value = np.array([noise.pnoise2(x, y, octaves, persistence, lacunarity, r) for x, y, r in zip(noise_x.flat, noise_y.flat, rand_seed)]) 

and it would be highly favorable to use an implementation which supports 2D input, preferably from NumPy, directly (see further reading below).

Nevertheless, the modifications bring the execution time down to a third of the original time on my machine (which is not that powerful):

True True True GENERATION TIME: loop: 338.094228, array: 101.549388 

Those three Trues are from a little test I added to check if the generated maps are the same within reasonable accuracy. For this purpose the additional random value in _noise was disabled.

Further reading

There have also already been similar questions on Code Review (see, e.g. here), where a reviewer created a Perlin noise implementation purely in Numpy. There also seems to be a GitHub project also doing Perlin noise with Numpy. So maybe have a look at them if your not forced to stick with noise.