Improving the speed of one-hot encoding a list of strings

Question

I've recently developed two functions to functions to essentially convert a list of strings that look something like this (these strings are 101 characters long in my case):

['AGT', 'AAT']

To a numpy array:

array([[[[1], [0], [0], [0]], [[0], [1], [0], [0]], [[0], [0], [0], [1]]], [[[1], [0], [0], [0]], [[1], [0], [0], [0]], [[0], [0], [1], [0]]]]) 

The shape of which is [2, 3, 4, 1] in this case

At the moment, my code essentially defines one function, in which I define a dictionary, which is then mapped to a single input string, like so:

def sequence_one_hot_encoder(seq): import numpy as np mapping = { "A": [[1], [0], [0], [0]], "G": [[0], [1], [0], [0]], "C": [[0], [0], [1], [0]], "T": [[0], [0], [0], [1]], "X": [[0], [0], [0], [0]], "N": [[1], [1], [1], [1]] } encoded_seq = np.array([mapping[i] for i in str(seq)]) return(encoded_seq) 

Following from this, I then create another function to map this function to my list of strings:

def sequence_list_encoder(sequence_file): import numpy as np one_hot_encoded_array = np.asarray(list(map(sequence_one_hot_encoder, sequence_file))) print(one_hot_encoded_array.shape) return(one_hot_encoded_array) 

At the moment, for a list containing 1,688,119 strings of 101 characters, it's taking around 7-8 minutes. I was curious if there was a better way of rewriting my two functions to reduce runtime?

Answer 1

sequence_one_hot_encoder(seq) builds an array of shape (len(seq), 4, 1). sequence_list_encoder() puts all these into a python list and then coverts the list into an array with shape (number_of_sequences, len(seq), 4, 1). It looks like there is a lot of overhead doing that. It is much faster to treat the one_hot_encoded_array as 1-D and then set the shape at the end.

def sequence_list_encoder(sequence_file): mapping = { "A": (1, 0, 0, 0), "G": (0, 1, 0, 0), "C": (0, 0, 1, 0), "T": (0, 0, 0, 1), "X": (0, 0, 0, 0), "N": (1, 1, 1, 1) } sequences = sequence_file.read().splitlines() bits = [b for seq in sequences for ch in seq for b in mapping[ch]] one_hot_encoded_array = np.fromiter(bits, dtype=np.uint8) one_hot_encoded_array.shape = (len(sequences), len(sequences[0]), 4, 1) return one_hot_encoded_array 

This runs in about 1/5 the time as your code.