My function gets the Cartesian product of a list of lists of tuples. The function is correctly producing the list of lists, but it's a memory hog (and leads to a MemoryError for large data sets, with the mapping/casting line probably the biggest culprit) and I have very little experience optimizing code with memory in mind. The function is given here:
def make_ratio_groups(self): list_of_lists_of_tuples = [] for part in whole: list_of_lists_of_tuples.append(part.list_of_tuples) all_groups = itertools.product(*list_of_lists_of_tuples) # Mapping and casting to list is necessary to put in the correct format for a subsequent function all_groups_list = list(map(list, all_groups)) return all_groups_list Is there a more memory-efficient way of doing this? Thanks in advance for your time and patience.
Edit: It's definitely the mapping and casting that's causing the problem. I created a generator that handled the Cartesian product calculations and didn't see a big improvement in performance, so I tried profiling and found that the mapping/casting is definitely the issue.
whole? — Also, definitely look into generators (i.e.yieldinstead ofreturn), so that you don't have to keep everything in memory at once in the first place.\$\endgroup\$partandwholeare complicated objects, but they have nothing to do with the code given here aside from demonstrating that the tuples in question are grabbed from a series ofpartobjects. That's not a memory-intensive chunk of code. The only two pertinent sections IMO are the Cartesian product calculations and the subsequent mapping and casting to list. The informal profiling I've done supports that. I've been able to make a generator to replaceproduct, but am not sure how to deal with map/cast.\$\endgroup\$