Adding string to a text file without duplicates

Question

I have a function below, store_in_matchid_file, and it adds certain strings to a .txt file as long as that string doesn't already exist in the file. However that file is getting to be millions of lines long and the process of checking is becoming too long.

I was hoping someone would be able to indicate a way I could speed up the process, by changing how I've coded the process.

def store_in_matchid_file(distinct_matchids_func): # Ids stored in the text file with open('MatchIds.txt') as f: current_id_list = f.read().splitlines() # Ids recently collected (Previously a set, converted to list to iterate over) distinct_matchids_list = list(distinct_matchids_func) # Adding Ids that don't exist in the file with open('MatchIds.txt', 'a') as file: for match in distinct_matchids_list: if match not in current_id_list: file.write(match + '\n') 

Answer 1

You could make some improvements:

• Open the file just once in read+update mode.
• Instead of current_id_list, use a set, which can test whether an item is one of the existing items quickly, regardless of how many existing items there are.
• Iterating over the file handle f will give you entries a line at a time; you don't need to read the entire file at once and call .splitlines() on it.
• You don't need to explicitly convert distinct_matchids_func into a list; you can just iterate over it.

Some other remarks:

• distinct_matchids_func is a weird name — is it a function or an iterable?
• We're assuming that the distinct_matchids_func are indeed distinct, and don't contain duplicates.
• You should write a docstring for the function rather than a comment.

def store_in_matchid_file(distinct_matchids_func): """ Append items from distinct_matchids_func as lines to MatchIds.txt that are not already present in the file. """ with open('MatchIds.txt', 'r+') as f: current_ids = set(f) for match in distinct_matchids_func: match += '\n' if match not in current_ids: f.write(match) 

But as @Reinderien suggests in a comment, storing these entries in a database such as sqlite could be a better option. It's going to perform better (since it uses data structures to facilitate quick searches), and it will be less prone to corruption (for example, if multiple processes try to manipulate the file at the same time).