How to change all the dictionary keys in a for loop with d.items()?

Question

I would like some help with understanding why this code is not working as expected.

If one wants to change the key of a dictionary but keep the value, he/she might use:

d[new_key] = d.pop[old_key] 

I want to modify all the keys (and keep the values in place) but the code below skips certain lines - ("col2") remains untouched. Is it because dictionaries are unordered and I keep changing the values in it?

How would I go about changing the keys and keep the values without creating a new dictionary?

import time import pprint name_dict = {"col1": 973, "col2": "1452 29th Street", "col3": "Here is a value", "col4" : "Here is another value", "col5" : "NULL", "col6": "Scottsdale", "col7": "N/A", "col8" : "41.5946922", "col9": "Building", "col10" : "Commercial"} for k, v in name_dict.items(): print("This is the key: '%s' and this is the value '%s'\n" % (k, v) ) new_key = input("Please enter a new key: ") name_dict[new_key] = name_dict.pop(k) time.sleep(4) pprint.pprint(name_dict) 

Answer 1

It's never a good idea to change the object you're iterating over. Normally dict even throws an exception when you attempt it:

name_dict = {1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6} for k, v in name_dict.items(): name_dict.pop(k) 

RuntimeError: dictionary changed size during iteration

However in your case you add one item for every removed item. That makes it more convolved. To understand what's happening you need to know that a dictionary is somewhat like a sparse table. For example a dictionary like {1: 1, 3: 3, 5: 5} could look like this (this changed in Python 3.6, for 3.6 and newer the following isn't correct anymore):

hash key value - - - 1 1 1 - - - 3 3 3 - - - 5 5 5 - - - - - - - - - 

That's also the order in which it is iterated. So in the first iteration it will go to the second item (where the 1: 1 is stored). Let's assume you change the key to 2 and remove the key 1 the dict would look like this:

hash key value - - - - - - 2 2 1 3 3 3 - - - 5 5 5 - - - - - - - - - 

But we're still at the second line, so the next iteration it will go to the next "not-empty" entry which is 2: 1. Oups ...

It's even more complicated with strings as keys because string hashes are randomized (on a per session basis) so the order inside the dictionary is unpredictable.

In 3.6 the internal layout was changed a bit but something similar happens here.

Assuming you have this loop:

name_dict = {1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 6} for k, v in name_dict.items(): # print(k, k+6, name_dict.__sizeof__()) name_dict[k+6] = name_dict.pop(k) # print(name_dict) 

The initial layout is like this:

key value 1 1 2 2 3 3 4 4 5 5 6 1 

The first loop removes 1 but adds 7. Because dictionaries are ordered in 3.6 this inserts a placeholder where 1 had been:

key value - - 2 2 3 3 4 4 5 5 6 1 7 2 

This goes on until you replace 4 with 10.

key value - - - - - - - - 5 5 6 1 7 2 8 3 9 4 10 5 

But when you replace 5 with 11 the dictionary will need to increase it's size. Then something special happens: The placeholders are removed:

key value 6 6 7 1 8 2 9 3 10 4 11 5 

So, we were at position 5 in the last iteration and now we change line 6. But line 6 contains 11: 5 right now. Oups...

Never change the object you're iterating over: Don't mess with the keys during iteration (values are okay)!

You could instead keep a "translation table" (don't know if that violates your "without creating a new dict" requirement but you need some kind of storage to make your code work correctly) and do the renaming after the loop:

translate = {} for k, v in name_dict.items(): print("This is the key: '%s' and this is the value '%s'\n" % (k, v) ) new_key = input("Please enter a new key: ") translate[k] = new_key time.sleep(4) for old, new in translate.items(): name_dict[new] = name_dict.pop(old) 

Answer 2

in python3 dict.items() is just a view on the dict. as you are not allowed to modify an iterable while iterating you are not allowed to modify a dict while iterating over dict.items(). you have to copy items() to a list before iterating

for k, v in list(name_dict.items()): ... name_dict[new_key] = name_dict.pop(k) 

this does fulfill your 'no new dict' requirement, although the list holds in fact a full copy of all your data.

you could relax the memory footprint a little by copying just the keys

for k in list(name_dict): v = name_dict.pop(k) ... name_dict[new_key] = v 

---

EDIT: credits to Sven Krüger, he raised the possibility of a old-key-new-key collision problem. in that case you have to go for

kv = list(name_dict.items()) name_dict.clear() for k, v in kv : ... name_dict[new_key] = v 

by the way, there is a use case for not creating a new dict, the current one might be referenced somwhere else.

Answer 3

In order to have a iterable object in your working memory which is not depending on your original dictionary you can use the method fromkeys. It is possible to now assign new keys with your old values. But there is one thing you have to keep in mind: You cannot assign a value to a new key that is not the certain old key while the new key is also another key from the old set of keys.

Old_Keys = { old_key_1, old_key_2, ..., old_key_n } 

So you assign the value related to the old key to the new key.

old_key_1 -> new_key_1 not in Old_Keys # Okay! old_key_2 -> new_key_2 == old_key_4 # Boom!... Error!... 

Be aware of this when you use the following!

CODE

D = {'key1': 'val1', 'key2': 'val2', 'key3': 'val3'} for key in D.fromkeys(D) : new_key = raw_input("Old Key: %s, New Key: " % key) D[new_key] = D.pop(key) print D 

CONSOLE

Old Key: key1, New Key: abc Old Key: key2, New Key: def Old Key: key3, New Key: ghi {"abc": 'val1', "def": 'val2', "ghi": 'val3'}