Histogram output from parse of json-ish data in python

Question

The idea is exploratory data analysis, the output rendered as a histogram to get an idea of relative frequency of records:

The example data set looks like this:

{"blockNumber":"1941895","blockHash":"0x53464299a83cecc3e4d930b617c9518b8f74139265423d8110a919f5180bec79","hash":"0x0abe75e40a954d4d355e25e4498f3580e7d029769897d4187c323080a0be0fdd","from":"0x4586ffaf28e08b1613dd96ced9b57d52e8ad9d72","to":"0x91337a300e0361bddb2e377dd4e88ccb7796663d","gas":"21000","gasUsed":"21000","gasPrice":"20000000000","input":"","logs":[],"nonce":"1","value":"0x22c06103f88111000","timestamp":"2016-07-24 20:47:25 UTC"} {"blockNumber":"1941645","blockHash":"0x78804d09bb4e7126f53133e33e3548e0f04a691c01661ab9b719c3811e54355e","hash":"0x22c2b6490900b21d67ca56066e127fa57c0af973b5d166ca1a4bf52fcb6cf81c","from":"0x81bbf9f19ffe8368efe7611ccf5dcbdb4618b645","to":"0xb01a7866a244dbb600a7bbd170d43d4221838868","gas":"90000","gasUsed":"21000","gasPrice":"20000000000","input":"","logs":[],"nonce":"0","value":"0x4563918244f40000","timestamp":"2016-07-24 19:57:50 UTC"} {"blockNumber":"1941910","blockHash":"0xc7ba89fc0110a033c4bd03be4505014761141b956c228bc51ec49c15a4508ce4","hash":"0x8570106b0385caf729a17593326db1afe0d75e3f8c6daef25cd4a0499a873a6f","from":"0x91337a300e0361bddb2e377dd4e88ccb7796663d","to":"0x9fde2180b544b7690c35bdc66182eb843ac38030","gas":"90000","gasUsed":"21000","gasPrice":"20000000000","input":"","logs":[],"nonce":"6356","value":"0x41e92b66341ef0000","timestamp":"2016-07-24 20:50:12 UTC"} {"blockNumber":"1941919","blockHash":"0x4785c1b1a678cf7058e1fed3fc1c7d33c4326c2fb309f5fc75688f23d496b61c","hash":"0x8adfe7fc3cf0eb34bb56c59fa3dc4fdd3ec3f3514c0100fef800f065219b7707","from":"0x69ca903e87329fd63a3c7b2d3efde6a9bf3c3d45","to":"0xbfc39b6f805a9e40e77291aff27aee3c96915bdd","gas":"40000","gasUsed":"29130","gasPrice":"30000000000","input":"","logs":[{"address":"0xbfc39b6f805a9e40e77291aff27aee3c96915bdd","topics":["0x23919512b2162ddc59b67a65e3b03c419d4105366f7d4a632f5d3c3bee9b1cff"],"data":"AAAAAAAAAAAAAAAAwNMyg48U70L83hzyUYxCfdtnZyk="}],"nonce":"20","value":"0x1d2eb2accbaf90800","timestamp":"2016-07-24 20:52:08 UTC"} {"blockNumber":"1941922","blockHash":"0xd46dbf526f6d7c9197e841c8a4d7b2f4abdac4a62860cffabb943a46d07a86d4","hash":"0x8b0fe2b7727664a14406e7377732caed94315b026b37577e2d9d258253067553","from":"0x0b2c5cba2dc240e867f7721412c20e6016596d26","to":"0x9c83fe12c7575ea7350019e04253d3620957851f","gas":"21000","gasUsed":"21000","gasPrice":"21000000000","input":"","logs":[],"nonce":"2","value":"0x7ce66c50e2840000","timestamp":"2016-07-24 20:52:51 UTC"} {"blockNumber":"1941688","blockHash":"0x86bb1e90d0fa7be11d3f196057976383bb73cbd1596992e868155a576b5ddfb9","hash":"0x244b29b60c696f4ab07c36342344fe6116890f8056b4abc9f734f7a197c93341","from":"0x006cdc135b4e3a89d3ac1027ec3de609b8fff500","to":"0x58ae42a38d6b33a1e31492b60465fa80da595755","gas":"50000","gasUsed":"50000","gasPrice":"20000000000","input":"","logs":[],"nonce":"47","value":"0xc7140013deaf40","error":"invalid jump destination (PUSH1) 2","timestamp":"2016-07-24 20:06:38 UTC"} {"blockNumber":"1941794","blockHash":"0x41ee74e34cbf9ef4116febea958dbc260e2da3a6bf6f601bfaeb2cd9ab944a29","hash":"0xf2b5b8fb173e371cbb427625b0339f6023f8b4ec3701b7a5c691fa9cef9daf63","from":"0x3c0cbb196e3847d40cb4d77d7dd3b386222998d9","to":"0x2ba24c66cbff0bda0e3053ea07325479b3ed1393","gas":"121000","gasUsed":"21000","gasPrice":"20000000000","input":"","logs":[],"nonce":"14","value":"0x24406420d09ce7440000","timestamp":"2016-07-24 20:28:11 UTC"} {"blockNumber":"1941716","blockHash":"0x75e1602cad967a781f4a2ea9e19c97405fe1acaa8b9ad333fb7288d98f7b49e3","hash":"0xf8f2a397b0f7bb1ff212b6bcc57e4a56ce3e27eb9f5839fef3e193c0252fab26","from":"0xa0480c6f402b036e33e46f993d9c7b93913e7461","to":"0xb2ea1f1f997365d1036dd6f00c51b361e9a3f351","gas":"121000","gasUsed":"21000","gasPrice":"20000000000","input":"","logs":[],"nonce":"1","value":"0xde0b6b3a7640000","timestamp":"2016-07-24 20:12:17 UTC"} {"blockNumber":"1941794","blockHash":"0x41ee74e34cbf9ef4116febea958dbc260e2da3a6bf6f601bfaeb2cd9ab944a29","hash":"0xf275b8fb173e371cbb427625b0339f6023f8b4ec3701b7a5c691fa9cef9daf63","from":"0x3c0cbb196e3847d40cb4d77d7dd3b386222998d9","to":"0x2ba24c66cbff0bda0e3053ea07325479b3ed1393","gas":"121000","gasUsed":"21000","gasPrice":"20000000000","input":"","logs":[],"nonce":"14","value":"0x24406420d09ce7440000","timestamp":"2016-07-24 20:28:11 UTC"} {"blockNumber":"1941794","blockHash":"0x41ee74e34cbf9ef4116febea958dbc260e2da3a6bf6f601bfaeb2cd9ab944a29","hash":"0xf285b8fb173e371cbb427625b0339f6023f8b4ec3701b7a5c691fa9cef9daf63","from":"0x3c0cbb196e3847d40cb4d77d7dd3b386222998d9","to":"0x2ba24c66cbff0bda0e3053ea07325479b3ed1393","gas":"121000","gasUsed":"21000","gasPrice":"20000000000","input":"","logs":[],"nonce":"14","value":"0x24406420d09ce7440000","timestamp":"2016-07-24 20:28:11 UTC"} {"blockNumber":"1941895","blockHash":"0x53464299a83cecc3e4d930b617c9518b8f74139265423d8110a919f5180bec79","hash":"0x0abg75e40a954d4d355e25e4498f3580e7d029769897d4187c323080a0be0fdd","from":"0x4586ffaf28e08b1613dd96ced9b57d52e8ad9d72","to":"0x91337a300e0361bddb2e377dd4e88ccb7796663d","gas":"21000","gasUsed":"21000","gasPrice":"20000000000","input":"","logs":[],"nonce":"1","value":"0x22c06103f88111000","timestamp":"2016-07-24 20:47:25 UTC"} 

The code is here:

#data processing import re import pprint #sorting import operator from collections import Counter #visualization rendering import matplotlib.pyplot as plt import numpy as np from operator import itemgetter # read in the data data = open('toy.json', 'r') new_dict = {} for line in data: identifier = re.search('(\"hash\"\:\s?\"(\w+)\")', line) if identifier: found_identifier = identifier.group(2) # print(found) gas = re.search('(\"gas\"\:\s?\"(\w+)\")', line) if gas: found_gas = gas.group(2) # print(found) new_dict.update({found_identifier:found_gas}) # make it into tuple sorted_x = sorted(new_dict.items(), key=operator.itemgetter(1)) # prepare for counter # flat_list = [x[1] for x in sorted_x if int(x[1]) > 1] flat_list = [x[1] for x in sorted_x] # count 'em up flat_list = Counter(flat_list) # prune threshold to_remove = set() for key, value in flat_list.viewitems(): if value < 2: to_remove.add(key) for key in to_remove: del flat_list[key] pprint.pprint(flat_list) # visualization c = Counter(flat_list).items() c.sort(key=itemgetter(1)) labels, values = zip(*c) indexes = np.arange(len(labels)) width = 1 plt.bar(indexes, values, width) plt.xticks(indexes + width * 0.5, labels) plt.show() 

Answer 1

Python has a built-in json module, so you should use that instead of parsing the lines by yourself, no need for regex here.

You can just iterate over the lines of the file (which seem to contain a valid JSON object each), and build a list from that:

import json from collections import Counter import matplotlib.pyplot as plt import numpy as np def at_least(c, threshold): """Return a Counter of values which are at_least (>=) threshold""" return Counter(el for el in c.elements() if c[el] >= threshold) def draw_hist(labels, values): indices = np.arange(len(labels)) width = 1 plt.bar(indices, values, width) plt.xticks(indices + width * 0.5, labels) plt.show() if __name__ == "__main__": with open("toy.json") as f: data = [json.loads(line) for line in f] gas = {line["hash"]: int(line["gas"]) for line in data} gas_hist_values = at_least(Counter(gas.values()), 2) labels, values = zip(*reversed(gas_hist_values.most_common())) draw_hist(labels, values) 

Note that if a hash appears twice, this code, just like your original code, overwrites it with the value that appears last.

I took out a lot of unnecessary sorting and calls of Counter. Instead I used the method Counter.most_common, which works like dict.items, except that it returns the items in order of decreasing counts. Then you just need to reverse them to get them in increasing order.

I also made your code more re-usable by encapsulating some of the functionality in functions and guarding the actual code with a if __name__ == "__main__" guard.

I also gave your variables clearer names.

---

Alternatively, since you are already using numpy and matplotlib, it might be worth it to take a look at pandas. It has a read_json method, which needs only a little preparation (you need to read the whole file at once in a string, though...):

import pandas as pd import numpy as np import matplotlib.pyplot as plt with open("toy.json") as f: df = pd.read_json("[" + ",".join(f) + "]", orient='records') values, labels = np.histogram(df["gas"]) above_threshold = np.where(values >= 2) plt.bar(labels[above_threshold], values[above_threshold], 1000) 

While this is way easier, the x-axis is now the actual numerical values, instead of different labels. But for this you could use your plotting again as well. Note that they are not sorted either, but it is usually easier to understand if numerical are in increasing order, instead of in order of increasing counts...