Blocking production lines in Python

Question

I have the following code that works great, but takes 40 mins to 1 hours to finish the task. It has at least 30 rules for blocking lines, and 30k+ orders to check. I'm new to Python (3 months working with it), so I don't know what can I change to make it run faster. The version in my computer is Python 2.7.9.

import codecs import re def find_word(text, search): result = re.findall('\\b'+search+'\\b', text, flags=re.IGNORECASE) if len(result)>0: return True else: return False with codecs.open('C:/inetpub/wwwroot/BloqueoLinea/DATA/FO.txt','r',encoding = 'utf-8') as FO: finalorder = FO.readlines() with codecs.open('C:/inetpub/wwwroot/BloqueoLinea/DATA/Table-Familias.txt','r',encoding = 'utf-8') as familias: list_familias = familias.readlines() fo = '' with codecs.open('C:/inetpub/wwwroot/BloqueoLinea/DATA/data.txt','w',encoding = 'utf-8') as data: for forder in finalorder: order = forder.split(';') orden = [order[0],order[1],order[15],order[24],order[2],'',''] for familia in list_familias: fam = familia.split(';') if order[2] in fam[0]: orden[5] = fam[1] orden[6] = fam[2] fo = ';'.join(orden) data.write(fo+'\n') with codecs.open('C:/inetpub/wwwroot/BloqueoLinea/DATA/data.txt','r',encoding = 'utf-8') as cross: cr = cross.readlines() with codecs.open('C:/inetpub/wwwroot/BloqueoLinea/App_Data/BloqLines.txt','r',encoding = 'utf-8') as bloqueos: block = bloqueos.readlines() with codecs.open('C:/inetpub/wwwroot/BloqueoLinea/DATA/nodos.txt','r',encoding = 'utf-8') as node: nodos = node.readlines() with codecs.open('C:/inetpub/wwwroot/BloqueoLinea/DATA/Table-Lineas.txt','r',encoding = 'utf-8') as listlineas: tlines = listlineas.readlines() count = 0 fileblock = codecs.open('C:/inetpub/wwwroot/BloqueoLinea/DATA/Restricciones_Ruteo.txt','w',encoding = 'utf-8') for forder in cr: fo = forder.split(';') for bloqueo in block: check = bloqueo.split(';') if 'True' in check[0]: #Primero comprobar si la regla se aplica o no fam87 = check[5].lower() fam87 = fam87.capitalize() est = check[6].lower() est = est.capitalize() #Logica taxativo clinea = 0 lista = [] ap = lista.append for line in tlines: if check[7] in line: nline = line.split(';') lin = nline[1] for line in tlines: nline = line.split(';') if lin in nline[1]: if line in lista: pass else: nline = line.split(';') ap(nline[0]) #Fam87 if forder.find(check[5]) >= 1 or (forder.find(fam87)) >= 1: if forder.find(check[6]) >= 1 or (forder.find(est)) >= 1: #Si EstCom coincide con bloqueo actual if fo[1] in check[2]: #Si ITEMREP coincide bloqueo actual for nodo in nodos: nod = nodo.split(';') if check[2] in nodo: #Si parametro de ITEMREP coincide en nodo actual if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Revisar si parametro linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Revisar si parametro linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 elif check[2] == '': #Cuando no hay ITEMREP for nodo in nodos: if fo[1] in nodo: nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 #No hay EstCom, pero hay ITEMREP elif fo[1] in check[2] and (check[6] == ''): #Si ITEMREP coincide con bloqueo actual for nodo in nodos: if check[2] in nodo: #Si parametro de ITEMREP coincide en nodo actual nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 elif check[2] == '' and check[6] == '': #Cuando no hay ITEMREP ni EstCom for nodo in nodos: if fo[1] in nodo: nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 #Familia N1 elif fo[5] in check[3] and (check[5] == ''): if fo[2] in check[6]: #EstCom if fo[1] in check[2]: #ITEMREP for nodo in nodos: if check[2] in nodo: #ITEMREP nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 elif check[2] == '': #No hay ITEMREP for nodo in nodos: if fo[1] in nodo: nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 #Cuando no hay EstCom, pero hay ITEMREP elif fo[1] in check[2] and (check[6] == ''): #Si ITEMREP coincide con parametro de bloqueo for nodo in nodos: if check[2] in nodo: #Si ITEMREP coincide en nodo actual nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 elif check[2] == '' and check[6] == '': #No hay ITEMREP ni EstCom for nodo in nodos: if fo[1] in nodo: nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 #Familia N2 elif fo[6] in check[4] and (check[3] == '') and (check[5] == ''): if fo[2] in check[6]: #EstCom if fo[1] in check[2]: #ITEMREP for nodo in nodos: if check[2] in nodo: #ITEMREP nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 elif check[2] == '': #Cuando no hay ITEMREP for nodo in nodos: if fo[1] in nodo: nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 #Cuando no hay EstCom, pero hay ITEMREP elif fo[1] in check[2] and (check[6] == ''): #Si ITEMREP coincide con parametro de bloqueo for nodo in nodos: if check[2] in nodo: #Si ITEMREP coincide en nodo actual nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 elif check[2] == '' and check[6] == '': #No hay ITEMREP for nodo in nodos: if fo[1] in nodo: nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 #Cuando no hay parametro para ninguna Familia, se sigue a bloquear EstCom elif fo[2] in check[6] and (check[4] == '') and (check[3] == '') and (check[5] == ''): #EstCom if fo[1] in check[2]: #ITEMREP for nodo in nodos: if check[2] in nodo: #ITEMREP nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 #Si no existe parametro de ITEMREP elif check[2] == '': for nodo in nodos: if fo[1] in nodo: nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 #Cuando no hay Familias, y no hay EstCom, bloquear por ITEMREP elif fo[1] in check[2] and (check[6] == '') and (check[4] == '') and (check[3] == '') and (check[5] == ''): for nodo in nodos: if fo[1] in nodo: nod = nodo.split(';') if bool(check[9]): for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Linea coincide en nodo actual count += 1 if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 print ('Finished') fileblock.close() 

At first I tried making some functions, but it started running slower, so I removed those functions and placed them in the code.

Basically, the code reads many text files. Some are static, but the FO.txt, nodos.txt and BloqLines.txt are different every day. When implemented, this code is supposed to run every morning.

It checks the rules (in BloqLines.txt) checks the orders (FO.txt gets some changes for data that it's needed and creates data.txt) and after checking with rules applies or not, it creates the Restriccion_Ruteos.txt and that's it.

If possible, how can I improve the performance of this code?

---

Samples from input files. Omitting the original files that are edited in the code.

BloqLines.txt :

True;Restringe GLV IREP4801 por G3U;IREP4801;;;;;G4_UNI;;False True;Restringe Angostos ME por el MC2;;;;ang;Expo;MC2_GUE;;False True;Restringe Anneal para CGL;;;ANN;;;G2_JUV;;False 

data.txt :

Orden00005990;IREP1554;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00005991;IREP1554;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00005992;IREP1554;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00005993;IREP1554;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00005994;IREP1554;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00005995;IREP1554;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00005996;IREP1554;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00005997;IREP1554;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00005998;IREP1555;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00005999;IREP1555;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00006000;IREP1555;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00006001;IREP1555;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD Orden00006002;IREP1555;Red Comercial;1;B_RCD_sSKP_Rto;LAC;RCD 

Table-Lineas.txt:

ING_MAQ;INGMAQ;0;INGMAQ;ING_MAQ COMP_SLB_TTC;COMP;0;COMP;COMP_SLB_TTC MC1_GUE;MC;1;GUE;MC1_GUE MC2_GUE;MC;1;GUE;MC2_GUE MC3_CHU;MC;1;CHU;MC3_CHU PREPA_GUE;PREPA;2;GUE;PREPA_GUE SKP_GUE;SKP;2;GUE;SKP_GUE SKP3_CHU;SKP;2;CHU;SKP3_CHU SKP4_CHU;SKP;2;CHU;SKP4_CHU DEC_PSQ;DEC;3;PSQ;DEC_PSQ DEC_UNI;DEC;3;UNI;DEC_UNI DEC1_CHU;DEC;3;CHU;DEC1_CHU DEC2_CHU;DEC;3;CHU;DEC2_CHU 

nodos.txt:

IREP1554Ruta_1_CUST_1;IREP1554;TEIREP1554_CUST_1;CUST;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1554Ruta_1_DEC3_CHU_1;IREP1554;TEIREP1554_DEC3_CHU_1;DEC3_CHU;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1554Ruta_1_ING_PT_1;IREP1554;TEIREP1554_ING_PT_1;ING_PT;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1554Ruta_1_MC3_CHU_1;IREP1554;TEIREP1554_MC3_CHU_1;MC3_CHU;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1554Ruta_1_NULL_1;IREP1554;NULL;NULL;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1554Ruta_1_PT_1;IREP1554;TEIREP1554_PT_1;PT;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1555Ruta_1_CUST_1;IREP1555;TEIREP1555_CUST_1;CUST;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1555Ruta_1_DEC3_CHU_1;IREP1555;TEIREP1555_DEC3_CHU_1;DEC3_CHU;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1555Ruta_1_ING_PT_1;IREP1555;TEIREP1555_ING_PT_1;ING_PT;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1555Ruta_1_MC3_CHU_1;IREP1555;TEIREP1555_MC3_CHU_1;MC3_CHU;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1555Ruta_1_NULL_1;IREP1555;NULL;NULL;B_RCD_sSKP_Rto;servicio;propaga;codproceso IREP1555Ruta_1_PT_1;IREP1555;TEIREP1555_PT_1;PT;B_RCD_sSKP_Rto;servicio;propaga;codproceso 

FO.txt:

Orden00005990;IREP1554;B_RCD_sSKP_Rto;Mas;44.835;44.835;44.835;0;0;20151130;20151130;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2015-11;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Mas;44.835;Comerciales;1;IREP1554;2015-11-23 12:43:39;0203;11 Orden00005991;IREP1554;B_RCD_sSKP_Rto;Mas;123.69;123.69;123.69;0;0;20151231;20151231;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2015-12;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Mas;123.69;Comerciales;1;IREP1554;2015-11-23 12:43:39;0203;12 Orden00005992;IREP1554;B_RCD_sSKP_Rto;Cus;40.296;40.296;40.296;0;0;20160131;20160131;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2016-01;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Cus;40.296;Comerciales;1;IREP1554;2015-11-23 12:43:39;0203;1 Orden00005993;IREP1554;B_RCD_sSKP_Rto;Mas;89.04;89.04;89.04;0;0;20160131;20160131;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2016-01;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Mas;89.04;Comerciales;1;IREP1554;2015-11-23 12:43:39;0203;1 Orden00005994;IREP1554;B_RCD_sSKP_Rto;Cus;90.187;90.187;90.187;0;0;20160229;20160229;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2016-02;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Cus;90.187;Comerciales;1;IREP1554;2015-11-23 12:43:39;0203;2 Orden00005995;IREP1554;B_RCD_sSKP_Rto;Mas;89.775;89.775;89.775;0;0;20160229;20160229;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2016-02;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Mas;89.775;Comerciales;1;IREP1554;2015-11-23 12:43:39;0203;2 Orden00005996;IREP1554;B_RCD_sSKP_Rto;Cus;97.637;97.637;97.637;0;0;20160331;20160331;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2016-03;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Cus;97.637;Comerciales;1;IREP1554;2015-11-23 12:43:39;0203;3 Orden00005997;IREP1554;B_RCD_sSKP_Rto;Mas;99.435;99.435;99.435;0;0;20160331;20160331;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2016-03;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Mas;99.435;Comerciales;1;IREP1554;2015-11-23 12:43:39;0203;3 Orden00005998;IREP1555;B_RCD_sSKP_Rto;Mas;42;42;42;0;0;20151130;20151130;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2015-11;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Mas;42;Comerciales;1;IREP1555;2015-11-23 12:43:39;0203;11 Orden00005999;IREP1555;B_RCD_sSKP_Rto;Mas;21;21;21;0;0;20151231;20151231;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2015-12;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Mas;21;Comerciales;1;IREP1555;2015-11-23 12:43:39;0203;12 Orden00006000;IREP1555;B_RCD_sSKP_Rto;Mas;42;42;42;0;0;20160131;20160131;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2016-01;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Mas;42;Comerciales;1;IREP1555;2015-11-23 12:43:39;0203;1 Orden00006001;IREP1555;B_RCD_sSKP_Rto;Cus;21.559;21.559;21.559;0;0;20160229;20160229;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2016-02;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Cus;21.559;Comerciales;1;IREP1555;2015-11-23 12:43:39;0203;2 Orden00006002;IREP1555;B_RCD_sSKP_Rto;Mas;42;42;42;0;0;20160229;20160229;30;VirtualRep;Primeras;Rep;Red Comercial;1-Decapado;2016-02;000001;482;Committed;B_RCD_sSKP_Rto_Red_Rep_Mas;42;Comerciales;1;IREP1555;2015-11-23 12:43:39;0203;2 

Table-Familias.txt:

B_RCD_SKP_Ang;LAC;RCD;ANG;RCD SKP Angosto;B_RCD_SKP_Ang B_RCD_SKP_Rto;LAC;RCD;RTO;RCD SKP Resto;B_RCD_SKP_Rto B_RCD_sSKP_Ang;LAC;RCD;ANG;RCD sSKP Angosto;B_RCD_sSKP_Ang B_RCD_sSKP_Rto;LAC;RCD;RTO;RCD sSKP Resto;B_RCD_sSKP_Rto B_RCD_TEM_Ang;LAC;RCD;RCD TEM;RCD Temple Angosto;B_RCD_TEM_Ang B_RCD_TEM_Rto;LAC;RCD;RCD TEM;RCD Temple Resto;B_RCD_TEM_Rto C_RFC_LAV;LAF;RFC;RFC;RFC LAV;C_RFC_LAV C_RFC_MAQ;LAF;RFC;RFC;RFC MAQ;C_RFC_MAQ C_RFC_MAQ_TNS;LAF;RFC;RFC;RFC MAQ TNS;C_RFC_MAQ_TNS C_RFC_sLAV;LAF;RFC;RFC;RFC sLAV;C_RFC_sLAV D_RFR_Anc_sTNS;LAF;RFR;ANCHO;RFR Ancho sTNS;D_RFR_Anc_sTNS D_RFR_Anc_TNS;LAF;RFR;ANCHO;RFR Ancho TNS;D_RFR_Anc_TNS D_RFR_Ang_LAV_sTNS;LAF;RFR;ANG LAV;RFR Angosto LAV sTNS;D_RFR_Ang_LAV_sTNS 

Restricciones_Ruteo.txt:

BloqLine(Orden00001443,DEC3_CHU) BloqLine(Orden00001444,DEC3_CHU) BloqLine(Orden00001445,DEC3_CHU) BloqLine(Orden00001446,DEC3_CHU) BloqLine(Orden00002095,CGL_PSQ) BloqLine(Orden00002095,G1P3_JUV) BloqLine(Orden00002095,G2_JUV) BloqLine(Orden00002095,G2_UNI) BloqLine(Orden00002095,G4_UNI) BloqLine(Orden00002095,G5_MVA) BloqLine(Orden00002096,CGL_PSQ) BloqLine(Orden00002096,G1P3_JUV) BloqLine(Orden00002096,G2_JUV) BloqLine(Orden00002096,G2_UNI) BloqLine(Orden00002096,G4_UNI) BloqLine(Orden00002096,G5_MVA) BloqLine(Orden00002097,CGL_PSQ) BloqLine(Orden00002097,G1P3_JUV) BloqLine(Orden00002097,G2_JUV) BloqLine(Orden00002097,G2_UNI) BloqLine(Orden00002097,G4_UNI) 

Answer 1

In comments you state the following on lines in the various files:

As for number of lines, the FO/data has at least 35-45k lines. nodos.txt has over 300k lines. Table-Familias and Table-Lineas has between 110-120 lines. BloqLineas.txt has 45 lines right now, it can change if the user adds more rules. And Restricciones_Ruteo.txt, right now has 8k lines, but it can change according to what lines can be blocked or nor.

As I see it, there is a dual issue in your code:

• You read each and every file into memory – You read at least 500K lines into memory. That is a lot of memory, and your computer is most likely switching memory most of the time

• Enormous amount of loops – Your main loop, for forder in cr:, does a secondary loop on the blocking rules, for bloqueo in block:, and ends up in doing a tertiary loop, for nodo in nodos: for a variety of if statements. Considering approx 40K entries in cr with 45 blocking rules and for each of these you do 300K loops within nodos, that means you are going to execute 540 000M loops doing your search, before output'ing a blocking line.

  For each of these loops you do a lot of splitting, and searching in a newly built regex, and other somewhat costly stuff.

I'm not surprised it is slow. You should really reconsider how you are processing your files. You would most likely get a major speedup if you could switch the loops around and have the nodos-loop as the main loop and not read it entirely into memory. This would allow for the rest of your code to accelerate. And at the same time move the blocking rules to the inner most looping level.

Simplifying your immense if structure would also most likely help, as you actually seem to do about the same in all of the branches.

Ugly restructure attempt

I started copying out similar blocks out of your original code, into make-shift functions (which won't work due to variable scoping and such). Doing this I found the following make shift functions to exist:

def refactor_first(): nod = nodo.split(';') if check[2] in nodo: #Si parametro de ITEMREP coincide en nodo actual if check[9]: for item in lista: if nod[3] in item: clinea += 1 ub = lista.index(item) del lista[ub] if clinea > 1: if find_word(nod[3], check[7]): #Revisar si parametro linea coincide en nodo actual count += 1 elif clinea == 1: pass else: if find_word(nod[3], check[7]): #Revisar si parametro linea coincide en nodo actual count += 1 def fileblock_write(): if count >= 1: fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])) count = 0 def fo1_in_nodo(): for nodo in nodos: if fo[1] in nodo: refactor_first() fileblock_write() def check2_in_nodo(): for nodo in nodos: if check[2] in nodo: #ITEMREP refactor_first() fileblock_write() def all_in_nodo(): for nodo in nodos: refactor_first() fileblock_write() 

Using these non-working functions your massive if block is simplified to this:

 #Fam87 if forder.find(check[5]) >= 1 or (forder.find(fam87)) >= 1: if forder.find(check[6]) >= 1 or (forder.find(est)) >= 1: #Si EstCom coincide con bloqueo actual if fo[1] in check[2]: #Si ITEMREP coincide bloqueo actual all_in_nodo(): elif check[2] == '': #Cuando no hay ITEMREP fo1_in_nodo() #No hay EstCom, pero hay ITEMREP elif fo[1] in check[2] and (check[6] == ''): #Si ITEMREP coincide con bloqueo actual check2_in_nodo() elif check[2] == '' and check[6] == '': #Cuando no hay ITEMREP ni EstCom fo1_in_nodo() #Familia N1 elif fo[5] in check[3] and (check[5] == ''): if fo[2] in check[6]: #EstCom if fo[1] in check[2]: #ITEMREP check2_in_nodo() elif check[2] == '': #No hay ITEMREP fo1_in_nodo() #Cuando no hay EstCom, pero hay ITEMREP elif fo[1] in check[2] and (check[6] == ''): #Si ITEMREP coincide con parametro de bloqueo check2_in_nodo() elif check[2] == '' and check[6] == '': #No hay ITEMREP ni EstCom fo1_in_nodo() #Familia N2 elif fo[6] in check[4] and (check[3] == '') and (check[5] == ''): if fo[2] in check[6]: #EstCom if fo[1] in check[2]: #ITEMREP check2_in_nodo() elif check[2] == '': #Cuando no hay ITEMREP fo1_in_nodo() #Cuando no hay EstCom, pero hay ITEMREP elif fo[1] in check[2] and (check[6] == ''): #Si ITEMREP coincide con parametro de bloqueo check2_in_nodo() elif check[2] == '' and check[6] == '': #No hay ITEMREP fo1_in_nodo() #Cuando no hay parametro para ninguna Familia, se sigue a bloquear EstCom elif fo[2] in check[6] and (check[4] == '') and (check[3] == '') and (check[5] == ''): #EstCom if fo[1] in check[2]: #ITEMREP check2_in_nodo() #Si no existe parametro de ITEMREP elif check[2] == '': fo1_in_nodo() #Cuando no hay Familias, y no hay EstCom, bloquear por ITEMREP elif fo[1] in check[2] and (check[6] == '') and (check[4] == '') and (check[3] == '') and (check[5] == ''): fo1_in_nodo() 

All the if branches here ends up in either: check2_in_nodo(), fo1_in_nodo or all_in_nodo(). And these vary by only one thing and that is if they check whether check[2] or fo[1] or all is present in the current nodo element. If present they all call the same refactor_first() which ends up doing the find_word(nod[3], check[7]), and if this is found you do the fileblock.write("BloqLine(%s,%s)\n" % (fo[0],check[7])).

This should be possible to restructure into a nicer and possibly faster structure:

• First of all, the lista should be prebuilt outside of the loops, as it is constant relative to the blocking rules. This can be copied at will for the next loop, but do not regenerate it all the time
• At start of loop you should precompile the regex as you always check against the check[7] word. Store this as check_pattern
• You should also break out of the loop immediately when count == 1, as you'll then always do a fileblock.write(). No need to check the rest of nodos in this case
• If doable, avoid doing the del lista[ub] as that is somewhat expensive
• Replace find_word with a cheaper check_pattern.search(nod[3])
• Instead of storing the entire nodos list, it seems like you can get away with only storing the nod[3] element. This should be done when reading in the nodos file. This will reduce memory issues and remove the need for some of the splits possibly.

In summary

I'm not going to do the entire refactoring, but you should focus on the following:

• Check if you can switch around the loops, so that you can have the largest file(s) only on disk, and not entirely in memory
• If that is not feasible, then strip away all data you can, and only leave the neccessary data in the variable you are working with
• Precompile regex as early as possible, and the same for static lists like lista
• If you're not able to restructure loops, make functions for the fo1_in_nodo(), check2_in_nodo and all_in_nodo make-shift functions
• Simplify the massive if structure which triggers the make-shift functions. This won't affect performance much, but it can help you understand what you are doing more easily

But at the end of the day, you also seriously need to reconsider what you are doing, because checking 40 000 orders (from cr) against 300 000 nodos seems just plain wrong, something you should be using a database to solve. This is a massive amount of data you are processing.

Answer 2

I'm going to give a frank assessment: I think that this code is effectively unmaintainable, and that you would be better of rewriting it from scratch, because you are using the wrong tool for the job. Essentially, you want to combine data from five tables to produce a query result. There are better ways to do that than writing 400 lines of for loops and if blocks. This is a solved problem in computing, and there is no sense in reinventing the wheel.

Option A: SQL

A lot of what you are doing is basically equivalent to a join in SQL. For example, this code…

fo = '' with codecs.open('data.txt','w',encoding = 'utf-8') as data: for forder in finalorder: order = forder.split(';') orden = [order[0],order[1],order[15],order[24],order[2],'',''] for familia in list_familias: fam = familia.split(';') if order[2] in fam[0]: orden[5] = fam[1] orden[6] = fam[2] fo = ';'.join(orden) data.write(fo+'\n') with codecs.open('data.txt','r',encoding = 'utf-8') as cross: cr = cross.readlines() 

… is just a textbook example of a LEFT OUTER JOIN operation:

CREATE VIEW CrossOrders AS SELECT Order.col0 AS order_id , Order.col1 AS item_rep , Order.col15 AS est_com , Order.col24 , Order.col2 , COALESCE(Familia.col1, '') , COALESCE(Familia.col2, '') FROM Order LEFT OUTER JOIN Familia ON Order.col2 = Familia.col0; 

(I don't know what all of your columns represent. I've used identifiers like col24 as placeholders, but of course you should define your database tables with column names that make sense, like quantity or supplier).

I'm not going to reverse-engineer all of your code, but I would be willing to bet that the entire program can be reduced to the VIEW I defined above, plus one giant SQL query.

So how would you go about rewriting the code? If you already have a database at your company, such as sql-server or postgresql, then great, use that instead of your .txt files.

If you don't have a database handy, then just install sqlite, which is an SQL engine that doesn't require a database server. Import your .txt files to SQLite, either using the sqlite shell or Python code. Even if you have to re-create a new SQLite database from scratch with every batch run, import all of the CSV data, and throw away the database after the query, it's still worth it for the massive productivity gains. The resulting program would have a chance of being maintainable. It would likely be faster as well, as SQL engines are specifically optimized to do this kind of thing.

Option B: PANDAS

If you prefer to avoid SQL and stick to Python and semicolon-delimited files, then there are libraries that can help you. In particular, pandas is designed to work with tabular data sets. The first page of code would look more like this:

import pandas as pd finalorders = pd.read_csv('FO.txt', encoding='UTF-8', sep=';', header=None) familias = pd.read_csv('Table-Familias.txt', encoding='UTF-8', sep=';', header=None) orden = pd.merge( left=finalorders, right=familias, how='left', left_on=0, right_on=2 )[['0_x', '1_x', 15, 24, '2_x', '1_y', '2_y']] normalize_caps = lambda s: s.lower().capitalize() blocks = pd.read_csv('BloqLines.txt', encoding='UTF-8', sep=';', header=None, converters={5:normalize_caps, 6:normalize_caps}) tlines = pd.read_csv('Table-Lineas.txt', encoding='UTF-8', sep=';', header=None) 

Personally, I'd prefer to go with SQL than PANDAS for this task. Even if you aren't using a database now, I'm guessing that you will eventually wish that you had one, because you're already part way to building a database system yourself anyway.

Additional observations

There are other general issues with your program.

• Use more list comprehensions. Not only do they make your code more compact and readable, they also reduce the number of variables floating around in your code. (And you have a lot of similarly named variables like node, nodos, and nod, which makes it confusing.)

• Each row of data is "stringly typed". That is, you're keeping each row as a semicolon-delimited string. As a result, you have line.split(';') littered all over the code, which is both inefficient and hard to follow.

  What you want to do instead is get it into 2D array form as soon as possible:

  with codecs.open('Table-Familias.txt', 'r', encoding='utf-8') as familias: list_familias = [line.rstrip().split(';') for line in familias] 

• As specified in PEP 8, indentation should be 4 spaces, not 3. Since whitespace is significant in Python, this is a pretty strong convention that you should follow.

• You're using the in keyword rather carelessly in many places.

  For example, when you wrote if order[2] in fam[0]:, you really meant if order[2] == fam[0]:. Also, instead of if 'True' in check[0]:, you should write if 'True' == check[0]. In those places, in would do a substring search rather than an exact string match. The code happens to work, but it's sloppy.

  Elsewhere, you're doing if check[7] in line:, which means that you are searching for check[7] across all of the columns of that row. This is a very unconventional kind of join condition: you really want to search for check[7] in column 0 or column 4, and you should write the code accordingly.

Answer 3

One fairly substantial optimization is removing extra computations.

Lets look at:

def find_word(text, search): result = re.findall('\\b'+search+'\\b', text, flags=re.IGNORECASE) if len(result)>0: return True else: return False 

Your use of findall is wasteful, as you only check if there's at least 1 match, you don't really care about the quantity/matches.

Thus we can rewrite it as follows:

def find_word(text, search): return re.search('\\b'+search+'\\b', text, flags=re.IGNORECASE) is not None 

and have the exact same effect at much lower cost, since search returns as soon as any match is found.

This should have a pretty substantial performance impact considering the number of times you call this function inside loops.

As for the code...readability has much to be desired. Having tons of nested conditionals/loops is really hard to read, so I would strongly recommend writing functions for each major chunk.

Answer 4

Here is an alternate approach related to optimising your code, if my previous answer doesn't help enough, and if you can't switch to using either a database, like sqlite, or use pandas. Before explaining the alternate I would like to recap that your current algorithm is something along the following lines:

• Read all files into memory, and preprocess the FO.txt and Table-Familias.txt into the new data.txt which are also read
• Main loop: For each orders in data.txt: (approx 40K entries)
  • Secondary loop: For each blocking rule: (approx 40-50 rules)
  • Continue with next order, if rule not active
  • If active blocking rule:
    • Rebuild Logica taxativo: lista_
    • Various large if tests, before doing:
    • Tertiary loop: For all nodo in nodos: (approx 300K entries)
      • Simple logic, fix lista, and output to file

Alternate approach using an intermediary file

The alternate approach is to split this into two, using an intermediary file. That is use the above approach, but instead of doing the tertiary loop, you write just those parameters needed for modifying the nodos list into a new file. Doing this you can skip loading nodos in the first pass, and do a memory efficient one pass of data.txt where you process it line by line.

And then you add a new pass where you read this intermediary file into memory, and do a memory efficient one pass of nodos.txt matching only those needed and doing the output to the final file.

Doing this you would simplify from potentially having a time and/or space complexity of \$O(n\cdot m)\$, where \$n\$ is number of lines in data.txt and \$m\$ is number of lines in nodos.txt, into having having a time complexity of \$O(n+m)\$, and a space complexity of \$\theta (p)\$ where p is lines in the intermediary file. This can have a huge impact on execution time.

To change from reading all lines into doing the memory efficient version, you change the first for loop into:

with codecs.open(`data.txt', 'r', ...) as orderfile, codecs.open('tmp_file.csv', ...) as tmpfile: for order in orderfile: # Process the order 

And similar for the other one. This also allows for a little better file handling as Python keeps track of the file handles, and you don't need to. :-)

Assign tuples to variables

Another point related to readability would be to do something like the following:

active_rule, title, item_repr, _, something = bloqueo.split() 

Then you can replace the cryptic check[2] with a more meaningful variable. Similar can of course also be done instead of using fo[7] and friends...

This alternate approach in addition to some of the other comments related to only read what you need, and avoid the in when you actually mean ==, and prebuilding regexps, and other suggestions in other answers should make your code a lot faster, and easier to maintain and understand.