Python Web scraping

Question

I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any.

What do I want to achieve?

I want to scrape all chapters of each Novel in each category and post it on a WordPress blog to test. Please point out anything that I missed, and is mandatory to run this script on the WordPress blog.

from requests import get from bs4 import BeautifulSoup import re r = get(site, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"}) soup = BeautifulSoup(r.text, "lxml") category = soup.findAll(class_="search-by-genre") # Getting all categories categories = [] for link in soup.findAll(href=re.compile(r'/category/\w+$')): print("Category:", link.text) category_link = link['href'] categories.append(category_link) # Getting all Novel Headers for category in categories: r = get(category_link, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"}) soup = BeautifulSoup(r.text, "lxml") Novels_header = soup.findAll(class_="top-novel-header") # Getting Novels' Title and Link for Novel_names in Novels_header: print("Novel:", Novel_names.text.strip()) Novel_link = Novel_names.find('a')['href'] # Getting Novel's Info r = get(Novel_link, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"}) soup = BeautifulSoup(r.text, "lxml") Novel_divs = soup.findAll(class_="chapter-chs") # Novel Chapters for articles in Novel_divs: article_ch = articles.findAll("a") for chapters in article_ch: ch = chapters["href"] # Getting article r = get(ch, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"}) soup = BeautifulSoup(r.content, "lxml") title = soup.find(class_="block-title") print(title.text.strip()) full_article = soup.find("div", {"class": "desc"}) # remove ads inside the text: for ads in full_article.select('center, small, a'): ads.extract() print(full_article.get_text(strip=True, separator='\n')) 

Answer 1

Naming

Variable names should be snake_case, and should represent what they are containing. I would also use req instead of r. The extra two characters aren't going to cause a heartache.

Constants

You have the same headers dict in four different places. I would instead define it once at the top of the file in UPPER_CASE, then just use that wherever you need headers. I would do the same for site.

List Comprehension

I would go about collecting categories in this way:

categories = [link['href'] for link in soup.findAll(href=re.compile(r'/category/\w+$'))] 

It's shorter and utilizes a quirk in the python language. Of course, if you want to print out each one, then add this just after:

for category in categories: print(category) 

Also, it seems like you assign category_link to the last element in the list, so that can go just outside the list comprehension.

Save your assignments

Instead of assigning the result of soup.find to a variable, then using it in a loop, simply put that soup.find in the loop. Take a look:

for articles in soup.findAll(class_="chapter-chs"): for chapters in articles.findAll("a"): .... 

---

As a result of the above changes, you code would look something like this:

from requests import get from bs4 import BeautifulSoup import re HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"} SITE = "https://readlightnovel.org/" req = get(SITE, headers=HEADERS) soup = BeautifulSoup(req.text, "lxml") category = soup.findAll(class_="search-by-genre") categories = [link['href'] for link in soup.findAll(href=re.compile(r'/category/\w+$'))] category_link = categories[-1] # Getting all Novel Headers for category in categories: req = get(category_link, headers=HEADERS) soup = BeautifulSoup(req.text, "lxml") novels_header = soup.findAll(class_="top-novel-header") # Getting Novels' Title and Link for novel_names in novels_header: print("Novel:", novel_names.text.strip()) novel_link = novel_names.find('a')['href'] # Getting Novel's Info req = get(novel_link, headers=HEADERS) soup = BeautifulSoup(req.text, "lxml") # Novel Chapters for articles in soup.findAll(class_="chapter-chs"): for chapters in articles.findAll("a"): ch = chapters["href"] # Getting article req = get(ch, headers=HEADERS) soup = BeautifulSoup(req.content, "lxml") title = soup.find(class_="block-title") print(title.text.strip()) full_article = soup.find("div", {"class": "desc"}) # remove ads inside the text: for ads in full_article.select('center, small, a'): ads.extract() print(full_article.get_text(strip=True, separator='\n')) 

Answer 2

I think you can even get rid of the regular expressions. I prefer to use the BS4 functions.

Instead of:

categories = [link['href'] for link in soup.findAll(href=re.compile(r'/category/\w+$'))] 

This statement is equivalent using a CSS selector:

categories = [link['href'] for link in soup.select("a[href*=\/category\/]")] 

That means: fetch all the a href tags that contain text /category/ (escaped).