All Questions

Here’s a brief overview of what I want to achieve Extract raw htmls and save them Use Crawl4AI to produce a ‘cleaner’ and smaller HTML that has a lot of information, including what I will eventually ...

I have been using Crawler4AI to try downloading a series of documents from this Website. However, since it requieres JavaScript code and I am using Python, I don't know hot to solve my error. Code, ...

I want to use the data in my codes with Transfermark Scraper for my own special purpose. I get all the desired data in the codes except Current Club, but I can't get the Club name. I tried all the ...

I am trying to scrape data from www.example.com but the below code returns error : import asyncio from crawl4ai import AsyncWebCrawler from crawl4ai.async_configs import BrowserConfig, ...

I'm trying to crawl https://kick.com/browse/categories with playwright which has infinite scroll. I've tried evaluating the below js code and wait for an extended period for loading. I'm turning off ...

I am seeking assistance in extracting data from a website with multiple tabs and saving it in a .csv format using Python and Selenium. The website in question is: https://www.amfiindia.com/research-...

I am crawling https://kick.com/browse/categories where every time you scroll it loads new cards of a category. I have tried multiple methods using playwright but none of them worked. Would appreciate ...

I have two scrapy Spiders: Spider 1 crawls a list of product links (~10000) and saves them to a csv file using a feed. It doesn't visit each of those links, only the categories (with multiple pages). ...

I have a code snippet to extract the inputable and clickable node elements (i.e. interactive elements) from the DOM tree of the web pages via Playwright in python. This code almost works properly but ...

Please, i need someone help me. I can't understand why I only crawl 25 movies instead of 250. My code: import pandas as pd import requests from bs4 import BeautifulSoup headers = {'User-Agent': '...

I am trying to download videos from a site, which requires extracting 1 "download url" that resides on each "video url". Example: "video url": https://www.example.com/...

working on the getting to run the Facebook-Scraper (cf https://github.com/kevinzg/facebook-scraper ) import facebook_scraper as fs # get POST_ID from the URL of the post which can have the following ...

I am using icrawler on python to scrape images online. I have a list of strings download_waitlist = ["cat","dog","car","motorbike","snoop dogg"] that ...

On the mentioned website, After searching for the token, a slider captcha appears. An example of the captcha: I want to bypass the slider captcha. I took reference from the first solution in Unable ...

I spend a lot of time searching about this. At the end of the day I combined a number of answers and it works. I share my answer and I'll appreciate it if anyone edits it or provides us with an easier ...