How does your scraping algorithm get through CloudFlare. #78
Replies: 3 comments
-
|
I had a similar doubt. |
Beta Was this translation helpful? Give feedback.
-
|
Does this method work reliably if Cloudflare’s protections increase, or does it mainly work because the requests mimic normal browser API calls? |
Beta Was this translation helpful? Give feedback.
-
|
There are basically 2 things we can do one is to have script that solves the cloudflare challenge using browser automation through SeleniumBase or even using pydoll and getting the cf_clearance token and cookies and then write another script that will use those token and cookies to send request to the same url where cloudflare was triggered using requests module that is how you can bypass cloudflare and make sure whenever there is a captcha get new cookies and then send requests using those new cookies |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi, I wanted to know how this project is able to scrape the leetcode website without using any browsing drivers like Selenium. Doesn't CloudFlare stop your python script from getting access to the data?
Beta Was this translation helpful? Give feedback.
All reactions