added js-css files webpage extractor tutorial

x4nth055 · x4nth055 · commit 114adc74e66e · 2020-02-09T19:17:04.000+01:00
diff --git a/README.md b/README.md
@@ -68,6 +68,7 @@ This is a repository of all the tutorials of [The Python Code](https://www.thepy
     - [How to Make an Email Extractor in Python](https://www.thepythoncode.com/article/extracting-email-addresses-from-web-pages-using-python). ([code](web-scraping/email-extractor))
     - [How to Convert HTML Tables into CSV Files in Python](https://www.thepythoncode.com/article/convert-html-tables-into-csv-files-in-python). ([code](web-scraping/html-table-extractor))
     - [How to Use Proxies to Anonymize your Browsing and Scraping using Python](https://www.thepythoncode.com/article/using-proxies-using-requests-in-python). ([code](web-scraping/using-proxies))
+    - [How to Extract Script and CSS Files from Web Pages in Python](https://www.thepythoncode.com/article/extract-web-page-script-and-css-files-in-python). ([code](web-scraping/webpage-js-css-extractor))
 
 - ### [Python Standard Library](https://www.thepythoncode.com/topic/python-standard-library)
     - [How to Transfer Files in the Network using Sockets in Python](https://www.thepythoncode.com/article/send-receive-files-using-sockets-python). ([code](general/transfer-files/))
diff --git a/web-scraping/webpage-js-css-extractor/README.md b/web-scraping/webpage-js-css-extractor/README.md
@@ -0,0 +1,8 @@
+# [How to Extract Script and CSS Files from Web Pages in Python](https://www.thepythoncode.com/article/extract-web-page-script-and-css-files-in-python)
+To run this:
+- `pip3 install -r requirements.txt`
+- Extracting `http://books.toscrape.com`'s CSS & Script files:
+    ```
+    python extractor.py http://books.toscrape.com/
+    ```
+    2 files will appear, one for javascript files (`javascript_files.txt`) and the other for CSS files (`css_files.txt`)
diff --git a/web-scraping/webpage-js-css-extractor/extractor.py b/web-scraping/webpage-js-css-extractor/extractor.py
@@ -0,0 +1,50 @@
+import requests
+from bs4 import BeautifulSoup as bs
+from urllib.parse import urljoin
+
+import sys
+
+# URL of the web page you want to extract
+url = sys.argv[1]
+
+# initialize a session
+session = requests.Session()
+# set the User-agent as a regular browser
+session.headers["User-Agent"] = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.157 Safari/537.36"
+
+# get the HTML content
+html = session.get(url).content
+
+# parse HTML using beautiful soup
+soup = bs(html, "html.parser")
+
+# get the JavaScript files
+script_files = []
+
+for script in soup.find_all("script"):
+    if script.attrs.get("src"):
+        # if the tag has the attribute 'src'
+        script_url = urljoin(url, script.attrs.get("src"))
+        script_files.append(script_url)
+
+# get the CSS files
+css_files = []
+
+for css in soup.find_all("link"):
+    if css.attrs.get("href"):
+        # if the link tag has the 'href' attribute
+        css_url = urljoin(url, css.attrs.get("href"))
+        css_files.append(css_url)
+
+
+print("Total script files in the page:", len(script_files))
+print("Total CSS files in the page:", len(css_files))
+
+# write file links into files
+with open("javascript_files.txt", "w") as f:
+    for js_file in script_files:
+        print(js_file, file=f)
+
+with open("css_files.txt", "w") as f:
+    for css_file in css_files:
+        print(css_file, file=f)
diff --git a/web-scraping/webpage-js-css-extractor/requirements.txt b/web-scraping/webpage-js-css-extractor/requirements.txt
@@ -0,0 +1,2 @@
+requests
+bs4
