PySpark-Tutorial provides basic algorithms using PySpark

Our style guide for writing readable and maintainable PySpark code.

All updated cheat sheets regarding data science, data analysis provided by Datacamp are here. These cheat sheets cover quick reads on Machine Learning, Deep Learning, Python, R, SQL and more. Perfect cheat sheets when you want to revise some topics in less time.

List of useful commands for Pyspark

Project based on application of azure databricks

This notebook contains the usage of Pyspark to build machine learning classifiers (note that almost ml_algorithm supported by Pyspark are used in this notebook)

This repository contains the Notes for Pyspark

This notebook performs EDA over a movie ratings dataset via pyspark sql.

This code demonstrates how to integrate PySpark with datasets and perform simple data transformations. It loads a sample dataset using PySpark's built-in functionalities or reads data from external sources and converts it into a PySpark DataFrame for distributed processing and manipulation.

Batch Processing using Apache Spark and Python for data exploration

Clustering vs Classification

Generate a synthetic dataset with one million records of employee information from a fictional company, load it into a PostgreSQL database, create analytical reports using PySpark and large-scale data analysis techniques, and implement machine learning models to predict trends in hiring and layoffs on a monthly and yearly basis.

This script builds a linear regression model using PySpark to predict student admissions at Unicorn University.

Inventory value is also important for determining a company's liquidity, or its ability to meet its short-term financial obligations. A high inventory value can indicate that a company has too much money tied up in inventory, which could make it difficult for the company to pay its bills.

Module 22 challenge: Using Google Colab to work on Big Data queries with PySpark SQL, parquet, and cache partitions

Network Intrusion Detector is a distributed intrusion detection system built with PySpark. It preprocesses, encodes, and models network traffic data to detect anomalies using a Random Forest classifier, achieving high accuracy and efficiency through feature selection and scalable data processing. The system is suitable for large-scale environments

Cardiovascular Disease Prediction

The notebook shows how tools of the PySpark SQL module work in practice.