Scribd Logo
Upload

Welcome to Scribd!

  • 7 views

    Colab Spark Initialize Step

    Uploaded by

    Esrat Sharmin Eliza

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd

    Colab Spark Initialize Step

    Uploaded by

    Esrat Sharmin Eliza
    7 views1 page

    Copyright

    © © All Rights Reserved

    Available Formats

    TXT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Download as txt, pdf, or txt
    7 views1 page

    Colab Spark Initialize Step

    Uploaded by

    Esrat Sharmin Eliza

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Download as txt, pdf, or txt
    You are on page 1of 1

    # Open colab and paste the following in a cell and execute

    !apt-get update # Update apt-get repository.


    !apt-get install openjdk-8-jdk-headless -qq > /dev/null # Install Java.
    !wget -q http://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-
    hadoop3.2.tgz # Download Apache Sparks.
    !tar xf spark-3.1.1-bin-hadoop3.2.tgz # Unzip the tgz file.
    !pip install -q findspark # Install findspark. Adds PySpark to the System path
    during runtime.

    # Set environment variables


    import os
    os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
    os.environ["SPARK_HOME"] = "/content/spark-3.1.1-bin-hadoop3.2"

    !ls

    # Initialize findspark
    import findspark
    findspark.init()

    # Create a PySpark session


    from pyspark.sql import SparkSession
    spark = SparkSession.builder.master("local[*]").getOrCreate()

    #Create a PySpark session


    from pyspark.sql import SparkSession
    spark = SparkSession.builder.master("local[*]").getOrCreate()

    spark = SparkSession.builder.appName("nik").getOrCreate()

    from pyspark import SparkContext


    sc = SparkContext.getOrCreate()

    You might also like