Scribd Logo
Upload

Welcome to Scribd!

  • 38 views

    Spark 101 - Overview and Efficient Use

    Uploaded by

    divya kolluri
    This document provides an overview of Spark and how to efficiently use it. It discusses Spark's RDD and DataFrame models for parallel computing, how Spark supports a wide range of input formats and can persist data for faster reuse. It also covers the difference between narrow and wide dependencies and the anatomy of a Spark job in terms of DAGs, jobs, stages and tasks.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd

    Spark 101 - Overview and Efficient Use

    Uploaded by

    divya kolluri
    38 views9 pages
    This document provides an overview of Spark and how to efficiently use it. It discusses Spark's RDD and DataFrame models for parallel computing, how Spark supports a wide range of input formats and can persist data for faster reuse. It also covers the difference between narrow and wide dependencies and the anatomy of a Spark job in terms of DAGs, jobs, stages and tasks.

    Original Description:

    spark

    Original Title

    Spark Session 4

    Copyright

    © © All Rights Reserved

    Available Formats

    PPTX, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document provides an overview of Spark and how to efficiently use it. It discusses Spark's RDD and DataFrame models for parallel computing, how Spark supports a wide range of input formats and can persist data for faster reuse. It also covers the difference between narrow and wide dependencies and the anatomy of a Spark job in terms of DAGs, jobs, stages and tasks.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Download as pptx, pdf, or txt
    38 views9 pages

    Spark 101 - Overview and Efficient Use

    Uploaded by

    divya kolluri
    This document provides an overview of Spark and how to efficiently use it. It discusses Spark's RDD and DataFrame models for parallel computing, how Spark supports a wide range of input formats and can persist data for faster reuse. It also covers the difference between narrow and wide dependencies and the anatomy of a Spark job in terms of DAGs, jobs, stages and tasks.

    Copyright:

    © All Rights Reserved
    Download as PPTX, PDF, TXT or read online from Scribd
    Download as pptx, pdf, or txt
    of 9

    Spark 101– Overview and Efficient Use

    Spark Model of Parallel Computing:


    RDDs & Dataframe

     Immutable and fault-tolerant

     Support vast spectrum of Input


    formats such as local file, hdfs, ES

     Can be Persisted for faster reuse

     Dataframe is distributed equivalent


    for pandas or R dataframe

     Lazily Evaluated
    Performance Comparison : RDD &
    Dataframe
    Lazy Evaluation

     Transformations

     Action

     Directed Acyclic Graph

     Logical and Execution plan


    Unified Spark Stack
    Spark SQL Flow
    Wide Versus Narrow Dependencies

     Narrow Dependencies, e.g. –


    map, filter, flatmap etc.

     Wide Dependencies, e.g. – sort,


    join, groupByKey etc.
    The Anatomy of a Spark Job

     The DAG

     JOB

     Stages

     Tasks

    You might also like