book

Data Algorithms with Spark

by Mahmoud Parsian

April 2022

Intermediate to advanced

435 pages

9h 44m

English

O'Reilly Media, Inc.

Book available

Start your free trial

Related skills

Apache Spark

Associated roles

Data engineer
Data/ML engineer
Data scientist
ML/AI architect

Senior software developer

+1 more

Why I Wrote This BookWho This Book Is ForHow This Book Is OrganizedConventions Used in This BookUsing Code ExamplesO’Reilly Online LearningHow to Contact UsAcknowledgments
Why Spark for Data AnalyticsThe Spark EcosystemSpark ArchitectureThe Power of PySparkPySpark ArchitectureSpark Data AbstractionsRDD ExamplesSpark RDD OperationsDataFrame ExamplesUsing the PySpark ShellLaunching the PySpark ShellCreating an RDD from a CollectionAggregating and Merging Values of KeysFiltering an RDD’s ElementsGrouping Similar KeysAggregating Values for Similar KeysETL Example with DataFramesExtractionTransformationLoadingSummary
The DNA Base Count ExampleThe DNA Base Count ProblemFASTA FormatSample DataDNA Base Count Solution 1Step 1: Create an RDD[String] from the InputStep 2: Define a Mapper FunctionStep 3: Find the Frequencies of DNA LettersPros and Cons of Solution 1DNA Base Count Solution 2Step 1: Create an RDD[String] from the InputStep 2: Define a Mapper FunctionStep 3: Find the Frequencies of DNA LettersPros and Cons of Solution 2DNA Base Count Solution 3The mapPartitions() TransformationStep 1: Create an RDD[String] from the InputStep 2: Define a Function to Handle a PartitionStep 3: Apply the Custom Function to Each PartitionPros and Cons of Solution 3Summary
Data Abstractions and MappersWhat Are Transformations?Lazy TransformationsThe map() TransformationDataFrame MapperThe flatMap() Transformationmap() Versus flatMap()Apply flatMap() to a DataFrameThe mapValues() TransformationThe flatMapValues() TransformationThe mapPartitions() TransformationHandling Empty PartitionsBenefits and DrawbacksDataFrames and mapPartitions() TransformationSummary
Creating Pair RDDsReduction TransformationsSpark’s ReductionsSimple Warmup ExampleSolving with reduceByKey()Solving with groupByKey()Solving with aggregateByKey()Solving with combineByKey()What Is a Monoid?Monoid and Non-Monoid ExamplesThe Movie ProblemInput Dataset to AnalyzeThe aggregateByKey() TransformationFirst Solution Using aggregateByKey()Second Solution Using aggregateByKey()Complete PySpark Solution Using groupByKey()Complete PySpark Solution Using reduceByKey()Complete PySpark Solution Using combineByKey()The Shuffle Step in ReductionsShuffle Step for groupByKey()Shuffle Step for reduceByKey()Summary
Introduction to PartitionsPartitions in SparkManaging PartitionsDefault PartitioningExplicit PartitioningPhysical Partitioning for SQL QueriesPhysical Partitioning of Data in SparkPartition as Text FormatPartition as Parquet FormatHow to Query Partitioned DataAmazon Athena ExampleSummary
Introduction to GraphsThe GraphFrames APIHow to Use GraphFramesGraphFrames Functions and AttributesGraphFrames AlgorithmsFinding TrianglesMotif FindingReal-World ApplicationsGene AnalysisSocial RecommendationsFacebook CirclesConnected ComponentsAnalyzing Flight DataSummary

Relational DatabasesReading from a DatabaseWriting a DataFrame to a DatabaseReading Text FilesReading and Writing CSV FilesReading CSV FilesWriting CSV FilesReading and Writing JSON FilesReading JSON FilesWriting JSON FilesReading from and Writing to Amazon S3Reading from Amazon S3Writing to Amazon S3Reading and Writing Hadoop FilesReading Hadoop Text FilesWriting Hadoop Text FilesReading and Writing HDFS SequenceFilesReading and Writing Parquet FilesWriting Parquet FilesReading Parquet FilesReading and Writing Avro FilesReading Avro FilesWriting Avro FilesReading from and Writing to MS SQL ServerWriting to MS SQL ServerReading from MS SQL ServerReading Image FilesCreating a DataFrame from ImagesSummary
Rank ProductCalculation of the Rank ProductFormalizing Rank ProductRank Product ExamplePySpark SolutionPageRankPageRank’s Iterative ComputationCustom PageRank in PySpark Using RDDsCustom PageRank in PySpark Using an Adjacency MatrixPageRank with GraphFramesSummary
Input-Map-OutputRDD SolutionDataFrame SolutionFlat Mapper functionalityInput-Filter-OutputRDD SolutionDataFrame SolutionDataFrame FilterInput-Map-Reduce-OutputRDD SolutionDataFrame SolutionInput-Multiple-Maps-Reduce-OutputRDD SolutionDataFrame SolutionInput-Map-Combiner-Reduce-OutputInput-MapPartitions-Reduce-OutputInverted IndexProblem StatementInputOutputPySpark SolutionSummary
In-Mapper CombiningBasic MapReduce AlgorithmIn-Mapper Combining per RecordIn-Mapper Combining per PartitionTop-10Top-N FormalizedPySpark SolutionFinding the Bottom 10MinMaxSolution 1: Classic MapReduceSolution 2: SortingSolution 3: Spark’s mapPartitions()The Composite Pattern and MonoidsMonoidsMonoidal and Non-Monoidal ExamplesNon-Monoid MapReduce ExampleMonoid MapReduce ExamplePySpark Implementation of Monoidal MeanFunctors and MonoidsConclusion on Using MonoidsBinningSortingSummary
Introduction to the Join OperationJoin in MapReduceMap PhaseReducer PhaseImplementation in PySparkMap-Side Join Using RDDsMap-Side Join Using DataFramesStep 1: Create Cache for AirportsStep 2: Create Cache for AirlinesStep 3: Create Facts TableStep 4: Apply Map-Side JoinEfficient Joins Using Bloom FiltersIntroduction to Bloom FiltersA Simple Bloom Filter ExampleBloom Filters in PythonUsing Bloom Filters in PySparkSummary
Introduction to Feature EngineeringAdding New FeaturesApplying UDFsCreating PipelinesBinarizing DataImputationTokenizationTokenizerRegexTokenizerTokenization with a PipelineStandardizationNormalizationScaling a Column Using a PipelineUsing MinMaxScaler on Multiple ColumnsNormalization Using NormalizerString IndexingApplying StringIndexer to a Single ColumnApplying StringIndexer to Several ColumnsVector AssemblyBucketingBucketizerQuantileDiscretizerLogarithm TransformationOne-Hot EncodingTF-IDFFeatureHasherSQLTransformerSummary

Overview

Apache Spark's speed, ease of use, sophisticated analytics, and multilanguage support makes practical knowledge of this cluster-computing framework a required skill for data engineers and data scientists. With this hands-on guide, anyone looking for an introduction to Spark will learn practical algorithms and examples using PySpark.

In each chapter, author Mahmoud Parsian shows you how to solve a data problem with a set of Spark transformations and algorithms. You'll learn how to tackle problems involving ETL, design patterns, machine learning algorithms, data partitioning, and genomics analysis. Each detailed recipe includes PySpark algorithms using the PySpark driver and shell script.

With this book, you will:

Learn how to select Spark transformations for optimized solutions
Explore powerful transformations and reductions including reduceByKey(), combineByKey(), and mapPartitions()
Understand data partitioning for optimized queries
Build and apply a model using PySpark design patterns
Apply motif-finding algorithms to graph data
Analyze graph data by using the GraphFrames API
Apply PySpark algorithms to clinical and genomics data
Learn how to use and apply feature engineering in ML algorithms
Understand and use practical and pragmatic data design patterns