Big Data
(Spark)
Instructor: Trong-Hop Do
May 5th 2021
S3Lab
Smart Software System Laboratory
1
“Big data is at the foundation of all the
megatrends that are happening today, from
social to mobile to cloud to gaming.”
– Chris Lynch, Vertica Systems
Big Data 2
Spark shell
● Open Spark shell
● Command: spark-shell
3
What is SparkContext?
● Spark SparkContext is an entry point to Spark and defined in
org.apache.spark package and used to programmatically create Spark
RDD, accumulators, and broadcast variables on the cluster.
● Its object sc is default variable available in spark-shell and it can be
programmatically created using SparkContext class.
● Most of the operations/methods or functions we use in Spark come from
SparkContext for example accumulators, broadcast variables, parallelize,
and more.
4
What is SparkContext?
● Test some methods with the default SparkContext object sc in Spark shell
● There are many more methods of SparkContext
5
Spark RDD
● Resilient Distributed Datasets (RDD) is a fundamental data structure of
Spark, It is an immutable distributed collection of objects. Each dataset in
RDD is divided into logical partitions, which may be computed on different
nodes of the cluster.
● Spark RDD can be created in several ways using Scala & Pyspark
languages, for example, It can be created by using
sparkContext.parallelize(), from text file, from another RDD, DataFrame,
and Dataset.
6
Spark RDD
Create an RDD through Parallelized Collection
● Spark shell provides SparkContext variable “sc”, use sc.parallelize() to create an RDD[Int].
Note: the printed list might be unordered as foreach runs in
parallel across the partitions which creates non-deterministic
ordering. The action collect() gives you an array of the partitions
concatenated in their sorted order. Or you might set the number
of parititions to 1.
7
Spark RDD
Create an RDD through Parallelized Collection
● Spark shell provides SparkContext variable “sc”, use sc.parallelize() to create an RDD[Int].
8
Spark RDD
Read single or multiple text files into single RDD?
● Create files and put them to HDFS
9
Spark RDD
Read single or multiple text files into single RDD?
● Read single file and create an RDD[String]
10
Spark RDD
Read single or multiple text files into single RDD?
● Read multiple files and create an RDD[String]
11
Spark RDD
Read single or multiple text files into single RDD?
● Read multiple specific files and create an RDD[String]
12
Spark RDD
Load CSV File into RDD
● Create .csv files and put them to HDFS
13
Spark RDD
Load CSV File into RDD
● textFile() method read an entire CSV record as a String and returns RDD[String]
14
Spark RDD
Load CSV File into RDD
● we would need every record in a CSV to split by comma delimiter and store it in RDD as
multiple columns, In order to achieve this, we should use map() transformation on RDD where
we will convert RDD[String] to RDD[Array[String] by splitting every record by comma delimiter.
map() method returns a new RDD instead of updating existing.
Each record in the csvData RDD is an Array[String]
f(0), f(1) are the first and second element in the array f
15
Spark RDD
Load CSV File into RDD
● Skip header from csv file
● Command: rdd.mapPartitionsWithIndex { (idx, iter) => if (idx == 0) iter.drop(1) else iter }
16
Spark RDD Operations
● https://spark.apache.org/docs/latest/api/python/reference/pyspark.html
● https://data-flair.training/blogs/spark-rdd-operations-transformations-actions/
17
Spark RDD Operations
Two type of RDD operations
● Apache Spark RDD supports two types of Operations-
○ Transformations
○ Actions
● Spark Transformation is a function that produces new RDD from the existing RDDs. It takes
RDD as input and produces one or more RDD as output. Each time it creates new RDD
when we apply any transformation.
● RDD actions are operations that return the raw values, In other words, any RDD function
that returns other than RDD is considered as an action in spark programming.
18
Spark RDD Operations
RDD Transformation
● Two types of RDD Transformation
○ Narrow transformations are the result of map() and filter() functions
and these compute data that live on a single partition meaning there
will not be any data movement between partitions to execute narrow
transformations.
○ Wider transformations are the result of groupByKey() and
reduceByKey() functions and these compute data that live on many
partitions meaning there will be data movements between partitions
to execute wider transformations.
19
Spark RDD Operations
RDD Transformation
20
Spark RDD Operations
RDD Action
21
Spark RDD Operations
RDD Transformation – Word count example
● Create a .txt file and put it to HDFS
22
Spark RDD Operations
RDD Transformation – Word count example
● Read the file and create an RDD[String]
23
Spark RDD Operations
RDD Transformation – Word count example
● Apply flatMap() Transformation
● flatMap() transformation flattens the RDD after applying the function and returns a new RDD. On the below
example, first, it splits each record by space in an RDD and finally flattens it. Resulting RDD consists of a
single word on each record.
24
Spark RDD Operations
RDD Transformation – Word count example
● Apply map() Transformation
● map() transformation is used the apply any complex operations like adding a column,
updating a column e.t.c, the output of map transformations would always have the same
number of records as input.
● In this word count example, we are adding a new column with value 1 for each word, the
result of the RDD is PairRDD which contains key-value pairs, word of type String as Key
and 1 of type Int as value.
Each record in testfile3 RDD is a tuple [String,Int] (not an array) 25
Spark RDD Operations
RDD Transformation – Word count example
● Let’s print out the content of these RDDs (each record is printed in a single line)
Key Value
26
Spark RDD Operations
RDD Transformation – Word count example
● Apply reduceByKey() Transformation
● reduceByKey() merges the values for each key with the function specified. In our example, it
reduces the word string by applying the sum function on value. The result of our RDD
contains unique words and their count.
● Note: reduceByKey() is a transformation, so count is a RDD
27
Spark RDD Operations
RDD Transformation – Word count example
● Let’s print out the result
28
Spark RDD Operations
RDD Transformation – Word count example
● Another way to use reduceByKey
29
Spark RDD Operations
RDD Transformation – Word count example
● Try sortByKey() Transformation
● sortByKey() transformation is used to sort RDD elements on key.
● In this example, we want to sort RDD elements on the number of each word. Therefore, we
need convert RDD[(String,Int]) to RDD[(Int,String]) using map transformation and then apply
sortByKey which ideally does sort on an integer key value.
Notice how each element in a tuple is accessed. If a is an array, the second and first element are accessed
by a(1) and a(0). In this case, a is a tuple, the second and first element are accessed by a._2 and a._1
30
Spark RDD Operations
RDD Transformation – Word count example
● Let’s print out the result (and see that the count number of each word now become the key)
31
Spark RDD Operations
RDD Transformation – Word count example
● Apply sortByKey() transformation
If there are more than 1 partitions, records in each partitions are sorted, but the
sorted results of multiple partitions might be mixed up.
To avoid this, use sortByKey(numPartitions=1)
32
Spark RDD Operations
RDD Transformation – Word count example
● You can also apply map transformation to switch the word back to the key
33
Spark RDD Operations
RDD Transformation – Word count example
● You can to all these steps in one line of code
34
Spark RDD Operations
RDD Transformation – Word count example
● Or you can use sortBy
35
Spark RDD Operations
RDD Transformation – Word count example
● Apply filter() Transformation
● filter() transformation is used to filter the records in an RDD. In this example we are filtering
all words starts with “t”.
36
Spark RDD Operations
RDD Transformation – Word count example
● Merge two RDD
37
Spark RDD Operations
RDD Action – Easiest Wordcount using countByValue
38
Spark RDD Operations
RDD Action
● Let’s create some RDD
39
Spark RDD Operations
RDD Action
● reduce() - Reduces the elements of the dataset using the specified binary
operator.
40
Spark RDD Operations
RDD Action
● collect() -Return the complete dataset as an Array.
41
Spark RDD Operations
RDD Action
● count() – Return the count of elements in the dataset.
● countApprox() – Return approximate count of elements in the dataset, this method returns
incomplete when execution time meets timeout.
● countApproxDistinct() – Return an approximate number of distinct elements in the dataset.
42
Spark RDD Operations
RDD Action
● first() – Return the first element in the dataset.
● top() – Return top n elements from the dataset.
● min() – Return the minimum value from the dataset.
● max() – Return the maximum value from the dataset.
● take() – Return the first num elements of the dataset.
● takeOrdered() – Return the first num (smallest) elements from the dataset and this is
the opposite of the take() action.
● takeSample() – Return the subset of the dataset in an Array.
43
Spark Pair RDD Functions
Pair RDD Transformation
Spark defines PairRDDFunctions class with several functions to work with Pair RDD or RDD key-value pair
44
Spark Pair RDD Functions
Pair RDD Action
Spark defines PairRDDFunctions class with several functions to work with Pair RDD or RDD key-value pair
45
Spark Pair RDD Functions
Wordcount using reduceByKey
46
Spark Pair RDD Functions
Wordcount using reduceByKey
● Save the output in a text file
47
Spark Pair RDD Functions
Wordcount using reduceByKey
● Check the result
48
Spark Pair RDD Functions
Another usage of reduceByKey
49
Spark Pair RDD Functions
Wordcount using countByKey action
50
Spark Pair RDD Functions
Join two RDDs
● Create two RDDs from text files
51
Spark Pair RDD Functions
Join two RDDs
● Check the result
If you have duplicates key in any RDD, join will result in cartesian product.
52
Spark Pair RDD Functions
Join two RDDs
● Use cogroup or groupByKey to get unique keys in the joining result
53
Spark shell Web UI
Accessing the Web UI of Spark
● Open http://quickstart.cloudera:4040 in web browser
54
Spark shell Web UI
Explore Spark shell Web UI
● Click DAG Visualization
Click
Number of partitions 55
Spark shell Web UI
Explore Spark shell Web UI
● View the Direct Acylic Graph (DAG) of the completed job
56
Spark shell Web UI
Partition and parallelism in RDDs
57
Spark shell Web UI
Partition and parallelism in RDDs
● Execute a parallel task in the shell
58
Spark shell Web UI
Partition and parallelism in RDDs
● Open Spark shell Web UI
Totally 5 partitions are done 59
Spark shell Web UI
Partition and parallelism in RDDs
Parallel execution of 5 completed jobs
60
Q&A
Cảm ơn đã theo dõi
Chúng tôi hy vọng cùng nhau đi đến thành công.
61
Big Data
