Data pipelines

from zero to solid

Lars Albertsson
www.mapflat.com

1
Who’s talking?
Swedish Institute of Computer Science (test tools)
Sun Microsystems (very large machines)
Google (Hangouts, productivity)
Recorded Future (NLP startup)
Cinnober Financial Tech. (trading systems)
Spotify (data processing & modelling)
Schibsted (data processing & modelling)
Independent data engineering consultant
2
Presentation goals
● Overview of data pipelines for analytics / data products
● Target audience: Big data starters
○ Seen wordcount, need the stuff around
● Overview of necessary components & wiring
● Base recipe
○ In vicinity of state-of-practice
○ Baseline for comparing design proposals
● Subjective best practices - not single truth
● Technology suggestions, (alternatives)
3
Presentation non-goals
● Stream processing
○ High complexity in practice
○ Batch processing yields > 90% of value
● Technology enumeration or (fair) comparison
● Writing data processing code
○ Already covered en masse

4
 Data product anatomy
Pipeline Export
Unified log
Job Dataset

DB Service
Service

Cluster storage
DB
DB Data Business
lake intelligence
Ingress ETL Egress

5
 Computer program anatomy
Input data Execution path Output
Function Variable data

Lookup
File
structure
HID

RAM
File
File
Window
Input Process Output

6
Data pipeline = yet another program
Don’t veer from best practices
● Regression testing
● Design: Separation of concerns, modularity, etc
● Process: CI/CD, code review, lint tools
● Avoid anti-patterns: Global state, hard-coding location,
duplication, ...
In data engineering, slipping is the norm... :-(
Solved by mixing strong software engineers with data
engineers/scientists. Mutual respect is crucial.
7
 Unified log
Immutable events, append-only,
Event collection source of truth

Unreliable
(Secor,
Reliable, simple, Camus)
write available

Service

Unreliable Bus with history Cluster storage

Kafka HDFS
(Kinesis, (NFS, S3, Google CS, C*)
Google Pub/Sub)

Immediate handoff to append-only replicated log.

Once in the log, events eventually arrive in storage. 8
 Events are safe
Event registration from here

Service
(unimportant)

Service
(important)
Replicated bus
with history
Unified log

Asynchronous fire-and-forget handoff for unimportant data.

Synchronous, replicated, with ack for important data 9
 Event transportation

Cluster storage
HDFS
Bus-to-bus WAN mirror (NFS, S3, Google CS, C*)
expect delays

Log has long history (months+) => robustness end to end.

Avoid risk of processing & decoration. Except timestamps. 10
Event arrival (Secor,
Camus)

clicks/2016/02/08/14

clicks/2016/02/08/15

Cluster storage

Bundle incoming events into datasets

● Sealed quickly, thereafter immutable
● Bucket on arrival / wall-clock time
● Predictable bucketing, e.g. hour 11
Database state collection
Source of truth sometimes in database.
Snapshot to cluster storage.
Easy on surface...

Service

Cluster storage
Service DB ? HDFS
(NFS, S3, Google CS, C*)

DB
12
 Anti-pattern: Send the oliphants!
● Sqoop (dump with MapReduce) production DB
● MapReduce from production API
Hadoop / Spark == internal DDoS service

Service

Cluster storage
Service HDFS
DB
(NFS, S3, Google CS, C*)

DB
Our preciousss 13
Deterministic slaves

Restore
Service

backup
DB snapshot DB

Restore backup to offline slave Cluster storage

HDFS
(NFS, S3, Google CS, C*)
+ Standard procedure
- Serial or resource consuming
14
Using snapshots
DB DB’
join?

● join(event, snapshot) => always time mismatch

● Usually acceptable
● Some behaviour difficult to catch with snapshots
○ E.g. user creates, then deletes account

15
Event sourcing
DB DB’

● Every change to unified log == source of truth

● snapshot(t + 1) = sum(snapshot(t), events(t, t+1))
● Allows view & join at any point in time

Application services still need DB for current state lookup

16
Event sourcing, synced database
A. Service interface generates events

API
and DB transactions

B. Generate stream from commit log

API
Postgres, MySQL -> Kafka

C. Build DB with stream processing

API
17
DB snapshot lessons learnt
● Put fences between online and offline components
○ The latter can kill the former
● Team that owns a database/service must own exporting
data to offline
○ Protect online stability
○ Affects choice of DB technology

18
The data lake
Unified log + snapshots
● Immutable datasets
● Raw, unprocessed
● Source of truth from batch
processing perspective Cluster storage
● Kept as long as permitted Data lake
● Technically homogeneous

19
Datasets
● Pipeline equivalent of objects
● Dataset class == homogeneous records, open-ended
○ Compatible schema
○ E.g. MobileAdImpressions
● Dataset instance = dataset class + parameters
○ Immutable
○ E.g. MobileAdImpressions(hour=”2016-02-06T13”)

20
Representation - data lake & pipes
● Directory with multiple files
○ Parallel processing
○ Sealed with _SUCCESS (Hadoop convention)
○ Bundled schema format
■ JSON lines, Avro, Parquet
○ Avoid old, inadequate formats
■ CSV, XML
○ RPC formats lack bundled schema
■ Protobuf, Thrift
21
 Directory datasets
Privacy Dataset Schema Instance parameters, Seal Partitions
level class version Hive convention

hdfs://red/pageviews/v1/country=se/year=2015/month=11/day=4/_SUCCESS
part-00000.json
part-00001.json

● Some tools, e.g. Spark, understand Hive name

conventions
22
Ingress / egress representation
Larger variation:
● Single file
● Relational database table
● Cassandra column family, other NoSQL
● BI tool storage
● BigQuery, Redshift, ...
Egress datasets are also atomic and immutable.
E.g. write full DB table / CF, switch service to use it, never
change it.
23
Schemas
● There is always a schema
○ Plan your evolution
● New field, same semantic == compatible change
● Incompatible schema change => new dataset class
● Schema on read - assumptions in code
○ Dynamic typing
○ Quick schema changes possible
● Schema on write - enumerated fields
○ Static typing & code generation possible
○ Changes must propagate down pipeline code 24
 Schema on read or write?
Production stability important here Export

DB Service
Service

Business
DB Change agility important here intelligence
DB
25
Batch processing Artifact of business value
E.g. service index

Data lake Pipeline

Gradual refinement Job
1. Wash
- time shuffle, dedup, ...
2. Decorate
- geo, demographic, ...
3. Domain model
- similarity, clusters, ...
4. Application model
- Recommendations, ...
26
Batch job code
● Components should scale up
○ Spark, (Scalding, Crunch)
● And scale down
○ More important!
○ Component should support local mode
■ Integration tests
■ Small jobs - less risk, easier debugging

27
Language choice
● People and community thing, not a technical thing
● Need for simple & quick experiments
○ Java - too much ceremony and boilerplate
● Stable and static enough for production
○ Python/R - too dynamic
● Scala connects both worlds
○ Current home of data innovation
● Beware of complexity - keep it sane and simple
○ Avoid spaceships: <|*|> |@| <**>
28
Batch job
Job == function([input datasets]): [output datasets]
● No orthogonal concerns
○ Invocation
○ Scheduling
○ Input / output location q

● Testable
● No other input factors
● No side-effects
● Ideally: atomic, deterministic, idempotent
29
Batch job class & instance
● Pipeline equivalent of Command pattern
● Parameterised
○ Higher order, c.f. dataset class & instance
○ Job instance == job class + parameters
○ Inputs & outputs are dataset classes
● Instances are ideally executed when input appears
○ Not on cron schedule

30
Pipelines
Data lake Intermediate
● Things will break
○ Input will be missing
○ Jobs will fail
○ Jobs will have bugs
● Datasets must be rebuilt
● Determinism,
idempotency Cluster storage

● Backfill missing / failed Pristine, Derived,

immutable regenerable
● Eventual correctness datasets
31
Workflow manager
● Dataset “build tool”
● Run job instance when
○ input is available DB
○ output missing
○ resources are available
● Backfill for previous failures
● DSL describes DAG
● Includes ingress & egress
Luigi, (Airflow, Pinball)
32
DSL DAG example (Luigi) Actions
class ClientActions(SparkSubmitTask):
hour = DateHourParameter()
UserDB
def requires(self):
return [Actions(hour=self.hour - timedelta(hours=h)) for h in range(0, 12)] + \
[UserDB(date=self.hour.date)]
...

Job (aka Task) classes Time shuffle,

class ClientSessions(SparkSubmitTask):
hour = DateHourParameter() user decorate
def requires(self):
return [ClientActions(hour=self.hour - timedelta(hours=h)) for h in range(0, 3)] ClientActions
...
Form sessions
class SessionsABResults(SparkSubmitTask):
hour = DateHourParameter()
def requires(self):
ClientSessions A/B tests
return [ClientSessions(hour=self.hour), ABExperiments(hour=self.hour)]
A/B compare
def output(self):
return HdfsTarget(“hdfs://production/red/ab_sessions/v1/” +
“{:year=%Y/month=%m/day=%d/hour=%H}”.format(self.hour))
A/B session
...
Dataset instance evaluation

● Expressive, embedded DSL - a must for ingress, egress

○ Avoid weak DSL tools: Oozie, AWS Data Pipeline 33
Egress datasets
● Serving
○ Precomputed user query answers
○ Denormalised
○ Cassandra, (many)
● Export & Analytics
○ SQL (single node / Hive, Presto, ..)
○ Workbenches (Zeppelin)
○ (Elasticsearch, proprietary OLAP)
● BI / analytics tool needs change frequently
○ Prepare to redirect pipelines 34
Test strategy considerations
● Developer productivity is the primary value of test
automation
● Test at stable interface
○ Minimal maintenance
○ No barrier to refactorings
● Focus: single job + end to end
○ Jobs & pipelines are pure functions - easy to test
● Component, unit - only if necessary
○ Avoid dependency injection ceremony
35
Testing single job Runs well in
CI / from IDE

Standard Scalatest harness

1. Generate input 2. Run in local mode 3. Verify output

f() p()

file://test_input/ Job file://test_output/

Don’t commit - ● (Tool-specific frameworks, e.g. for Spark?)

expensive to maintain. ○ Usable, but rarely cover I/O - home of many bugs.
○ Tied to processing technology
Generate / verify with
code. 36
 Testing pipelines - two options
+ Runs in CI
+ Runs in IDE
A: Standard Scalatest harness + Quick setup
- Multi-job
1. Generate input 2. Run custom multi-job 3. Verify output maintenance

f() Test job with sequence of jobs p()

file://test_input/ file://test_output/

f() p()
+ Tests workflow logic
+ More authentic
- Workflow mgr setup
B: Customised workflow manager setup for testability
- Difficult to debug
- Dataset handling
with Python
● Both can be extended with Kafka, egress DBs 37
Deployment
Hg/git
my-pipe-7.tar.gz
repo Luigi DSL, jars, config HDFS

All that a pipeline needs, installed atomically > pip install my-pipe-7.tar.gz

* 10 * * * bin/my_pipe_daily \
Redundant cron schedule, higher --backfill 14
frequency + backfill (Luigi range tools)

Worker Luigi
Worker daemon
Worker
Worker
Worker
Worker
Worker
Worker

38
Continuous deployment
Hg/git
my-pipe-7.tar.gz
repo Luigi DSL, jars, config HDFS

● Poll and pull latest on worker nodes my_cd.py hdfs://pipelines/

○ virtualenv package/version
> virtualenv my_pipe/7
■ No need to sync > pip install my-pipe-7.tar.gz

environment & versions * 10 * * * my_pipe/7/bin/*

○ Cron package/latest/bin/*
Worker
■ Old versions run pipelines to
completion, then exit
39
Start lean: assess needs
Your data & your jobs:
A. Fit in one machine, and will continue to do so
B. Fit in one machine, but grow faster than Moore’s law
C. Do not fit in one machine

● Most datasets / jobs: A

○ Even at large companies with millions of users
● cost(C) >> cost(A)
● Running A jobs on C infrastructure is expensive
40
Lean MVP
● Start simple, lean, end-to-end
○ No parallel cluster computations necessary?
○ Custom jobs or local Spark/Scalding/Crunch
● Shrink data
○ Downsample
○ Approximate algorithms (e.g. Count-min sketch)
● Get workflows running
○ Serial jobs on one/few machines
○ Simple job control (Luigi only / simple work queue)
41
Scale carefully
● Get end-to-end workflows in production for evaluation
○ Improvements driven by business value, not tech
● Keep focus small
○ Business value
○ Privacy needs attention early
● Keep iterations swift
○ Integration test end-to-end
○ Efficient code/test/deploy cycle
● Parallelise jobs only when forced
42
Protecting privacy in practice
● Removing old personal identifiable information (PII)
● Right to be forgotten
● Access control to PII data
● Audit of access and processing

● PII content definition is application-specific

● PII handling subject to business priorities
○ But you should have a plan from day one

43
Data retention
● Remove old, promote derived datasets to lake
Data lake Derived Data lake Derived

Cluster storage Cluster storage

44
PII removal Split out PII,
wash on user
deletion
Key on PII => difficult to wash 34ac,bobwhite
56bd,null

bobwhite,http://site_a/,2015-01-03T bobwhite,Bath,uk 34ac,http://site_a/,2015-01-03T 34ac,Bath,uk

bobwhite,http://site_b/,2015-01-03T joeblack,Bristol,uk 34ac,http://site_b/,2015-01-03T 56db,Bristol,uk
joeblack,http://site_c/,2015-01-03T 56bd,http://site_c/,2015-01-03T

bobwhite,http://site_a/,2015-01-03T,Bath,
bobwhite,http://site_a/,2015-01-03T,Bath,uk uk
bobwhite,http://site_b/,2015-01-03T,Bath,uk bobwhite,http://site_b/,2015-01-03T,Bath,
joeblack,http://site_c/,2015-01-03T,Bristol,uk uk
null,http://site_c/,2015-01-03T,Bristol,uk

● Must rebuild downstream datasets regularly

○ In order for PII to be washed in x days
45
Simple PII audit
● Classify PII level
○ Name, address, messages, ...
○ IP, city, ...
○ Total # page views, …
● Tag datasets and jobs in code
● Manual access through gateway tool
○ Verify permission, log
○ Dedicated machines only
● Log batch jobs
○ Deploy with CD only, log hg/git commit hash 46
Parting words + sales plug
Keep things simple; batch, homogeneity & little state
Focus on developer code, test, debug cycle - end to end
Harmony with technical ecosystems
Little technology overlap with yesterday - follow leaders
Plan early: Privacy, retention, audit, schema evolution

Please give feedback -- mapflat.com/feedback

I help companies plan and build these things
47
Bonus slides

48
Cloud or not?
+ Operations
+ Security
+ Responsive scaling
- Development workflows
- Privacy
- Vendor lock-in
Security?
● Afterthought add-on for big data components
○ E.g. Kerberos support
○ Always trailing - difficult to choose global paradigm
● Container security simpler
○ Easy with cloud
○ Immature with on-premise solutions?

50
Data pipelines example

Raw Derived
Views with Conversion
Users
demographics analytics

Page Sales with

views demographics

Sales
Sales
reports
51
Data pipelines team organisation

Form teams that are driven by business cases & need

Forward-oriented -> filters implicitly applied
Beware of: duplication, tech chaos/autonomy, privacy loss
Conway’s law

“Organizations which design systems ... are

constrained to produce designs which are
copies of the communication structures of
these organizations.”

Better organise to match desired design, then.

Personae - important characteristics
Architect Backend developer
- Technology updated - Simplicity oriented
- Holistic: productivity, privacy - Engineering practices obsessed
- Identify and facilitate governance - Adapt to data world

Data scientist Product owner

- Capable programmer - Trace business value to
- Product oriented upstream design
- Find most ROI through difficult
questions

Manager Devops
- Explain what and why - Always increase automation
- Facilitate process to determine how - Enable, don’t control
- Enable, enable, enable
 Protect production servers
Service

DB offline Cluster storage

slave HDFS
(NFS, S3, Google CS, C*)

+ Online service is safe

- Replication may be out of sync
- Cluster storage may be write unavailable
=> Delayed, inaccurate snapshot
55
 Deterministic slaves
Service Incremental, Service
controlled replay Restore

commit backup
DB DB DB DB
log snapshot

+ Deterministic + Standard procedure

- Ad-hoc solution - Serial or resource
- Serial => not scalable consuming

56
PII privacy control
● Simplify with coarse classification (red/yellow/green)
○ Datasets, potentially fields
○ Separate production areas
● Log batch jobs
○ Code checksum -> commit id -> source code
○ Tag job class with classification
■ Aids PII consideration in code review
■ Enables ad-hoc verification

57
Audit
● Audit manual access
● Wrap all functionality in gateway tool
○ Log datasets, output, code used
○ Disallow download to laptop
○ Wrapper tool happens to be great for enabling data
scientists, too - shields them from operations.

58