Apache
## 1. Fundamentals
- Lazy Evaluation
- Data Persistence
- Fault Tolerance
- Data Partitioning
## 2. Architecture
1. **Driver Program**
- Creates SparkContext
2. **Cluster Manager**
- Standalone Scheduler
- YARN
- Mesos
- Kubernetes 3. **Worker Nodes**
- Execute tasks
- Cache data
- Optimization opportunities
- Task scheduling
2. **Stage Generation**
- Pipeline operations
- Shuffle boundaries
- Task creation
3. **Task Scheduling**
- Data locality
- Resource allocation
- Load balancing
## 3. Core Abstractions
1. **Characteristics**
- Immutable
- Distributed
- Fault-tolerant
- Lazy evaluation
- Typed 2. **Operations**
- Transformations
- map
- filter
- flatMap
- union
- intersection
- Actions
- collect
- count
- first
- take
- reduce
3. **Persistence Options**
- MEMORY_ONLY
- MEMORY_AND_DISK
- DISK_ONLY
- MEMORY_ONLY_SER
- OFF_HEAP
1. **Structure**
- Named columns
- Schema definition
- Optimized execution
2. **Operations**
- select
- groupBy - join
- orderBy
3. **Optimization**
- Catalyst optimizer
- Code generation
- Predicate pushdown
- Type-safe
- Object-oriented interface
- Encoder-based serialization
- Performance optimizations
## 4. Spark Components
1. **Features**
- SQL interface
- Schema inference
- UDF support
2. **Data Sources**
- Parquet
- ORC
- JSON
- CSV
- JDBC ### 4.2 Spark Streaming
1. **DStream Abstraction**
- Micro-batch processing
- Windowed computations
- Stateful operations
2. **Input Sources**
- Kafka
- Flume
- Kinesis
- TCP sockets
3. **Output Operations**
- foreachRDD
- saveAsTextFiles
- saveAsHadoopFiles
1. **Algorithms**
- Classification
- Regression
- Clustering
- Recommendation
2. **Features**
- Feature engineering
- Pipeline API
- Model persistence
- Evaluation metrics
- Built-in algorithms
- Graph operators
- Graph builders
## 5. Performance Optimization
1. **Memory Architecture**
- Execution memory
- Storage memory
- User memory
- Reserved memory
2. **Tuning Parameters**
- spark.memory.fraction
- spark.memory.storageFraction
- spark.default.parallelism
- Kryo serialization
- Java serialization
- Custom serializers
- Compression settings
1. **Executor Settings**
- Number of executors
- Executor memory
- Executor cores 2. **Driver Settings**
- Driver memory
- Driver cores
- Local directory
## 6. Best Practices
- Partition size
- Number of partitions
- Partition pruning
- Partition schemes
- Broadcast joins
- Shuffle joins
- Sort-merge joins
- Join hints
- Cache levels
- Cache management
- Unpersist timing
- Memory pressure
## 7. Deployment
1. **Standalone Mode**
- Master configuration
- Worker configuration - High availability
2. **YARN Mode**
- Client mode
- Cluster mode
3. **Kubernetes**
- Pod specification
- Service accounts
- Dynamic allocation
1. **Web UI**
- Job progress
- Stage details
- Storage usage
- Executor metrics
2. **Metrics System**
- JMX metrics
- Custom metrics
- Ganglia integration
- Graphite integration
## 8. Advanced Features
- Stream processing
- Continuous processing
- Watermarking - State management
- Automatic scaling
- Resource sharing
- Executor management
- Cost optimization
1. **Authentication**
- Kerberos
- SSL/TLS
- ACLs
2. **Authorization**
- File permissions
- RPC authentication
- Web UI security
