Lightbend Fast Data Platform
•
7 likes•1,251 views
Dean Wampler, O’Reilly author and Big Data Strategist in the office of the CTO at Lightbend discusses practical tips for architecting stream-processing applications and explains how you can tame some of the complexity in moving from data at rest to data in motion.
Lightbend Fast Data Platform
- 1. Lightbend Fast Data Platform A Technical Overview
- 2. A Bit of History: Monoliths to Microservices
- 4. • Tangled responsibilities, lead to… • Infrequent, “big-bang” deployments • Runtimes: months to years • Latencies: μsecs to seconds Monoliths
- 6. • Each does one thing, so… • Updates easier to deploy frequently • Message driven & asynchronous • Runtimes: minutes! to forever • Latencies: higher than function calls Microservices
- 9. A Bit of History: Big Data
- 11. YARN HDFS MR job #1 MR job #2 Flume Sqoop DBs Slave Node DiskDiskDiskDiskDisk Node Mgr Data Node Master Resource Manager Name Node Hadoop 2013: Embrace Spark Spark job #1 Spark job #2
- 12. YARN HDFS MR job #1 MR job #2 Flume Sqoop DBs Slave Node DiskDiskDiskDiskDisk Node Mgr Data Node Master Resource Manager Name Node Go beyond batch? Spark job #1 Spark job #2
- 13. The Emergence of Fast Data: (Time Is Money)
- 15. IoT
- 17. Fast Data and Microservices; Are they Converging?
- 18. •Each (stream|μservice): •does one responsibility •has unending (data|messages) Synergies 18 Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
- 19. •Each (stream|μservice): •encourages asynchrony •offers never-ending service Synergies 19 Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
- 20. •Reasons these architectures will converge: 1.Similar design problems 2.Data becomes dominate problem Thesis 20 Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Streaming Streams SQL Batch … Streaming Flink
- 22. 1. An accelerated on ramp for building streaming data systems and applications. Value Three Ways
- 23. 2. Best practices guidance, through sample apps, documentation, and enablement services for what tools and techniques to use for solving specific design problems. Value Three Ways
- 24. 3. Intelligent monitoring and management, driven by machine learning, to keep your systems resilient, scalable, and responsive with minimal user intervention. Value Three Ways
- 25. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 26. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 27. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 28. Service 1 Log & Other Files Internet Services Service 2 Service 3 Services Services N * M links ConsumersProducers
- 29. Service 1 Log & Other Files Internet Services Service 2 Service 3 Services Services N + M links ConsumersProducers
- 30. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 31. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 32. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 33. •Low latency? How low? •High volume? How high? Streaming Tradeoffs (1/3) 33
- 34. •Which kinds of data processing & analytics are required? •How will this processing be done? •Individual processing of events? •Bulk processing of records? Streaming Tradeoffs (2/3) 34
- 35. •Which tools and data sources/sinks must interoperate with your streaming tool? Streaming Tradeoffs (3/3) 35
- 36. 36
- 37. 37 •Low latency •Low volume •Complex flows •Complex Event Processing
- 38. 38 •Med. latency •High volume •Data flows, SQL •En masse processing
- 39. 39 •Low latency •High volume •Data flows, correctness •En masse processing
- 40. 40 •Low latency •Med. volume •ETL, “tables” •Data flow or Per Event
- 41. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 42. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 43. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 44. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 45. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS: On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted FDP.ai Cluster Analysis Machine LearningMicroservices ProducDon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles
- 46. Lightbend Fast Data Platform V1.0 Internet Logs Sockets DC/OS, Marathon On Premise or Cloud FDP Kafka Connect Storage HDFS, S3, … SQL/ NoSQL ES Kafka Streams Cloud Hosted Graviton Cluster Analysis Machine LearningMicroservices ProducCon Suite Machine Learning ML Streaming Streams SQL Batch … Streaming Flink Intelligent Management…Consoles For More Information: lightbend.com/fast-data-platform
- 48. • FDP: lightbend.com/fast-data-platform • Book: bit.ly/lightbend-fast-data