On Thursday we released a new version of Apache Kafka that dramatically strengthens the semantic guarantees it provides. This release came at the tail end of several years of thinking through how to do reliable stream processing in a way that is fast, practical, and correct. The implementation effort itself was on the order of about a year, including an extended period in which about a hundred pag
This document discusses exactly once semantics in Apache Kafka 0.11. It provides an overview of how Kafka achieved exactly once delivery between producers and consumers. Key points include: - Kafka 0.11 introduced exactly once semantics with changes to support transactions and deduplication. - Producers can write in a transactional fashion and receive acknowledgments of committed writes from broke
The Lambda Architecture has its merits, but alternatives are worth exploring. Nathan Marz wrote a popular blog post describing an idea he called the Lambda Architecture (“How to beat the CAP theorem“). The Lambda Architecture is an approach to building stream processing applications on top of MapReduce and Storm or similar systems. This has proven to be a surprisingly popular idea, with a dedicate
I try to use Kafka with version 0.9.0 with port 9092. If I use telnet, I successfully connect to this address, but I fail to connect to Kafka server with Java API Here is my Java example exactly use the official supplied documentation: Properties props = new Properties(); props.put("bootstrap.servers", "192.168.174.128:9092"); props.put("acks", "all"); props.put("retries", 0); props.put("batch.siz
Trifecta is a Command Line Interface (CLI) tool that enables users to quickly and easily inspect, publish and verify messages (or data) in Kafka, Storm and Zookeeper View project on GitHub Trifecta is a Command Line Interface (CLI) tool that enables users to quickly and easily inspect, verify and even query Kafka messages. In addition, Trifecta offers data import/export functions for transferring
This document provides a summary of Amazon Kinesis and Apache Kafka, two platforms for processing real-time streaming data at large scale. It describes key features of each system such as durability, interfaces, processing options, and deployment. Kinesis is a fully managed cloud service that provides high durability for data across AWS availability zones. Kafka is an open source platform that off
Let’s take an example. Consider a Facebook-like social networking app (albeit a completely hypothetical one) that updates the profiles database when a user updates their Facebook profile. There are several applications that need to be notified when a user updates their profile — the search application so the user’s profile can be reindexed to be searchable on the changed attribute; the newsfeed ap
When Apache Kafka® was originally created, it shipped with a Scala producer and consumer client. Over time we came to realize many of the limitations of these APIs. For example, we had a “high-level” consumer API which supported consumer groups and handled failover, but didn’t support many of the more complex usage scenarios. We also had a “simple” consumer client which provided full control, but
Apache Kafka is a high-throughput distributed message system that is being adopted by hundreds of companies to manage their real-time data. Companies use Kafka for many applications (real time stream processing, data synchronization, messaging, and more), but one of the most popular applications is ETL pipelines. Kafka is a perfect tool for building data pipelines: it’s reliable, scalable, and eff
"multas per gentes et multa per aequora" [1] The life of a request to CloudFlare begins and ends at the edge. But the afterlife! Like Catullus to Bithynia, the log generated by an HTTP request or a DNS query has much, much further to go. This post comes from CloudFlare's Data Team. It reports the state of processing these sort of edge logs, including what's worked well for us and what remains a ch
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
