Skip to content

Commit d4e85ec

Browse files
author
Dana Powers
committed
Update docs for release w/ new async classes
1 parent 2a2e77a commit d4e85ec

14 files changed

+283
-238
lines changed

‎docs/apidoc/BrokerConnection.rst

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
BrokerConnection
2+
================
3+
4+
.. autoclass:: kafka.BrokerConnection
5+
:members:

‎docs/apidoc/KafkaClient.rst

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
KafkaClient
2+
===========
3+
4+
.. autoclass:: kafka.KafkaClient
5+
:members:

‎docs/apidoc/KafkaConsumer.rst

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
KafkaConsumer
2+
=============
3+
4+
.. autoclass:: kafka.KafkaConsumer
5+
:members:

‎docs/apidoc/KafkaProducer.rst

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
KafkaProducer
2+
=============
3+
4+
<unreleased> See :class:`kafka.producer.SimpleProducer`

‎docs/apidoc/SimpleProducer.rst

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
SimpleProducer
2+
==============
3+
4+
.. autoclass:: kafka.producer.SimpleProducer
5+
:members:
6+
:show-inheritance:
7+
8+
.. autoclass:: kafka.producer.KeyedProducer
9+
:members:
10+
:show-inheritance:
11+
12+
.. automodule:: kafka.producer.base
13+
:members:
14+
:show-inheritance:

‎docs/apidoc/modules.rst

+7-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
1-
kafka
2-
=====
1+
kafka-python API
2+
****************
33

44
.. toctree::
5-
:maxdepth: 4
65

7-
kafka
6+
KafkaConsumer
7+
KafkaProducer
8+
KafkaClient
9+
BrokerConnection
10+
SimpleProducer

‎docs/compatibility.rst

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
Compatibility
2+
-------------
3+
4+
.. image:: https://img.shields.io/badge/kafka-0.9%2C%200.8.2%2C%200.8.1%2C%200.8-brightgreen.svg
5+
:target: https://kafka-python.readthedocs.org/compatibility.html
6+
.. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg
7+
:target: https://pypi.python.org/pypi/kafka-python
8+
9+
kafka-python is compatible with (and tested against) broker versions 0.9.0.0
10+
through 0.8.0 . kafka-python is not compatible with the 0.8.2-beta release.
11+
12+
kafka-python is tested on python 2.6, 2.7, 3.3, 3.4, 3.5, and pypy.
13+
14+
Builds and tests via Travis-CI. See https://travis-ci.org/dpkp/kafka-python

‎docs/conf.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@
4949

5050
# General information about the project.
5151
project = u'kafka-python'
52-
copyright = u'2015 - David Arthur, Dana Powers, and Contributors'
52+
copyright = u'2016 -- Dana Powes, David Arthur, and Contributors'
5353

5454
# The version info for the project you're documenting, acts as replacement for
5555
# |version| and |release|, also used in various other places throughout the

‎docs/index.rst

+63-43
Original file line numberDiff line numberDiff line change
@@ -1,66 +1,86 @@
11
kafka-python
2-
============
2+
############
33

4-
This module provides low-level protocol support for Apache Kafka as well as
5-
high-level consumer and producer classes. Request batching is supported by the
6-
protocol as well as broker-aware request routing. Gzip and Snappy compression
7-
is also supported for message sets.
4+
.. image:: https://img.shields.io/badge/kafka-0.9%2C%200.8.2%2C%200.8.1%2C%200.8-brightgreen.svg
5+
:target: https://kafka-python.readthedocs.org/compatibility.html
6+
.. image:: https://img.shields.io/pypi/pyversions/kafka-python.svg
7+
:target: https://pypi.python.org/pypi/kafka-python
8+
.. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github
9+
:target: https://coveralls.io/github/dpkp/kafka-python?branch=master
10+
.. image:: https://travis-ci.org/dpkp/kafka-python.svg?branch=master
11+
:target: https://travis-ci.org/dpkp/kafka-python
12+
.. image:: https://img.shields.io/badge/license-Apache%202-blue.svg
13+
:target: https://github.com/dpkp/kafka-python/blob/master/LICENSE
814

9-
Coordinated Consumer Group support is under development - see Issue #38.
15+
>>> pip install kafka-python
1016

11-
On Freenode IRC at #kafka-python, as well as #apache-kafka
17+
kafka-python is a client for the Apache Kafka distributed stream processing
18+
system. It is designed to function much like the official java client, with a
19+
sprinkling of pythonic interfaces (e.g., iterators).
1220

13-
For general discussion of kafka-client design and implementation (not python specific),
14-
see https://groups.google.com/forum/m/#!forum/kafka-clients
1521

16-
For information about Apache Kafka generally, see https://kafka.apache.org/
22+
KafkaConsumer
23+
*************
1724

18-
Status
19-
------
25+
>>> from kafka import KafkaConsumer
26+
>>> consumer = KafkaConsumer('my_favorite_topic')
27+
>>> for msg in consumer:
28+
... print (msg)
2029

21-
The current stable version of this package is `0.9.5 <https://github.com/dpkp/kafka-python/releases/tag/v0.9.5>`_ and is compatible with:
30+
:class:`~kafka.consumer.KafkaConsumer` is a full-featured,
31+
high-level message consumer class that is similar in design and function to the
32+
new 0.9 java consumer. Most configuration parameters defined by the official
33+
java client are supported as optional kwargs, with generally similar behavior.
34+
Gzip and Snappy compressed messages are supported transparently.
2235

23-
Kafka broker versions
36+
In addition to the standard
37+
:meth:`~kafka.consumer.KafkaConsumer.poll` interface (which returns
38+
micro-batches of messages, grouped by topic-partition), kafka-python supports
39+
single-message iteration, yielding :class:`~kafka.consumer.ConsumerRecord`
40+
namedtuples, which include the topic, partition, offset, key, and value of each
41+
message.
2442

25-
* 0.9.0.0
26-
* 0.8.2.2
27-
* 0.8.2.1
28-
* 0.8.1.1
29-
* 0.8.1
30-
* 0.8.0
43+
By default, :class:`~kafka.consumer.KafkaConsumer` will attempt to auto-commit
44+
message offsets every 5 seconds. When used with 0.9 kafka brokers,
45+
:class:`~kafka.consumer.KafkaConsumer` will dynamically assign partitions using
46+
the kafka GroupCoordinator APIs and a
47+
:class:`~kafka.coordinator.assignors.roundrobin.RoundRobinPartitionAssignor`
48+
partitioning strategy, enabling relatively straightforward parallel consumption
49+
patterns. See :doc:`usage` for examples.
3150

32-
Python versions
3351

34-
* 3.5 (tested on 3.5.0)
35-
* 3.4 (tested on 3.4.2)
36-
* 3.3 (tested on 3.3.5)
37-
* 2.7 (tested on 2.7.9)
38-
* 2.6 (tested on 2.6.9)
39-
* pypy (tested on pypy 2.5.0 / python 2.7.8)
52+
KafkaProducer
53+
*************
4054

41-
License
42-
-------
55+
TBD
4356

44-
Apache License, v2.0. See `LICENSE <https://github.com/dpkp/kafka-python/blob/master/LICENSE>`_.
4557

46-
Copyright 2015, David Arthur, Dana Powers, and Contributors
47-
(See `AUTHORS <https://github.com/dpkp/kafka-python/blob/master/AUTHORS.md>`_).
58+
Protocol
59+
********
4860

61+
A secondary goal of kafka-python is to provide an easy-to-use protocol layer
62+
for interacting with kafka brokers via the python repl. This is useful for
63+
testing, probing, and general experimentation. The protocol support is
64+
leveraged to enable a :meth:`~kafka.KafkaClient.check_version()`
65+
method that probes a kafka broker and
66+
attempts to identify which version it is running (0.8.0 to 0.9).
67+
68+
69+
Low-level
70+
*********
71+
72+
Legacy support is maintained for low-level consumer and producer classes,
73+
SimpleConsumer and SimpleProducer.
4974

50-
Contents
51-
--------
5275

5376
.. toctree::
77+
:hidden:
5478
:maxdepth: 2
5579

56-
usage
80+
Usage Overview <usage>
81+
API </apidoc/modules>
5782
install
5883
tests
59-
API reference </apidoc/modules>
60-
61-
Indices and tables
62-
==================
63-
64-
* :ref:`genindex`
65-
* :ref:`modindex`
66-
* :ref:`search`
84+
compatibility
85+
support
86+
license

‎docs/install.rst

+6-6
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
Install
2-
=======
2+
#######
33

44
Install with your favorite package manager
55

66
Latest Release
7-
--------------
7+
**************
88
Pip:
99

1010
.. code:: bash
@@ -15,7 +15,7 @@ Releases are also listed at https://github.com/dpkp/kafka-python/releases
1515

1616

1717
Bleeding-Edge
18-
-------------
18+
*************
1919

2020
.. code:: bash
2121
@@ -39,10 +39,10 @@ Using `setup.py` directly:
3939
4040
4141
Optional Snappy install
42-
-----------------------
42+
***********************
4343

4444
Install Development Libraries
45-
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
45+
=============================
4646

4747
Download and build Snappy from http://code.google.com/p/snappy/downloads/list
4848

@@ -70,7 +70,7 @@ From Source:
7070
sudo make install
7171
7272
Install Python Module
73-
^^^^^^^^^^^^^^^^^^^^^
73+
=====================
7474

7575
Install the `python-snappy` module
7676

‎docs/license.rst

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
License
2+
-------
3+
4+
.. image:: https://img.shields.io/badge/license-Apache%202-blue.svg
5+
:target: https://github.com/dpkp/kafka-python/blob/master/LICENSE
6+
7+
Apache License, v2.0. See `LICENSE <https://github.com/dpkp/kafka-python/blob/master/LICENSE>`_.
8+
9+
Copyright 2016, David Arthur, Dana Powers, and Contributors
10+
(See `AUTHORS <https://github.com/dpkp/kafka-python/blob/master/AUTHORS.md>`_).

‎docs/support.rst

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
Support
2+
-------
3+
4+
For support, see github issues at https://github.com/dpkp/kafka-python
5+
6+
Limited IRC chat at #kafka-python on freenode (general chat is #apache-kafka).
7+
8+
For information about Apache Kafka generally, see https://kafka.apache.org/
9+
10+
For general discussion of kafka-client design and implementation (not python
11+
specific), see https://groups.google.com/forum/m/#!forum/kafka-clients

‎docs/tests.rst

+50-26
Original file line numberDiff line numberDiff line change
@@ -1,59 +1,83 @@
11
Tests
22
=====
33

4-
Run the unit tests
5-
------------------
4+
.. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github
5+
:target: https://coveralls.io/github/dpkp/kafka-python?branch=master
6+
.. image:: https://travis-ci.org/dpkp/kafka-python.svg?branch=master
7+
:target: https://travis-ci.org/dpkp/kafka-python
68

7-
.. code:: bash
9+
Test environments are managed via tox. The test suite is run via pytest.
10+
Individual tests are written using unittest, pytest, and in some cases,
11+
doctest.
12+
13+
Linting is run via pylint, but is generally skipped on python2.6 and pypy
14+
due to pylint compatibility / performance issues.
15+
16+
For test coverage details, see https://coveralls.io/github/dpkp/kafka-python
817

9-
tox
18+
The test suite includes unit tests that mock network interfaces, as well as
19+
integration tests that setup and teardown kafka broker (and zookeeper)
20+
fixtures for client / consumer / producer testing.
21+
22+
23+
Unit tests
24+
------------------
1025

26+
To run the tests locally, install tox -- `pip install tox`
27+
See http://tox.readthedocs.org/en/latest/install.html
1128

12-
Run a subset of unit tests
13-
--------------------------
29+
Then simply run tox, optionally setting the python environment.
30+
If unset, tox will loop through all environments.
1431

1532
.. code:: bash
1633
34+
tox -e py27
35+
tox -e py35
36+
1737
# run protocol tests only
1838
tox -- -v test.test_protocol
1939
20-
# test with pypy only
21-
tox -e pypy
40+
# re-run the last failing test, dropping into pdb
41+
tox -e py27 -- --lf --pdb
42+
43+
# see available (pytest) options
44+
tox -e py27 -- --help
2245
23-
# Run only 1 test, and use python 2.7
24-
tox -e py27 -- -v --with-id --collect-only
2546
26-
# pick a test number from the list like #102
27-
tox -e py27 -- -v --with-id 102
47+
Integration tests
48+
-----------------
2849

50+
.. code:: bash
2951
30-
Run the integration tests
31-
-------------------------
52+
KAFKA_VERSION=0.9.0.0 tox -e py27
53+
KAFKA_VERSION=0.8.2.2 tox -e py35
3254
33-
The integration tests will actually start up real local Zookeeper
34-
instance and Kafka brokers, and send messages in using the client.
3555
36-
First, get the kafka binaries for integration testing:
56+
Integration tests start Kafka and Zookeeper fixtures. This requires downloading
57+
kafka server binaries:
3758

3859
.. code:: bash
3960
4061
./build_integration.sh
4162
42-
By default, the build_integration.sh script will download binary
43-
distributions for all supported kafka versions.
44-
To test against the latest source build, set KAFKA_VERSION=trunk
45-
and optionally set SCALA_VERSION (defaults to 2.8.0, but 2.10.1 is recommended)
63+
By default, this will install 0.8.1.1, 0.8.2.2, and 0.9.0.0 brokers into the
64+
servers/ directory. To install a specific version, set `KAFKA_VERSION=1.2.3`:
4665

4766
.. code:: bash
4867
49-
SCALA_VERSION=2.10.1 KAFKA_VERSION=trunk ./build_integration.sh
68+
KAFKA_VERSION=0.8.0 ./build_integration.sh
5069
5170
Then run the tests against supported Kafka versions, simply set the `KAFKA_VERSION`
5271
env variable to the server build you want to use for testing:
5372

5473
.. code:: bash
5574
56-
KAFKA_VERSION=0.8.0 tox
57-
KAFKA_VERSION=0.8.1 tox
58-
KAFKA_VERSION=0.8.1.1 tox
59-
KAFKA_VERSION=trunk tox
75+
KAFKA_VERSION=0.9.0.0 tox -e py27
76+
77+
To test against the kafka source tree, set KAFKA_VERSION=trunk
78+
[optionally set SCALA_VERSION (defaults to 2.10)]
79+
80+
.. code:: bash
81+
82+
SCALA_VERSION=2.11 KAFKA_VERSION=trunk ./build_integration.sh
83+
KAFKA_VERSION=trunk tox -e py35

0 commit comments

Comments
 (0)