Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: dpkp/kafka-python
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: master
Choose a base ref
...
head repository: hackaugusto/kafka-python
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 5 commits
  • 7 files changed
  • 3 contributors

Commits on Jan 11, 2021

  1. Avoid 100% CPU usage while socket is closed (sleep)

    After stop/start kafka service, kafka-python may use 100% CPU caused by
    busy-retry while the socket was closed. This fix the issue by sleep 0.1
    second if the fd is negative.
    orange-kao authored and hackaugusto committed Jan 11, 2021
    Configuration menu
    Copy the full SHA
    b061f3d View commit details
    Browse the repository at this point in the history
  2. Support connecting through SOCKS5 proxies (#2169)

    Implement support for SOCKS5 proxies. Implement a new proxy wrapper
    that handles SOCKS5 connection, authentication and requesting
    connections to the actual Kafka broker endpoint.
    
    The proxy can be configured via a new keyword argument `socks5_proxy`
    to consumers, producers or admin client. The value is URL with optional
    username and password. E.g.
    `socks5://user:secret@proxy.example.com:10800`
    
    The implementation is done in state machine that emulates repeated
    calls to connect_ex. The rationale with this design is minimal changes
    to the actual BrokerConnection object.
    hnousiainen authored and hackaugusto committed Jan 11, 2021
    Configuration menu
    Copy the full SHA
    346c01e View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2021

  1. bugfix: fix infinite loop on KafkaAdminClient

    An infinite loop may happen with the following pattern:
    
        self._send_request_to_node(self._client.least_loaded_node(), request)
    
    The problem happens when `self._client`'s cluster metadata is out-of-date, and the
    result of `least_loaded_node()` is a node that has been removed from the cluster but
    the client is unware of it. When this happens `_send_request_to_node` will enter an
    infinite loop waiting for the chosen node to become available, which won't happen,
    resulting in an infinite loop.
    
    This commit introduces a new method named `_send_request_to_least_loaded_node` which
    handles the case above. This is done by regularly checking if the target node is
    available in the cluster metadata, and if not, a new node is chosen.
    
    Notes:
    
    - This does not yet cover every call site to `_send_request_to_node`, there are some
      other places were similar race conditions may happen.
    - The code above does not guarantee that the request itself will be sucessful, since
      it is still possible for the target node to exit, however, it does remove the
      infinite loop which can render client code unusable.
    hackaugusto committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    1e2ac76 View commit details
    Browse the repository at this point in the history
  2. bugfix: race among _connecting and cluster metadata

    A call to `maybe_connect` can be performed while the cluster metadata is
    being updated. If that happens, the assumption that every entry in
    `_connecting` has metadata won't hold. The existing assert will then
    raise on every subsequent call to `poll` driving the client instance
    unusable.
    
    This fixes the issue by ignoring connetion request to nodes that do not
    have the metadata available anymore.
    hackaugusto committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    8c6ab20 View commit details
    Browse the repository at this point in the history
  3. bugfix: infinite loop when send msgs to controller

    If the value `_controller_id` is out-of-date and the node was removed
    from the cluster, `_send_request_to_node` would enter an infinite loop.
    hackaugusto committed Jan 14, 2021
    Configuration menu
    Copy the full SHA
    2315638 View commit details
    Browse the repository at this point in the history
Loading