oremanj · oremanj · Jan 12, 2022 · Jan 12, 2022 · Jan 12, 2022 · Jan 12, 2022
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -0,0 +1,34 @@
+name: CI
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+
+jobs:
+  Ubuntu:
+    name: 'Ubuntu (${{ matrix.python }})'
+    timeout-minutes: 10
+    runs-on: 'ubuntu-latest'
+    strategy:
+      fail-fast: false
+      matrix:
+        python:
+          - '3.6'
+          - '3.7'
+          - '3.8'
+          - '3.9'
+          - '3.10'
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Setup python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ fromJSON(format('["{0}", "{1}"]', format('{0}.0-alpha - {0}.X', matrix.python), matrix.python))[startsWith(matrix.python, 'pypy')] }}
+      - name: Run tests
+        run: ./ci.sh
+        env:
+          # Should match 'name:' up above
+          JOB_NAME: 'Ubuntu (${{ matrix.python }})'
diff --git a/CHANGES.txt b/CHANGES.txt
@@ -1,4 +1,13 @@
-v0.8.1 30 Jan 2017
+v0.9.0, unreleased
+    Improve usability when Packet objects are retained past the callback
+    Add Packet.retain() to save the packet contents in such cases
+    Eliminate warnings during build on py3
+    Add CI and basic test suite
+    Raise a warning, not an error, if we don't get the bufsize we want
+    Don't allow bind() more than once on the same NetfilterQueue, since
+      that would leak the old queue handle
+
+v0.8.1, 30 Jan 2017
     Fix bug #25- crashing when used in OUTPUT or POSTROUTING chains
 
 v0.8, 15 Dec 2016

diff --git a/README.rst b/README.rst
@@ -3,9 +3,10 @@ NetfilterQueue
 ==============
 
 NetfilterQueue provides access to packets matched by an iptables rule in
-Linux. Packets so matched can be accepted, dropped, altered, or given a mark.
+Linux. Packets so matched can be accepted, dropped, altered, reordered,
+or given a mark.
 
-Libnetfilter_queue (the netfilter library, not this module) is part of the
+libnetfilter_queue (the netfilter library, not this module) is part of the
 `Netfilter project <http://netfilter.org/projects/libnetfilter_queue/>`_.
 
 Example
@@ -15,18 +16,18 @@ The following script prints a short description of each packet before accepting
 it. ::
 
     from netfilterqueue import NetfilterQueue
-    
+
     def print_and_accept(pkt):
         print(pkt)
         pkt.accept()
-    
+
     nfqueue = NetfilterQueue()
     nfqueue.bind(1, print_and_accept)
     try:
         nfqueue.run()
     except KeyboardInterrupt:
         print('')
-    
+
     nfqueue.unbind()
 
 You can also make your own socket so that it can be used with gevent, for example. ::
@@ -56,7 +57,7 @@ To send packets destined for your LAN to the script, type something like::
 Installation
 ============
 
-NetfilterQueue is a C extention module that links against libnetfilter_queue. 
+NetfilterQueue is a C extention module that links against libnetfilter_queue.
 Before installing, ensure you have:
 
 1. A C compiler
@@ -81,9 +82,9 @@ From source
 
 To install from source::
 
-    git clone git@github.com:kti/python-netfilterqueue.git
+    git clone https://github.com/oremanj/python-netfilterqueue
     cd python-netfilterqueue
-    python setup.py install
+    pip install .
 
 If Cython is installed, Distutils will use it to regenerate the .c source from the .pyx. It will then compile the .c into a .so.
 
@@ -104,9 +105,12 @@ NetfilterQueue objects
 A NetfilterQueue object represents a single queue. Configure your queue with
 a call to ``bind``, then start receiving packets with a call to ``run``.
 
-``QueueHandler.bind(queue_num, callback[, max_len[, mode[, range, [sock_len]]]])``
-    Create and bind to the queue. ``queue_num`` must match the number in your
-    iptables rule. ``callback`` is a function or method that takes one
+``QueueHandler.bind(queue_num, callback[, max_len[, mode[, range[, sock_len]]]])``
+    Create and bind to the queue. ``queue_num`` uniquely identifies this
+    queue for the kernel. It must match the ``--queue-num`` in your iptables
+    rule, but there is no ordering requirement: it's fine to either ``bind()``
+    first or set up the iptables rule first.
+    ``callback`` is a function or method that takes one
     argument, a Packet object (see below). ``max_len`` sets the largest number
     of packets that can be in the queue; new packets are dropped if the size of
     the queue reaches this number. ``mode`` determines how much of the packet
@@ -119,17 +123,21 @@ a call to ``bind``, then start receiving packets with a call to ``run``.
     Remove the queue. Packets matched by your iptables rule will be dropped.
 
 ``QueueHandler.get_fd()``
-    Get the file descriptor of the queue handler.
+    Get the file descriptor of the socket used to receive queued
+    packets and send verdicts. If you're using an async event loop,
+    you can poll this FD for readability and call ``run(False)`` every
+    time data appears on it.
 
 ``QueueHandler.run([block])``
-    Send packets to your callback. By default, this method blocks. Set
-    block=False to let your thread continue. You can get the file descriptor
-    of the socket with the ``get_fd`` method.
+    Send packets to your callback. By default, this method blocks, running
+    until an exception is raised (such as by Ctrl+C). Set
+    block=False to process the pending messages without waiting for more.
+    You can get the file descriptor of the socket with the ``get_fd`` method.
 
 ``QueueHandler.run_socket(socket)``
     Send packets to your callback, but use the supplied socket instead of
     recv, so that, for example, gevent can monkeypatch it. You can make a
-    socket with ``socket.fromfd(nfqueue.get_fd(), socket.AF_UNIX, socket.SOCK_STREAM)``
+    socket with ``socket.fromfd(nfqueue.get_fd(), socket.AF_NETLINK, socket.SOCK_RAW)``
     and optionally make it non-blocking with ``socket.setblocking(False)``.
 
 Packet objects
@@ -138,55 +146,78 @@ Packet objects
 Objects of this type are passed to your callback.
 
 ``Packet.get_payload()``
-    Return the packet's payload as a string (Python 2) or bytes (Python 3).
+    Return the packet's payload as a bytes object. The returned value
+    starts with the IP header. You must call ``retain()`` if you want
+    to be able to ``get_payload()`` after your callback has returned.
 
 ``Packet.set_payload(payload)``
-    Set the packet payload. ``payload`` is a bytes.
+    Set the packet payload. Call this before ``accept()`` if you want to
+    change the contents of the packet before allowing it to be released.
+    Don't forget to update the transport-layer checksum (or clear it,
+    if you're using UDP), or else the recipient is likely to drop the
+    packet. If you're changing the length of the packet, you'll also need
+    to update the IP length, IP header checksum, and probably some
+    transport-level fields (such as UDP length for UDP).
 
 ``Packet.get_payload_len()``
     Return the size of the payload.
 
 ``Packet.set_mark(mark)``
-    Give the packet a kernel mark. ``mark`` is a 32-bit number.
+    Give the packet a kernel mark, which can be used in future iptables
+    rules. ``mark`` is a 32-bit number.
 
 ``Packet.get_mark()``
-    Get the mark already on the packet.
+    Get the mark already on the packet (either the one you set using
+    ``set_mark()``, or the one it arrived with if you haven't called
+    ``set_mark()``).
 
 ``Packet.get_hw()``
     Return the hardware address as a Python string.
 
+``Packet.retain()``
+    Allocate a copy of the packet payload for use after the callback
+    has returned. ``get_payload()`` will raise an exception at that
+    point if you didn't call ``retain()``.
+
 ``Packet.accept()``
-    Accept the packet.
+    Accept the packet. You can reorder packets by accepting them
+    in a different order than the order in which they were passed
+    to your callback.
 
 ``Packet.drop()``
     Drop the packet.
-   
+
 ``Packet.repeat()``
-    Iterate the same cycle once more.
-
+    Restart processing of this packet from the beginning of its
+    Netfilter hook (iptables chain, roughly). Any changes made
+    using ``set_payload()`` or ``set_mark()`` are preserved; in the
+    absence of such changes, the packet will probably come right
+    back to the same queue.
+
 Callback objects
 ----------------
 
-Your callback can be function or a method and must accept one argument, a
-Packet object. You must call either Packet.accept() or Packet.drop() before
-returning.
-
-``callback(packet)`` or ``callback(self, packet)``
-    Handle a single packet from the queue. You must call either
-    ``packet.accept()`` or ``packet.drop()``.
+Your callback can be any one-argument callable and will be invoked with
+a ``Packet`` object as argument. You must call ``retain()`` within the
+callback if you want to be able to ``get_payload()`` after the callback
+has returned. You can hang onto ``Packet`` objects and resolve them later,
+but note that packets continue to count against the queue size limit
+until they've been given a verdict (accept, drop, or repeat). Also, the
+kernel stores the enqueued packets in a linked list, so keeping lots of packets
+outstanding is likely to adversely impact performance.
 
 Usage
 =====
 
 To send packets to the queue::
 
     iptables -I <table or chain> <match specification> -j NFQUEUE --queue-num <queue number>
-    
+
 For example::
 
     iptables -I INPUT -d 192.168.0.0/24 -j NFQUEUE --queue-num 1
-    
-The only special part of the rule is the target. Rules can have any match and 
+
+The only special part of the rule is the target. Rules can have any match and
 can be added to any table or chain.
 
 Valid queue numbers are integers from 0 to 65,535 inclusive.
@@ -228,7 +259,7 @@ Limitations
     * Omits methods for getting information about the interface a packet has
       arrived on or is leaving on
     * Probably other stuff is omitted too
-    
+
 Source
 ======
 
@@ -237,7 +268,7 @@ https://github.com/kti/python-netfilterqueue
 License
 =======
 
-Copyright (c) 2011, Kerkhoff Technologies, Inc.
+Copyright (c) 2011, Kerkhoff Technologies, Inc, and contributors.
 
 `MIT licensed <https://github.com/kti/python-netfilterqueue/blob/master/LICENSE.txt>`_
 
diff --git a/ci.sh b/ci.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+set -ex -o pipefail
+
+pip install -U pip setuptools wheel
+sudo apt-get install libnetfilter-queue-dev
+python setup.py sdist --formats=zip
+pip install dist/*.zip
+pip install -r test-requirements.txt
+
+cd tests
+pytest -W error -ra -v .