Skip to content

Commit 6fb345e

Browse files
authored
Merge pull request oremanj#77 from oremanj/fixes
Fixes for several open issues
2 parents ddbc12a + c7fd3e5 commit 6fb345e

File tree

9 files changed

+413
-119
lines changed

9 files changed

+413
-119
lines changed

.github/workflows/ci.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,6 @@ jobs:
2525
check_lint: ['0']
2626
extra_name: ['']
2727
include:
28-
- python: '2.7'
29-
extra_name: ', build only'
3028
- python: '3.9'
3129
check_lint: '1'
3230
extra_name: ', check lint'

CHANGES.txt

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,10 @@
1+
v1.0.0, unreleased
2+
Propagate exceptions raised by the user's packet callback
3+
Warn about exceptions raised by the packet callback during queue unbinding
4+
Raise an error if a packet verdict is set after its parent queue is closed
5+
set_payload() now affects the result of later get_payload()
6+
Handle signals received when run() is blocked in recv()
7+
18
v0.9.0, 12 Jan 2021
29
Improve usability when Packet objects are retained past the callback
310
Add Packet.retain() to save the packet contents in such cases

README.rst

Lines changed: 76 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -90,21 +90,17 @@ To install from source::
9090
API
9191
===
9292

93-
``NetfilterQueue.COPY_NONE``
94-
95-
``NetfilterQueue.COPY_META``
96-
97-
``NetfilterQueue.COPY_PACKET``
93+
``NetfilterQueue.COPY_NONE``, ``NetfilterQueue.COPY_META``, ``NetfilterQueue.COPY_PACKET``
9894
These constants specify how much of the packet should be given to the
99-
script- nothing, metadata, or the whole packet.
95+
script: nothing, metadata, or the whole packet.
10096

10197
NetfilterQueue objects
10298
----------------------
10399

104100
A NetfilterQueue object represents a single queue. Configure your queue with
105101
a call to ``bind``, then start receiving packets with a call to ``run``.
106102

107-
``QueueHandler.bind(queue_num, callback[, max_len[, mode[, range[, sock_len]]]])``
103+
``NetfilterQueue.bind(queue_num, callback, max_len=1024, mode=COPY_PACKET, range=65535, sock_len=...)``
108104
Create and bind to the queue. ``queue_num`` uniquely identifies this
109105
queue for the kernel. It must match the ``--queue-num`` in your iptables
110106
rule, but there is no ordering requirement: it's fine to either ``bind()``
@@ -118,22 +114,23 @@ a call to ``bind``, then start receiving packets with a call to ``run``.
118114
the source and destination IPs of a IPv4 packet, ``range`` could be 20.
119115
``sock_len`` sets the receive socket buffer size.
120116

121-
``QueueHandler.unbind()``
117+
``NetfilterQueue.unbind()``
122118
Remove the queue. Packets matched by your iptables rule will be dropped.
123119

124-
``QueueHandler.get_fd()``
120+
``NetfilterQueue.get_fd()``
125121
Get the file descriptor of the socket used to receive queued
126122
packets and send verdicts. If you're using an async event loop,
127123
you can poll this FD for readability and call ``run(False)`` every
128124
time data appears on it.
129125

130-
``QueueHandler.run([block])``
126+
``NetfilterQueue.run(block=True)``
131127
Send packets to your callback. By default, this method blocks, running
132128
until an exception is raised (such as by Ctrl+C). Set
133-
block=False to process the pending messages without waiting for more.
134-
You can get the file descriptor of the socket with the ``get_fd`` method.
129+
``block=False`` to process the pending messages without waiting for more;
130+
in conjunction with the ``get_fd`` method, you can use this to integrate
131+
with async event loops.
135132

136-
``QueueHandler.run_socket(socket)``
133+
``NetfilterQueue.run_socket(socket)``
137134
Send packets to your callback, but use the supplied socket instead of
138135
recv, so that, for example, gevent can monkeypatch it. You can make a
139136
socket with ``socket.fromfd(nfqueue.get_fd(), socket.AF_NETLINK, socket.SOCK_RAW)``
@@ -148,6 +145,8 @@ Objects of this type are passed to your callback.
148145
Return the packet's payload as a bytes object. The returned value
149146
starts with the IP header. You must call ``retain()`` if you want
150147
to be able to ``get_payload()`` after your callback has returned.
148+
If you have already called ``set_payload()``, then ``get_payload()``
149+
returns what you passed to ``set_payload()``.
151150

152151
``Packet.set_payload(payload)``
153152
Set the packet payload. Call this before ``accept()`` if you want to
@@ -166,12 +165,46 @@ Objects of this type are passed to your callback.
166165
rules. ``mark`` is a 32-bit number.
167166

168167
``Packet.get_mark()``
169-
Get the mark already on the packet (either the one you set using
168+
Get the mark on the packet (either the one you set using
170169
``set_mark()``, or the one it arrived with if you haven't called
171170
``set_mark()``).
172171

173172
``Packet.get_hw()``
174-
Return the hardware address as a Python string.
173+
Return the source hardware address of the packet as a Python
174+
bytestring, or ``None`` if the source hardware address was not
175+
captured (packets captured by the ``OUTPUT`` or ``PREROUTING``
176+
hooks). For example, on Ethernet the result will be a six-byte
177+
MAC address. The destination hardware address is not available
178+
because it is determined in the kernel only after packet filtering
179+
is complete.
180+
181+
``Packet.get_timestamp()``
182+
Return the time at which this packet was received by the kernel,
183+
as a floating-point Unix timestamp with microsecond precision
184+
(comparable to the result of ``time.time()``, for example).
185+
Packets captured by the ``OUTPUT`` or ``POSTROUTING`` hooks
186+
do not have a timestamp, and ``get_timestamp()`` will return 0.0
187+
for them.
188+
189+
``Packet.id``
190+
The identifier assigned to this packet by the kernel. Typically
191+
the first packet received by your queue starts at 1 and later ones
192+
count up from there.
193+
194+
``Packet.hw_protocol``
195+
The link-layer protocol for this packet. For example, IPv4 packets
196+
on Ethernet would have this set to the EtherType for IPv4, which is
197+
``0x0800``.
198+
199+
``Packet.mark``
200+
The mark that had been assigned to this packet when it was enqueued.
201+
Unlike the result of ``get_mark()``, this does not change if you call
202+
``set_mark()``.
203+
204+
``Packet.hook``
205+
The netfilter hook (iptables chain, roughly) that diverted this packet
206+
into our queue. Values 0 through 4 correspond to PREROUTING, INPUT,
207+
FORWARD, OUTPUT, and POSTROUTING respectively.
175208

176209
``Packet.retain()``
177210
Allocate a copy of the packet payload for use after the callback
@@ -249,20 +282,39 @@ The fields are:
249282
Limitations
250283
===========
251284

252-
* Compiled with a 4096-byte buffer for packets, so it probably won't work on
253-
loopback or Ethernet with jumbo packets. If this is a problem, either lower
254-
MTU on your loopback, disable jumbo packets, or get Cython,
255-
change ``DEF BufferSize = 4096`` in ``netfilterqueue.pyx``, and rebuild.
256-
* Full libnetfilter_queue API is not yet implemented:
285+
* We use a fixed-size 4096-byte buffer for packets, so you are likely
286+
to see truncation on loopback and on Ethernet with jumbo packets.
287+
If this is a problem, either lower the MTU on your loopback, disable
288+
jumbo packets, or get Cython, change ``DEF BufferSize = 4096`` in
289+
``netfilterqueue.pyx``, and rebuild.
290+
291+
* Not all information available from libnetfilter_queue is exposed:
292+
missing pieces include packet input/output network interface names,
293+
checksum offload flags, UID/GID and security context data
294+
associated with the packet (if any).
257295

258-
* Omits methods for getting information about the interface a packet has
259-
arrived on or is leaving on
260-
* Probably other stuff is omitted too
296+
* Not all information available from the kernel is even processed by
297+
libnetfilter_queue: missing pieces include additional link-layer
298+
header data for some packets (including VLAN tags), connection-tracking
299+
state, and incoming packet length (if truncated for queueing).
300+
301+
* We do not expose the libnetfilter_queue interface for changing queue flags.
302+
Most of these pertain to other features we don't support (listed above),
303+
but there's one that could set the queue to accept (rather than dropping)
304+
packets received when it's full.
261305

262306
Source
263307
======
264308

265-
https://github.com/kti/python-netfilterqueue
309+
https://github.com/oremanj/python-netfilterqueue
310+
311+
Authorship
312+
==========
313+
314+
python-netfilterqueue was originally written by Matthew Fox of
315+
Kerkhoff Technologies, Inc. Since 2022 it has been maintained by
316+
Joshua Oreman of Hudson River Trading LLC. Both authors wish to
317+
thank their employers for their support of open source.
266318

267319
License
268320
=======

ci.sh

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,6 @@ python setup.py sdist --formats=zip
1313
pip uninstall -y cython
1414
pip install dist/*.zip
1515

16-
if python --version 2>&1 | fgrep -q "Python 2.7"; then
17-
# The testsuite doesn't run on 2.7, so do just a basic smoke test.
18-
unshare -Urn python -c "from netfilterqueue import NetfilterQueue as NFQ; NFQ()"
19-
exit $?
20-
fi
21-
2216
pip install -Ur test-requirements.txt
2317

2418
if [ "$CHECK_LINT" = "1" ]; then

netfilterqueue.pxd

Lines changed: 22 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ cdef extern from "<errno.h>":
88

99
# dummy defines from asm-generic/errno.h:
1010
cdef enum:
11+
EINTR = 4
1112
EAGAIN = 11 # Try again
1213
EWOULDBLOCK = EAGAIN
1314
ENOBUFS = 105 # No buffer space available
@@ -115,15 +116,17 @@ cdef extern from "libnetfilter_queue/libnetfilter_queue.h":
115116
u_int16_t num,
116117
nfq_callback *cb,
117118
void *data)
118-
int nfq_destroy_queue(nfq_q_handle *qh)
119119

120-
int nfq_handle_packet(nfq_handle *h, char *buf, int len)
121-
122-
int nfq_set_mode(nfq_q_handle *qh,
123-
u_int8_t mode, unsigned int len)
124-
125-
q_set_queue_maxlen(nfq_q_handle *qh,
126-
u_int32_t queuelen)
120+
# Any function that parses Netlink replies might invoke the user
121+
# callback and thus might need to propagate a Python exception.
122+
# This includes nfq_handle_packet but is not limited to that --
123+
# other functions might send a query, read until they get the reply,
124+
# and find a packet notification before the reply which they then
125+
# must deal with.
126+
int nfq_destroy_queue(nfq_q_handle *qh) except? -1
127+
int nfq_handle_packet(nfq_handle *h, char *buf, int len) except? -1
128+
int nfq_set_mode(nfq_q_handle *qh, u_int8_t mode, unsigned int len) except? -1
129+
int nfq_set_queue_maxlen(nfq_q_handle *qh, u_int32_t queuelen) except? -1
127130

128131
int nfq_set_verdict(nfq_q_handle *qh,
129132
u_int32_t id,
@@ -137,7 +140,6 @@ cdef extern from "libnetfilter_queue/libnetfilter_queue.h":
137140
u_int32_t mark,
138141
u_int32_t datalen,
139142
unsigned char *buf) nogil
140-
int nfq_set_queue_maxlen(nfq_q_handle *qh, u_int32_t queuelen)
141143

142144
int nfq_fd(nfq_handle *h)
143145
nfqnl_msg_packet_hdr *nfq_get_msg_packet_hdr(nfq_data *nfad)
@@ -146,7 +148,7 @@ cdef extern from "libnetfilter_queue/libnetfilter_queue.h":
146148
nfqnl_msg_packet_hw *nfq_get_packet_hw(nfq_data *nfad)
147149
int nfq_get_nfmark (nfq_data *nfad)
148150
nfnl_handle *nfq_nfnlh(nfq_handle *h)
149-
151+
150152
# Dummy defines from linux/socket.h:
151153
cdef enum: # Protocol families, same as address families.
152154
PF_INET = 2
@@ -166,12 +168,19 @@ cdef enum:
166168
NF_STOP
167169
NF_MAX_VERDICT = NF_STOP
168170

171+
cdef class NetfilterQueue:
172+
cdef object __weakref__
173+
cdef object user_callback # User callback
174+
cdef nfq_handle *h # Handle to NFQueue library
175+
cdef nfq_q_handle *qh # A handle to the queue
176+
169177
cdef class Packet:
170-
cdef nfq_q_handle *_qh
178+
cdef NetfilterQueue _queue
171179
cdef bint _verdict_is_set # True if verdict has been issued,
172180
# false otherwise
173181
cdef bint _mark_is_set # True if a mark has been given, false otherwise
174182
cdef bint _hwaddr_is_set
183+
cdef bint _timestamp_is_set
175184
cdef u_int32_t _given_mark # Mark given to packet
176185
cdef bytes _given_payload # New payload of packet, or null
177186
cdef bytes _owned_payload
@@ -189,16 +198,15 @@ cdef class Packet:
189198
cdef u_int8_t hw_addr[8]
190199

191200
# TODO: implement these
192-
#cdef u_int8_t hw_addr[8] # A eui64-formatted address?
193201
#cdef readonly u_int32_t nfmark
194202
#cdef readonly u_int32_t indev
195203
#cdef readonly u_int32_t physindev
196204
#cdef readonly u_int32_t outdev
197205
#cdef readonly u_int32_t physoutdev
198206

199-
cdef set_nfq_data(self, nfq_q_handle *qh, nfq_data *nfa)
207+
cdef set_nfq_data(self, NetfilterQueue queue, nfq_data *nfa)
200208
cdef drop_refs(self)
201-
cdef void verdict(self, u_int8_t verdict)
209+
cdef int verdict(self, u_int8_t verdict) except -1
202210
cpdef Py_ssize_t get_payload_len(self)
203211
cpdef double get_timestamp(self)
204212
cpdef bytes get_payload(self)
@@ -209,11 +217,3 @@ cdef class Packet:
209217
cpdef accept(self)
210218
cpdef drop(self)
211219
cpdef repeat(self)
212-
213-
cdef class NetfilterQueue:
214-
cdef object user_callback # User callback
215-
cdef nfq_handle *h # Handle to NFQueue library
216-
cdef nfq_q_handle *qh # A handle to the queue
217-
cdef u_int16_t af # Address family
218-
cdef packet_copy_size # Amount of packet metadata + data copied to buffer
219-

0 commit comments

Comments
 (0)