-
Notifications
You must be signed in to change notification settings - Fork 630
Improve slcan.py #1490
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve slcan.py #1490
Conversation
1. Improve receiving performance 2. Fix an issue that the first read may blocking even if the `timeout` parameter is not `None`.
test codebus = slcanBus(channel="COM11", bitrate=20000)
tx_msg = Message(
arbitration_id=0x3C,
is_extended_id=False,
is_remote_frame=False,
dlc=8,
data=bytes(
[
0x01,
0x02,
0x10,
0x01,
0xFF,
0xFF,
0xFF,
0xFF,
]
),
)
rx_msg = Message(
arbitration_id=0x3D,
is_extended_id=False,
is_remote_frame=True,
dlc=8,
)
for _ in range(16):
bus.send(tx_msg)
bus.send(rx_msg)
msg = bus.recv()
print(msg)
bus.shutdown() gap is 39msgap is 1ms
|
@@ -161,7 +162,6 @@ def _read(self, timeout: Optional[float]) -> Optional[str]: | |||
while not ( | |||
ord(self._OK) in self._buffer or ord(self._ERROR) in self._buffer | |||
): | |||
self.serialPortOrig.timeout = time_left |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why did you remove this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Setting the serial port timeout in the while loop slow down the RX performance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a way to ensure, that the function respects the timeout
parameter?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a way to ensure, that the function respects the
timeout
parameter?
- This is the test code for testing the
timeout
parameter
import time
from can.interfaces.slcan import slcanBus
from can.message import Message
bus = slcanBus(channel="COM11", bitrate=20000, sleep_after_open=0)
tx_msg = Message(
arbitration_id=0x3C,
is_extended_id=False,
is_remote_frame=False,
dlc=8,
data=bytes(
[
0x01,
0x02,
0x10,
0x01,
0xFF,
0xFF,
0xFF,
0xFF,
]
),
)
rx_msg = Message(
arbitration_id=0x3D,
is_extended_id=False,
is_remote_frame=True,
dlc=8,
)
count = 5
timeout = 1
start_time = time.time()
for _ in range(count):
bus.send(tx_msg)
bus.send(rx_msg)
msg = bus.recv(timeout)
assert msg is None
print(msg)
end_time = time.time()
spend_time = end_time - start_time
assert abs((timeout * count) - spend_time) < 0.1
print(f"{spend_time:.3f}")
bus.shutdown()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With this proposed change I think there's still a scenario where the timeout is not enforced.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
read_all
sounds good to me.
def _read(self, timeout: Optional[float]) -> Optional[str]:
_timeout = serial.Timeout(timeout)
with error_check("Could not read from serial device"):
while not _timeout.expired():
new_data = self.serialPortOrig.read_all()
if new_data is not None:
self._buffer.extend(new_data)
else:
time.sleep(0.001)
continue
for terminator in (self._ERROR, self._OK):
if terminator in self._buffer:
i = self._buffer.index(terminator) + 1
string = self._buffer[:i].decode()
del self._buffer[:i]
return string
return None
The serial port must be instantiated with timeout=0
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But what about when there is data in the buffer? What if there were an infinite amount of 'junk'?
def read(self, __size: int = ...) -> bytes: ...
def read_all(self):
return self.read(self.in_waiting)
- If there is data in the buffer, the
read
method only copies it out. Only when the data bytes in the buffer are smaller than the parameter__size
, theread
method will blocking until the data bytes in the buffer is__size
, then copy it out. - Since the
in_waiting
parameter shows the current number of bytes in the buffer, blocking will not occur. (The time to copy data out is negligibly small) - According to my test, the
MAX
receive buffer size in win10 is16384
, and ubuntu 20.04 is4095
. I'm not sure whether it can be changed. But I think there will not beinfinite
junk.
import serial
import time
with serial.Serial("COM11") as ser:
# send a start command to make
# the STM32 dev board start sending data.
ser.write(b"start")
while True:
time.sleep(0.5)
print(f"in_waiting {ser.in_waiting}")
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Just a few minor changes requested.
According to my test, the read_all
method returns b''
if there is no data that can be read out.
So the if new_data is not None:
should be changed to if new_data:
, otherwise it will never do time.sleep
, which will cause high CPU usage.
def _read(self, timeout: Optional[float]) -> Optional[str]:
_timeout = serial.Timeout(timeout)
with error_check("Could not read from serial device"):
while not _timeout.expired():
new_data = self.serialPortOrig.read_all()
if new_data:
self._buffer.extend(new_data)
else:
time.sleep(0.001)
continue
for terminator in (self._ERROR, self._OK):
if terminator in self._buffer:
i = self._buffer.index(terminator) + 1
string = self._buffer[:i].decode()
del self._buffer[:i]
return string
return None
The serial port must be instantiated with timeout=0.
By the way, I did not initialize the timeout
parameter of serialPortOrig
at all and it defaults to None
. It seems still works for me.
I'm curious. If you don't set the timeout
to 0, what will happen to you?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm curious. If you don't set the timeout to 0, what will happen to you?
Nothing, i don't have an interface to test this 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to my test, the MAX receive buffer size in win10 is 16384, and ubuntu 20.04 is 4095. I'm not sure whether it can be changed. But I think there will not be infinite junk.
Interesting! And I agree -- I wasn't considering the physical limitations of the hardware/kernel. Good catch 🙂
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The timeout
parameter still needs to improve.
There is a slight performance difference between the Performance Comparison
the performance test codefrom can.interfaces.slcan import slcanBus
from can.message import Message
bus = slcanBus(channel="COM11", bitrate=20000, sleep_after_open=0)
data_frm = Message(
arbitration_id=0x3C,
is_extended_id=False,
is_remote_frame=False,
dlc=8,
)
remote_frm = Message(
arbitration_id=0x3D,
is_extended_id=False,
is_remote_frame=True,
dlc=8,
)
for _ in range(16):
# send a data frame
bus.send(data_frm)
# send a remote frame
bus.send(remote_frm)
# receive a data frame if a node respond the remote frame
msg = bus.recv()
print(msg)
# bus.shutdown() _read method using pyserial.read_alldef _read(self, timeout: Optional[float]) -> Optional[str]:
_timeout = serial.Timeout(timeout)
with error_check("Could not read from serial device"):
while True:
new_data = self.serialPortOrig.read_all()
if new_data:
self._buffer.extend(new_data)
else:
if _timeout.expired():
break
else:
time.sleep(0.001)
continue
for terminator in (self._ERROR, self._OK):
if terminator in self._buffer:
i = self._buffer.index(terminator) + 1
string = self._buffer[:i].decode()
del self._buffer[:i]
return string
return None _read method using pyserial.readdef _read(self, timeout: Optional[float]) -> Optional[str]:
_timeout = serial.Timeout(timeout)
self.serialPortOrig.timeout = timeout
with error_check("Could not read from serial device"):
while True:
new_data = self.serialPortOrig.read()
if new_data:
self._buffer.extend(new_data)
else:
if _timeout.expired():
break
else:
continue
for terminator in (self._ERROR, self._OK):
if terminator in self._buffer:
i = self._buffer.index(terminator) + 1
string = self._buffer[:i].decode()
del self._buffer[:i]
return string
return None |
Can you tell whether the delay is due to |
When I remove |
Yes, i agree, your |
Yes, you are right. python 3.11.1 have better performance than python 3.10.8.
In python 3.11.1, almost all gaps are 2ms, and some gaps are 1ms |
Since def _read(self, timeout: Optional[float]) -> Optional[str]:
_timeout = serial.Timeout(timeout)
self.serialPortOrig.timeout = timeout
with error_check("Could not read from serial device"):
while True:
new_byte = self.serialPortOrig.read(size=1)
if new_byte:
self._buffer.extend(new_byte)
if new_byte in (self._ERROR, self._OK):
string = self._buffer.decode()
del self._buffer[:]
return string
if _timeout.expired():
break
return None I added |
Yes, Looks better. 👍 |
to make the code easier to understand and read.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still need to improve.
the full SLCAN message.
the class variable `self._buffer` can be replaced by the local variable `buffer`.
This reverts commit 3eb80b9.
can/interfaces/slcan.py
Outdated
string = self._buffer[:i].decode() | ||
del self._buffer[:i] | ||
return string | ||
else: | ||
if _timeout.expired(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we really need _timeout
to handle timeout?
The timeout can be determined when pyserial.read
returns b''
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The timeout of pyserial.read
is reset on every read
call, so it might add up. The timeout argument of the user should be respected, _timeout
takes care of that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the timeout's add up
should be ignored.
For python on windows, the timer's resolution is about 16ms, obviously it is not very accurate.
And I think the timeout add up
is not as big as the impact of timer jitter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That depends on the timeout
value, It might be seconds, or minutes, or hours, i don't know what users might be doing.
You call it with timeout=0, it times out immediately. Call it with timeout=None for a blocking call. That behaviour seems correct to me. |
Yes, calling it with def _read(self, timeout: Optional[float]) -> Optional[str]:
_timeout = serial.Timeout(timeout)
self.serialPortOrig.timeout = timeout
with error_check("Could not read from serial device"):
while True:
new_byte = self.serialPortOrig.read(1)
if new_byte:
self._buffer.extend(new_byte)
if new_byte in (self._ERROR, self._OK):
string = self._buffer.decode()
del self._buffer[:]
return string
if _timeout.expired():
break
return None |
How about this? I added the def _read(self, timeout: Optional[float]) -> Optional[str]:
_timeout = serial.Timeout(timeout)
self.serialPortOrig.timeout = timeout
with error_check("Could not read from serial device"):
while True:
new_byte = self.serialPortOrig.read(size=1)
if new_byte:
self._buffer.extend(new_byte)
if new_byte in (self._ERROR, self._OK):
string = self._buffer.decode()
del self._buffer[:]
return string
if self.serialPortOrig.in_waiting:
continue
if _timeout.expired():
break
return None |
Unit test passed. |
What if we call def _read(self, timeout: Optional[float]) -> Optional[str]:
_timeout = serial.Timeout(timeout)
self.serialPortOrig.timeout = timeout
with error_check("Could not read from serial device"):
while True:
for _ in range(max(1, self.serialPortOrig.in_waiting)):
new_byte = self.serialPortOrig.read(size=1)
if new_byte:
self._buffer.extend(new_byte)
else:
break
if new_byte in (self._ERROR, self._OK):
string = self._buffer.decode()
del self._buffer[:]
return string
if _timeout.expired():
break
return None Edit: i added |
Better than |
But how can you ensure the timeout? Try this example: import time
from threading import Thread
from can.interfaces.slcan import slcanBus
def send_thread(_bus: slcanBus):
while True:
bus.serialPortOrig.write(b"f")
time.sleep(2.0)
if __name__ == '__main__':
bus = slcanBus(channel="loop://")
thread = Thread(target=send_thread, args=(bus,))
thread.start()
while True:
t0 = time.perf_counter()
bus.recv(3.0)
print(time.perf_counter() - t0) It should time out every 3 seconds, but it blocks forever. |
Thanks for the example code, I finally get your point. I noticed that there are two types of SLCAN devices categorized by how data is transmitted to the PC.
Since my SLCAN device is CDC VCOM based. Before seeing your example code, I don't understand why there is an add-up effect.😂 |
@zariiii9003 |
@@ -287,7 +287,9 @@ def get_version( | |||
|
|||
string = self._read(timeout) | |||
|
|||
if len(string) == 6 and string[0] == cmd: | |||
if not string: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the data in the input buffer.
@zariiii9003 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I will merge if you are happy with the performance.
Thanks for your review. The performance seems to be good so far. |
timeout
parameter is notNone
.