Skip to content

Raspberry Pi Pico _thread OSError: TinyUSB callback can't recurse #15390

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
IDreamed opened this issue Jul 2, 2024 · 7 comments · Fixed by #15448
Closed

Raspberry Pi Pico _thread OSError: TinyUSB callback can't recurse #15390

IDreamed opened this issue Jul 2, 2024 · 7 comments · Fixed by #15448
Labels
bug port-rp2

Comments

@IDreamed
Copy link

IDreamed commented Jul 2, 2024

Port, board and/or hardware

Raspberry Pi Pico

MicroPython version

MicroPython v1.23.0 on 2024-06-02; Raspberry Pi Pico with RP2040

Reproduction

import time
import _thread
def test(name):
... print(1)
... time.sleep(1)
...
...
...
_thread.start_new_thread(test,('name',))
2
1
FATAL: uncaught exception 200127f0
OSError: TinyUSB callback can't recurse

Expected behaviour

I found that this problem does not exist in version 1.22.2

Observed behaviour

My previous program didn't work after I updated the firmware. I was sure the program would work, so I changed the firmware to 1.22.2 and it worked.I found when the "while True" in "_thread.start_new_thread" , this problem must occur. I guess it is caused by USB-related changes.

Additional Information

No, I've provided everything above.

Code of Conduct

Yes, I agree
@IDreamed IDreamed added the bug label Jul 2, 2024
@projectgus
Copy link
Contributor

Thanks @IDreamed for the clear report. I ran into this same thing earlier today while testing something else, hadn't been back to check more closely.

@IDreamed
Copy link
Author

IDreamed commented Jul 2, 2024

@projectgus My pleasure.

@riaancillie
Copy link

I'm receiving the same running the micropython RP2 multicore example. Every version since v1.22 results in some or other freeze or crash when creating a thread. I've pulled and built the master branch today and received the same error as you with the example:

FATAL: uncaught exception 200127f0
OSError: TinyUSB callback can't recurse

Any advice would be appreciated since my application needs multicore and the USBDevice class added in v1.23

@IDreamed
Copy link
Author

IDreamed commented Jul 3, 2024

Sorry I can't help, I don’t know much about embedded development, I suspect these issues are because the RP2’s multithread runs on real cores, which leads to resource contention since there can only be one core. I have encountered other issues related to _thread and USB.

@TRadigk
Copy link

TRadigk commented Jul 6, 2024

I got the same error text but with a different hex value only after flashing the firmware on my raspberry pico (on pico w this is even worse and the device is not recognized anymore):
#15230

projectgus added a commit to projectgus/micropython that referenced this issue Jul 11, 2024
@projectgus
usb,rp2: Fix "TinyUSB callback can't recurse" error when using threads.
fd7f64f 
Looks like there was a long standing underlying bug here when using rp2
threads, where either CPU may poll CDC input and trigger the TinyUSB task.
TinyUSB could run on both CPUs concurrently, which may have lead to some
incorrect behaviour.

The race started triggering an exception when runtime USB support was
added, and a check was added for the USB task recursing on itself from a
Python handler function.

The race is most commonly triggered when working from the interactive REPL,
even a minimal running thread can trigger it. This commit adds a test case
that triggers it in a different way (polling stdin from a thread).

Fix is to add a port-level macro that indicates whether the TinyUSB task
can run.

Closes micropython#15390.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
@projectgus projectgus mentioned this issue Jul 11, 2024
Merged
projectgus added a commit to projectgus/micropython that referenced this issue Jul 11, 2024
@projectgus
usb,rp2: Fix "TinyUSB callback can't recurse" error when using threads.
c56f73a 
Looks like there was a long standing underlying bug here when using rp2
threads, where either CPU may poll CDC input and trigger the TinyUSB task.
TinyUSB could run on both CPUs concurrently, which may have lead to some
incorrect behaviour.

The race started triggering an exception when runtime USB support was
added, and a check was added for the USB task recursing on itself from a
Python handler function.

The race is most commonly triggered when working from the interactive REPL,
even a minimal running thread can trigger it. This commit adds a test case
that triggers it in a different way (polling stdin from a thread).

Fix is to add a port-level macro that indicates whether the TinyUSB task
can run.

Closes micropython#15390.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
projectgus added a commit to projectgus/micropython that referenced this issue Jul 11, 2024
@projectgus
usb,rp2: Fix "TinyUSB callback can't recurse" error when using threads.
215791a 
Looks like there was a long standing underlying bug here when using rp2
threads, where either CPU may poll CDC input and trigger the TinyUSB task.
TinyUSB could run on both CPUs concurrently, which may have lead to some
incorrect behaviour.

The race started triggering an exception when runtime USB support was
added, and a check was added for the USB task recursing on itself from a
Python handler function.

The race is most commonly triggered when working from the interactive REPL,
even a minimal running thread can trigger it. This commit adds a test case
that triggers it in a different way (polling stdin from a thread).

Fix is to add a port-level macro that indicates whether the TinyUSB task
can run.

Closes micropython#15390.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
projectgus added a commit to projectgus/micropython that referenced this issue Jul 11, 2024
@projectgus
usb,rp2: Fix "TinyUSB callback can't recurse" error when using threads.
c902051 
Looks like there was a long standing underlying bug here when using rp2
threads, where either CPU may poll CDC input and trigger the TinyUSB task.
TinyUSB could run on both CPUs concurrently, which may have lead to some
incorrect behaviour.

The race started triggering an exception when runtime USB support was
added, and a check was added for the USB task recursing on itself from a
Python handler function.

The race is most commonly triggered when working from the interactive REPL,
even a minimal running thread can trigger it. This commit adds a test case
that triggers it in a different way (polling stdin from a thread).

Fix is to add a port-level macro that indicates whether the TinyUSB task
can run.

Closes micropython#15390.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
@tudorapo tudorapo mentioned this issue Aug 26, 2024
Closed
projectgus added a commit to projectgus/micropython that referenced this issue Sep 5, 2024
@projectgus
shared/tinyusb: Only run TinyUSB on the main thread.
ca18233 
Calls to run TinyUSB from other threads now schedule it to run
on the main thread, instead. Depends on the parent commit which
changes the scheduler to only run on the main thread.

Fixes micropython#15390 - "TinyUSB callback can't recurse" exceptions on rp2 when
using _thread module and USB serial I/O.

Adds a unit test for stdin functioning correctly in threads (fails on rp2
port without this fix).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
@adrianblakey
Copy link

MicroPython v1.24.0-preview.201.g269a0e0e1 on 2024-08-09; Raspberry Pi Pico2 with RP2350

Console output from Thonny:

MPY: soft reboot
Booting:  Pico2
INFO:device:the_device not yet defined
DEBUG:device:Old thread: my CPU id is 0
Before calibration
1725900134315,1.334,1.522,-51.150
After calibration
1725900134317,1.882,2.106,4.004
�>DEBUG:device:new thread: my CPU id is 1
DEBUG:device:Write to queue
DEBUG:device:Write to queue
DEBUG:device:Write to queue
DEBUG:device:Write to queue
DEBUG:device:Write to queue
DEBUG:device:Write to queue
DEBUG:device:Write to queue
DEBUG:device:Write to queue
DEBUG:device:Write to queue
DEBUG:device:Write to queue
DEBUG:device:Filled
DEBUG:device:Write to queue
DEBUG:device:Filled
DEBUG:device:Write to queue
DEBUG:device:Filled
DEBUG:device:Write to queue
DEBUG:device:Filled
FATAL: uncaught exception 20014aa0
OSError: TinyUSB callback can't recurse

From running this:

# run ADC collection on core 2 using _thread
from machine import ADC, Pin, mem32
import logging
import time
import random
import math

import asyncio
from primitives import AADC

from threadsafe import ThreadSafeQueue
import _thread


log = logging.getLogger("device")
logging.basicConfig(level=logging.DEBUG)
# 12 bit adc 0-4095, transformed by python to 0-65535
#SF: float = 17.966 / 3.3  # Voltage scaling. ~18V = 3.3V signal
CF: float = 3.3 / 65535    # Convert int to signal voltage
SC: float = 17.966 / 65535 

STOP: bool = False

class Device():
    
    # Singleton
    _instance: Optional[Device] = None

    def __new__(cls) -> Device:
        instance: Device
        if cls._instance is None:
            instance = object.__new__(cls)
            cls._instance = instance
        else:
            instance = cls._instance
        return instance
    
    def __init__(self):
        self._zero_current = 32420  # Good representative value
        self._adc0 = AADC(ADC(Pin(26))) # current
        self._adc1 = AADC(ADC(Pin(27))) # volts to car
        self._adc2 = AADC(ADC(Pin(28))) # track
        self._adc_filtered: float = 0
        self._device_queue: ThreadSafeQueue = ThreadSafeQueue([0 for _ in range(10)]) # 10 element list
        cpuid = mem32[0xd0000000]
        log.debug(f'Old thread: my CPU id is {cpuid}')
        # Pin 29 unused

    def core2_run(self):
        _thread.start_new_thread(self.core2_collect, () )

    def core2_collect(self):
        # Run the data collection exclusively on core 2
        global STOP
        cpuid = mem32[0xd0000000]
        log.debug("new thread: my CPU id is %s", cpuid)
        while True:
            if STOP:
                break
            log.debug('Write to queue')
            try:
                self._device_queue.put_sync(self.read_all(), block=False)
            except IndexError:
                log.debug('Filled')
                #STOP = True
            time.sleep_ms(100)
        
    def calibrate_current(self) -> None:
        # Measure the current 10 times and average
        i = 0
        count: int = 0
        while i < 10:
            count += self._adc0.read_u16()
            i += 1
        self._zero_current = round(count / 10)
        #log.debug('Calibrated current ~32767, raw u_16 value: %s', self._zero_current)

    def filter(self, value: float) -> float:
        self._adc_filtered = self._adc_filtered + value - (self._adc_filtered / divisor)
        return self._adc_filtered / divisor
        
    def __current(self) -> float:
        # Scale the current
        # log.debug('Current %s', self._adc0.read_u16() - self._zero_current)
        return (self._adc0.read_u16() - self._zero_current) * CF / .025  # 3.3/2 = 0 point, .025 = 1 amp
    
    def read_all(self) -> str:
        return f"{str(time.time_ns())[:-6]},{self._adc2.read_u16() * SC:.3f},{self._adc1.read_u16() * SC:.3f},{self.__current():.3f}"
    
    def read(self) -> str:
        return f"{self._adc2.read_u16() * SC:.3f},{self._adc1.read_u16() * SC:.3f},{self.__current():.3f}"
        
    async def exec(self) -> None:
        while True:
            self._adc0.sense(normal=False)
            value = await self._adc0(100, 65_000)  # Wait until in range
            print('current In range:', value)
            self._adc0.sense(normal=True)
            value = await self._adc0()  # Wait until out of range
            print('Out of range:', value)
          
    async def collect(self) -> None:
        while True:
            the_device.read_all()
        await asyncio.sleep_ms(10)
        
    async def sender(self, to_queue: ThreadSafeQueue):
        x = 0
        while True:
            to_queue.put_sync(self.read_all(), block=True)
            log.info('Queued')
            await asyncio.sleep_ms(10)                # log every 10 ms  
        
try:
    the_device
except NameError:
    log.info('the_device not yet defined')
    the_device = Device()
        
if __name__ == "__main__":
    print('Before calibration')
    print(the_device.read_all())
    the_device.calibrate_current()
    print('After calibration')
    print(the_device.read_all())
    
    try:
        the_device.core2_run()
    except:  # ctrl-c or program error
        log.debug('Excp - stop')
        STOP = True

Secondary issue - uncomment line 66 STOP = True - thread does not terminate.

@projectgus
Copy link
Contributor

@adrianblakey I think the PR linked above will fix this when it's merged.

Secondary issue - uncomment line 66 STOP = True - thread does not terminate.

This looks like it should work (in cases where the OSError: TinyUSB callback can't recurse error hasn't triggered, at least), so please open a new issue for it. If you can trim down your example to something which just exhibits this problem, that would be ideal - thanks!

projectgus added a commit to projectgus/micropython that referenced this issue Sep 10, 2024
@projectgus
shared/tinyusb: Only run TinyUSB on the main thread if GIL is disabled.
a0e1d30 
If GIL is disabled then there's threat of a race condition if some other
code specifically requests USB processing (i.e. to unblock stdio), while
a scheduled TinyUSB callback is already running on another thread.

Relies on the change in the parent commit, where scheduler is restricted
to main thread if GIL is disabled.

Fixes micropython#15390 - "TinyUSB callback can't recurse" exceptions on rp2 when
using _thread module and USB serial I/O.

Adds a unit test for stdin functioning correctly in threads (fails on rp2
port without this fix).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
projectgus added a commit to projectgus/micropython that referenced this issue Sep 10, 2024
@projectgus
shared/tinyusb: Only run TinyUSB on the main thread if GIL is disabled.
4732969 
If GIL is disabled then there's threat of a race condition if some other
code specifically requests USB processing (i.e. to unblock stdio), while
a scheduled TinyUSB callback is already running on another thread.

Relies on the change in the parent commit, where scheduler is restricted
to main thread if GIL is disabled.

Fixes micropython#15390 - "TinyUSB callback can't recurse" exceptions on rp2 when
using _thread module and USB serial I/O.

Adds a unit test for stdin functioning correctly in threads (fails on rp2
port without this fix).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
projectgus added a commit to projectgus/micropython that referenced this issue Sep 11, 2024
@projectgus
shared/tinyusb: Only run TinyUSB on the main thread if GIL is disabled.
54df502 
If GIL is disabled then there's threat of a race condition if some other
code specifically requests USB processing (i.e. to unblock stdio), while
a scheduled TinyUSB callback is already running on another thread.

Relies on the change in the parent commit, where scheduler is restricted
to main thread if GIL is disabled.

Fixes micropython#15390 - "TinyUSB callback can't recurse" exceptions on rp2 when
using _thread module and USB serial I/O.

Adds a unit test for stdin functioning correctly in threads (fails on rp2
port without this fix).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
projectgus added a commit to projectgus/micropython that referenced this issue Sep 17, 2024
@projectgus
shared/tinyusb: Only run TinyUSB on the main thread if GIL is disabled.
cf8d5f4 
If GIL is disabled then there's threat of a race condition if some other
code specifically requests USB processing (i.e. to unblock stdio), while
a scheduled TinyUSB callback is already running on another thread.

Relies on the change in the parent commit, where scheduler is restricted
to main thread if GIL is disabled.

Fixes micropython#15390 - "TinyUSB callback can't recurse" exceptions on rp2 when
using _thread module and USB serial I/O.

Adds a unit test for stdin functioning correctly in threads (fails on rp2
port without this fix).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
graeme-winter pushed a commit to winter-special-projects/micropython that referenced this issue Sep 21, 2024
@projectgus @graeme-winter
shared/tinyusb: Only run TinyUSB on the main thread if GIL is disabled.
4a8a6fa 
If GIL is disabled then there's threat of a race condition if some other
code specifically requests USB processing (i.e. to unblock stdio), while
a scheduled TinyUSB callback is already running on another thread.

Relies on the change in the parent commit, where scheduler is restricted
to main thread if GIL is disabled.

Fixes micropython#15390 - "TinyUSB callback can't recurse" exceptions on rp2 when
using _thread module and USB serial I/O.

Adds a unit test for stdin functioning correctly in threads (fails on rp2
port without this fix).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
wiznet-grace pushed a commit to wiznet-grace/micropython that referenced this issue Feb 27, 2025
@projectgus @wiznet-grace
shared/tinyusb: Only run TinyUSB on the main thread if GIL is disabled.
daf0469 
If GIL is disabled then there's threat of a race condition if some other
code specifically requests USB processing (i.e. to unblock stdio), while
a scheduled TinyUSB callback is already running on another thread.

Relies on the change in the parent commit, where scheduler is restricted
to main thread if GIL is disabled.

Fixes micropython#15390 - "TinyUSB callback can't recurse" exceptions on rp2 when
using _thread module and USB serial I/O.

Adds a unit test for stdin functioning correctly in threads (fails on rp2
port without this fix).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
wiznet-grace pushed a commit to WIZnet-ioNIC/WIZnet-ioNIC-micropython that referenced this issue Feb 28, 2025
@projectgus @wiznet-grace
shared/tinyusb: Only run TinyUSB on the main thread if GIL is disabled.
d0f5270 
If GIL is disabled then there's threat of a race condition if some other
code specifically requests USB processing (i.e. to unblock stdio), while
a scheduled TinyUSB callback is already running on another thread.

Relies on the change in the parent commit, where scheduler is restricted
to main thread if GIL is disabled.

Fixes micropython#15390 - "TinyUSB callback can't recurse" exceptions on rp2 when
using _thread module and USB serial I/O.

Adds a unit test for stdin functioning correctly in threads (fails on rp2
port without this fix).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug port-rp2
Projects
None yet
Milestone
No milestone
6 participants
@adrianblakey @projectgus @dpgeorge @riaancillie @IDreamed @TRadigk