Skip to content

rp2: 1.22.0, 1.22.1: Lockup on machine.I2S initialization #13380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
roman-yepishev opened this issue Jan 8, 2024 · 8 comments
Closed

rp2: 1.22.0, 1.22.1: Lockup on machine.I2S initialization #13380

roman-yepishev opened this issue Jan 8, 2024 · 8 comments

Comments

@roman-yepishev
Copy link

STR:

  1. Flash Raspberry Pi Pico W with RPI_PICO_W-20231227-v1.22.0.uf2 or RPI_PICO_W-20240105-v1.22.1.uf2:

    MicroPython v1.22.1 on 2024-01-05; Raspberry Pi Pico W with RP2040
    
  2. Connect over USB serial console (e.g. with mpremote).

  3. Execute the following (no need to actually connect any I2S device):

    import machine
    i2s = machine.I2S(0, sck=machine.Pin(13), ws=machine.Pin(14),
                      sd=machine.Pin(15), mode=machine.I2S.TX,
                      bits=16, format=machine.I2S.MONO, rate=8000, ibuf=2000)
    

Expected results:

  • I2S is initialized and the object is returned.

Actual results:

  • The board stops transmitting data over USB serial console and does not return from that call (i.e. if I attempt to play audio afterwards as part of main.py, there is no output).

Re-flashing RPI_PICO_W-20231005-v1.21.0.uf2 allows the use of I2S on the same pins without lockup:

MicroPython v1.21.0 on 2023-10-06; Raspberry Pi Pico W with RP2040
Type "help()" for more information.
>>> import machine
>>> i2s = machine.I2S(0, sck=machine.Pin(13), ws=machine.Pin(14),
...                   sd=machine.Pin(15), mode=machine.I2S.TX,
...                               bits=16, format=machine.I2S.MONO, rate=8000, ibuf=2000)
>>> help(i2s)
object I2S(id=0,
...
@dpgeorge
Copy link
Member

dpgeorge commented Jan 8, 2024

Thanks for the report. I can confirm the issue. It was introduced by commit cfc212b, this line:

irq_set_exclusive_handler(DMA_IRQ_0, rp2_dma_irq_handler);

The I2S driver uses DMA IRQs, but the new rp2.DMA class claims the IRQ handlers for itself.

The fix is probably to change irq_set_exclusive_handler() to irq_add_shared_handler().

@nickovs what do you think?

@nickovs
Copy link
Contributor

nickovs commented Jan 8, 2024

I think that should be just fine. The IRQ handlers are only used for Python callbacks, not for any driver processing, so the added latency of the shared handler is unlikely to be noticeable.

@dpgeorge
Copy link
Member

dpgeorge commented Jan 8, 2024

We do have tests for I2S, and they pass on rp2. But those tests use I2S(1) which does still work because it uses DMA IRQ1, and that's different to what the rp2.DMA class uses (which is IRQ0).

So until this is fixed it's possible to use I2S(1).

@dpgeorge
Copy link
Member

dpgeorge commented Jan 8, 2024

@nickovs this is a bit more complicated to fix than using irq_add_shared_handler(), because the management of enabling/disable the IRQs must also be shared with I2S.

Was there a reason you only supported DMA_IRQ_0 in your driver, and not also DMA_IRQ_1? Eg if you want to use DMA on both cores then each of them should use a separate IRQ.

@nickovs
Copy link
Contributor

nickovs commented Jan 8, 2024

@dpgeorge No, there was no reason, aside from naivety and not having written a new driver from scratch before.

@nickovs
Copy link
Contributor

nickovs commented Jan 21, 2024

Looking into this a bit more, I think that the way to deal with this is for both the DMA and I2S code to ensure that their chosen DMA interrupt is enabled when they init, but to not switch it off when they deinit. They can then install shared handlers to handle the interrupts on the specific channels that they are using.

For the DMA interrupts there are two separate layers of enablement: there are the DMA_IRQ_0 and DMA_IRQ_1 flags in the main NVIC mask and then there are separate, per-channel flags in the DMA controller's INTE0 and INTE1 registers. Interrupts will only reach the CPU when a given DMA channel completes if the channel is enabled in INTEn and DMA_IRQ_n is also enabled. This means that as long as drivers properly set the flags in INTEn, not clearing the bit in the interrupt mark when de-initialising shouldn't cause any unnecessary noise but should allow both drivers to call irq_add_shared_handler() as needed.

dpgeorge added a commit to dpgeorge/micropython that referenced this issue Jan 30, 2024
These separate drivers must share the DMA resource with each other.

Fixes issue micropython#13380.

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
Copy link
Member

See #13555 for a fix for this issue.

@dpgeorge
Copy link
Member

Fixed by 807c25d

dpgeorge added a commit that referenced this issue Feb 20, 2024
These separate drivers must share the DMA resource with each other.

Fixes issue #13380.

Signed-off-by: Damien George <damien@micropython.org>
tytgatlieven pushed a commit to tytgatlieven/micropython-esp32c6 that referenced this issue Mar 19, 2024
These separate drivers must share the DMA resource with each other.

Fixes issue micropython#13380.

Signed-off-by: Damien George <damien@micropython.org>
graeme-winter pushed a commit to winter-special-projects/micropython that referenced this issue Sep 21, 2024
These separate drivers must share the DMA resource with each other.

Fixes issue micropython#13380.

Signed-off-by: Damien George <damien@micropython.org>
ThomasHornschuh added a commit to bonfireprocessor/micropython that referenced this issue Feb 15, 2025
* rp2/rp2_flash: Lockout second core only when doing flash erase/write.

Using the multicore lockout feature in the general atomic section makes it
much more difficult to get correct.

Signed-off-by: Damien George <damien@micropython.org>

* rp2/mutex_extra: Implement additional mutex functions.

These allow entering/exiting a mutex and also disabling/restoring
interrupts, in an atomic way.

Signed-off-by: Damien George <damien@micropython.org>

* rp2/mpthreadport: Fix race with IRQ when entering atomic section.

Prior to this commit there is a potential deadlock in
mp_thread_begin_atomic_section(), when obtaining the atomic_mutex, in the
following situation:
- main thread calls mp_thread_begin_atomic_section() (for whatever reason,
  doesn't matter)
- the second core is running so the main thread grabs the mutex via the
  call mp_thread_mutex_lock(&atomic_mutex, 1), and this succeeds
- before the main thread has a chance to run save_and_disable_interrupts()
  a USB IRQ comes in and the main thread jumps off to process this IRQ
- that USB processing triggers a call to the dcd_event_handler() wrapper
  from commit bcbdee2
- that then calls mp_sched_schedule_node()
- that then attempts to obtain the atomic section, calling
  mp_thread_begin_atomic_section()
- that call then blocks trying to obtain atomic_mutex
- core0 is now deadlocked on itself, because the main thread has the mutex
  but the IRQ handler (which preempted the main thread) is blocked waiting
  for the mutex, which will never be free

The solution in this commit is to use mutex enter/exit functions that also
atomically disable/restore interrupts.

Fixes issues micropython#12980 and micropython#13288.

Signed-off-by: Damien George <damien@micropython.org>

* all: Bump version to 1.22.1.

Signed-off-by: Damien George <damien@micropython.org>

* Generic STM32F401CD Port Compiles (not working yet...)

* rp2/rp2_dma: Fix fetching 'write' buffers for writing not reading.

Signed-off-by: Nicko van Someren <nicko@nicko.org>

* rp2/machine_uart: Fix event wait in uart.flush() and uart.read().

Do not wait in the worst case up to the timeout.

Fixes issue micropython#13377.

Signed-off-by: robert-hh <robert@hammelrath.com>

* renesas-ra/ra: Fix SysTick clock source.

The SysTick_Config function must use the system/CPU clock to configure the
ticks.

Signed-off-by: iabdalkader <i.abdalkader@gmail.com>

* renesas-ra/boards/ARDUINO_PORTENTA_C33: Fix the RTC clock source.

Switch the RTC clock source to Sub-clock (XCIN). This board has an
accurate LSE crystal, and it should be used for the RTC clock
source.

Signed-off-by: iabdalkader <i.abdalkader@gmail.com>

* extmod/asyncio: Support gather of tasks that finish early.

Adds support to asyncio.gather() for the case that one or more (or all)
sub-tasks finish and/or raise an exception before the gather starts.

Signed-off-by: Damien George <damien@micropython.org>

* mimxrt/modmachine: Fix deepsleep wakeup pin ifdef.

Signed-off-by: Kwabena W. Agyeman <kwagyeman@live.com>

* extmod/modssl_mbedtls: Fix cipher iteration in SSLContext.get_ciphers.

Prior to this commit it would skip every second cipher returned from
mbedtls.

The corresponding test is also updated and now passes on esp32, rp2, stm32
and unix.

Signed-off-by: Damien George <damien@micropython.org>

* rp2: Change machine.I2S and rp2.DMA to use shared DMA IRQ handlers.

These separate drivers must share the DMA resource with each other.

Fixes issue micropython#13380.

Signed-off-by: Damien George <damien@micropython.org>

* py/compile: Fix potential Py-stack overflow in try-finally with return.

If a return is executed within the try block of a try-finally then the
return value is stored on the top of the Python stack during the execution
of the finally block.  In this case the Python stack is one larger than it
normally would be in the finally block.

Prior to this commit, the compiler was not taking this case into account
and could have a Python stack overflow if the Python stack used by the
finally block was more than that used elsewhere in the function.  In such
a scenario the last argument of the function would be clobbered by the
top-most temporary value used in the deepest Python expression/statement.

This commit fixes that case by making sure enough Python stack is allocated
to the function.

Fixes issue micropython#13562.

Signed-off-by: Damien George <damien@micropython.org>

* renesas-ra/ra/ra_i2c: Fix 1 byte and 2 bytes read issue.

Tested on Portenta C33 with AT24256B (addrsize=16) and SSD1306.

Fixes issue micropython#13280.

Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>

* extmod/btstack: Reset pending_value_handle before calling write-done cb.

The pending_value_handle needs to be freed and reset before calling
mp_bluetooth_gattc_on_read_write_status(), which will call the Python IRQ
handler, which may in turn call back into BTstack to perform an action like
a write.  In that case the pending_value_handle will need to be available
for the write/read/etc to proceed.

Fixes issue micropython#13611.

Signed-off-by: Damien George <damien@micropython.org>

* extmod/btstack: Reset pending_value_handle before calling read-done cb.

Similar to the previous commit but for MP_BLUETOOTH_IRQ_GATTC_READ_DONE:
the pending_value_handle needs to be reset before calling
mp_bluetooth_gattc_on_read_write_status(), which will call the Python IRQ
handler, which may in turn call back into BTstack to perform an action like
a write.  In that case the pending_value_handle will need to be available
for the write/read/etc to proceed.

Fixes issue micropython#13634.

Signed-off-by: Damien George <damien@micropython.org>

* esp32/mpnimbleport: Release the GIL while doing NimBLE port deinit.

In case callbacks must run (eg a disconnect event happens during the
deinit) and the GIL must be obtained to run the callback.

Fixes part of issue micropython#12349.

Signed-off-by: Damien George <damien@micropython.org>

* esp32: Increase NimBLE task stack size and overflow detection headroom.

The Python BLE IRQ handler will most likely run on the NimBLE task, so its
C stack must be large enough to accommodate reasonably complicated Python
code (eg a few call depths).  So increase this stack size.

Also increase the headroom from 1024 to 2048 bytes.  This is needed because
(1) the esp32 architecture uses a fair amount of stack in general; and (2)
by the time execution gets to setting the Python stack top via
`mp_stack_set_top()` in this interlock code, about 600 bytes of stack are
already used, which reduces the amount available for Python.

Fixes issue micropython#12349.

Signed-off-by: Damien George <damien@micropython.org>

* all: Bump version to 1.22.2.

Signed-off-by: Damien George <damien@micropython.org>

* Submodule update

---------

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Nicko van Someren <nicko@nicko.org>
Signed-off-by: robert-hh <robert@hammelrath.com>
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
Signed-off-by: Kwabena W. Agyeman <kwagyeman@live.com>
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
Co-authored-by: Damien George <damien@micropython.org>
Co-authored-by: Nicko van Someren <nicko@nicko.org>
Co-authored-by: robert-hh <robert@hammelrath.com>
Co-authored-by: iabdalkader <i.abdalkader@gmail.com>
Co-authored-by: Kwabena W. Agyeman <kwagyeman@live.com>
Co-authored-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
ThomasHornschuh added a commit to bonfireprocessor/micropython that referenced this issue Feb 16, 2025
* rp2/rp2_flash: Lockout second core only when doing flash erase/write.

Using the multicore lockout feature in the general atomic section makes it
much more difficult to get correct.

Signed-off-by: Damien George <damien@micropython.org>

* rp2/mutex_extra: Implement additional mutex functions.

These allow entering/exiting a mutex and also disabling/restoring
interrupts, in an atomic way.

Signed-off-by: Damien George <damien@micropython.org>

* rp2/mpthreadport: Fix race with IRQ when entering atomic section.

Prior to this commit there is a potential deadlock in
mp_thread_begin_atomic_section(), when obtaining the atomic_mutex, in the
following situation:
- main thread calls mp_thread_begin_atomic_section() (for whatever reason,
  doesn't matter)
- the second core is running so the main thread grabs the mutex via the
  call mp_thread_mutex_lock(&atomic_mutex, 1), and this succeeds
- before the main thread has a chance to run save_and_disable_interrupts()
  a USB IRQ comes in and the main thread jumps off to process this IRQ
- that USB processing triggers a call to the dcd_event_handler() wrapper
  from commit bcbdee2
- that then calls mp_sched_schedule_node()
- that then attempts to obtain the atomic section, calling
  mp_thread_begin_atomic_section()
- that call then blocks trying to obtain atomic_mutex
- core0 is now deadlocked on itself, because the main thread has the mutex
  but the IRQ handler (which preempted the main thread) is blocked waiting
  for the mutex, which will never be free

The solution in this commit is to use mutex enter/exit functions that also
atomically disable/restore interrupts.

Fixes issues micropython#12980 and micropython#13288.

Signed-off-by: Damien George <damien@micropython.org>

* all: Bump version to 1.22.1.

Signed-off-by: Damien George <damien@micropython.org>

* Generic STM32F401CD Port Compiles (not working yet...)

* rp2/rp2_dma: Fix fetching 'write' buffers for writing not reading.

Signed-off-by: Nicko van Someren <nicko@nicko.org>

* rp2/machine_uart: Fix event wait in uart.flush() and uart.read().

Do not wait in the worst case up to the timeout.

Fixes issue micropython#13377.

Signed-off-by: robert-hh <robert@hammelrath.com>

* renesas-ra/ra: Fix SysTick clock source.

The SysTick_Config function must use the system/CPU clock to configure the
ticks.

Signed-off-by: iabdalkader <i.abdalkader@gmail.com>

* renesas-ra/boards/ARDUINO_PORTENTA_C33: Fix the RTC clock source.

Switch the RTC clock source to Sub-clock (XCIN). This board has an
accurate LSE crystal, and it should be used for the RTC clock
source.

Signed-off-by: iabdalkader <i.abdalkader@gmail.com>

* extmod/asyncio: Support gather of tasks that finish early.

Adds support to asyncio.gather() for the case that one or more (or all)
sub-tasks finish and/or raise an exception before the gather starts.

Signed-off-by: Damien George <damien@micropython.org>

* mimxrt/modmachine: Fix deepsleep wakeup pin ifdef.

Signed-off-by: Kwabena W. Agyeman <kwagyeman@live.com>

* extmod/modssl_mbedtls: Fix cipher iteration in SSLContext.get_ciphers.

Prior to this commit it would skip every second cipher returned from
mbedtls.

The corresponding test is also updated and now passes on esp32, rp2, stm32
and unix.

Signed-off-by: Damien George <damien@micropython.org>

* rp2: Change machine.I2S and rp2.DMA to use shared DMA IRQ handlers.

These separate drivers must share the DMA resource with each other.

Fixes issue micropython#13380.

Signed-off-by: Damien George <damien@micropython.org>

* py/compile: Fix potential Py-stack overflow in try-finally with return.

If a return is executed within the try block of a try-finally then the
return value is stored on the top of the Python stack during the execution
of the finally block.  In this case the Python stack is one larger than it
normally would be in the finally block.

Prior to this commit, the compiler was not taking this case into account
and could have a Python stack overflow if the Python stack used by the
finally block was more than that used elsewhere in the function.  In such
a scenario the last argument of the function would be clobbered by the
top-most temporary value used in the deepest Python expression/statement.

This commit fixes that case by making sure enough Python stack is allocated
to the function.

Fixes issue micropython#13562.

Signed-off-by: Damien George <damien@micropython.org>

* renesas-ra/ra/ra_i2c: Fix 1 byte and 2 bytes read issue.

Tested on Portenta C33 with AT24256B (addrsize=16) and SSD1306.

Fixes issue micropython#13280.

Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>

* extmod/btstack: Reset pending_value_handle before calling write-done cb.

The pending_value_handle needs to be freed and reset before calling
mp_bluetooth_gattc_on_read_write_status(), which will call the Python IRQ
handler, which may in turn call back into BTstack to perform an action like
a write.  In that case the pending_value_handle will need to be available
for the write/read/etc to proceed.

Fixes issue micropython#13611.

Signed-off-by: Damien George <damien@micropython.org>

* extmod/btstack: Reset pending_value_handle before calling read-done cb.

Similar to the previous commit but for MP_BLUETOOTH_IRQ_GATTC_READ_DONE:
the pending_value_handle needs to be reset before calling
mp_bluetooth_gattc_on_read_write_status(), which will call the Python IRQ
handler, which may in turn call back into BTstack to perform an action like
a write.  In that case the pending_value_handle will need to be available
for the write/read/etc to proceed.

Fixes issue micropython#13634.

Signed-off-by: Damien George <damien@micropython.org>

* esp32/mpnimbleport: Release the GIL while doing NimBLE port deinit.

In case callbacks must run (eg a disconnect event happens during the
deinit) and the GIL must be obtained to run the callback.

Fixes part of issue micropython#12349.

Signed-off-by: Damien George <damien@micropython.org>

* esp32: Increase NimBLE task stack size and overflow detection headroom.

The Python BLE IRQ handler will most likely run on the NimBLE task, so its
C stack must be large enough to accommodate reasonably complicated Python
code (eg a few call depths).  So increase this stack size.

Also increase the headroom from 1024 to 2048 bytes.  This is needed because
(1) the esp32 architecture uses a fair amount of stack in general; and (2)
by the time execution gets to setting the Python stack top via
`mp_stack_set_top()` in this interlock code, about 600 bytes of stack are
already used, which reduces the amount available for Python.

Fixes issue micropython#12349.

Signed-off-by: Damien George <damien@micropython.org>

* all: Bump version to 1.22.2.

Signed-off-by: Damien George <damien@micropython.org>

* Submodule update

---------

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Nicko van Someren <nicko@nicko.org>
Signed-off-by: robert-hh <robert@hammelrath.com>
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
Signed-off-by: Kwabena W. Agyeman <kwagyeman@live.com>
Signed-off-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
Co-authored-by: Damien George <damien@micropython.org>
Co-authored-by: Nicko van Someren <nicko@nicko.org>
Co-authored-by: robert-hh <robert@hammelrath.com>
Co-authored-by: iabdalkader <i.abdalkader@gmail.com>
Co-authored-by: Kwabena W. Agyeman <kwagyeman@live.com>
Co-authored-by: Takeo Takahashi <takeo.takahashi.xv@renesas.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants