One option is to modify MICROPY_EVENT_POLL_HOOK to take an argument which is the maximum time that it can delay. Some ports (eg stm32) can ignore this argument and use existing behaviour. Others (eg esp32) can do a proper yield if the delay is longer than, eg, 10ms.

Another option for esp32 is to not use extmod/moduselect.c at all but rather replace it with a custom select module that uses select()/poll() provided by the IDF. Like how the unix port works, in fact esp32 could probably reuse the unix ports select module implementation (with tweaks). Whether or not this works in the long term depends on how powerful/extendable the IDF's select/poll code is, eg to hook in custom events. The advantage of this approach is that (hopefully) the IDF can go to sleep mode while waiting if the wait is long enough.

Changing the tick rate to 1kHz is the simplest option, but it's still going to suffer from the same problems, just at a 1ms resolution instead of 10ms. Eg adding a yield in MICROPY_EVENT_POLL_HOOK will probably make data (TCP/UDP) transfer slower because it essentially throttles it to 1 packet per 1ms tick. (On stm32 this doesn't happen because the WFI wakes on any interrupt, eg incoming wifi packet.)

Thanks for the food for thought.

For the short term, would you entertain a PR that sets the tick rate to 1kHz and makes sure that more than one packet per ms can be sent/received? If so, I'd look into that.

For the long term, you suggest perhaps using the IDF poll and to perhaps hook in custom events. I have to admit I was hoping we could go the other way, which is to avoid the poll stuff (unless explicitly used by the app) and move to a more event driven system where the MP task can go to sleep, just like a WFI, and be awoken by interrupt handlers or other events. I haven't looked into the new event stuff in the IDF but for sockets there's always the possibility of having a separate task doing the poll calls and converting the outcome into events. (I mean "events" in the very general sense as something that sets some flag and ensures the MP task is running.)

would you entertain a PR that sets the tick rate to 1kHz and makes sure that more than one packet per ms can be sent/received?

Yes. I'm not certain about the lag in packets, but it'd need to be tested (see eg uiperf3.py).

For the long term, you suggest perhaps using the IDF poll and to perhaps hook in custom events. I have to admit I was hoping we could go the other way, which is to avoid the poll stuff (unless explicitly used by the app) and move to a more event driven system where the MP task can go to sleep,

Yes this is also a good idea. Basically implementing a custom select module based on events rather than file polling.