Dynamic USB descriptors #4689

dhalbert · 2021-04-30T03:19:02Z

Implements dynamic USB descriptors, including HID.
Fixes #4351, fixes #1015, fixes #4242, fixes #2087.
Not yet included: ability to change interface names (#4433), boot keyboard (#1136).
#620 (touchpad) would implemented as a custom HID device and a library.
This will break the existing VENDOR WebUSB implementation and that will have to be redone.

API Summary [edited to reflect final API after PR was merged]

All configuration is done in boot.py.

MSC and MIDI are simple on/off choices:

storage.disable_usb_device()    # Turn off MSC device
usb_midi.disable()   # Turn off MIDI device

CDC: you can enable/disable REPL or data channel (API changed from 6.x):

# Turn on both REPL and data channels; default is just REPL.
usb_cdc.enable(console=True, data=True)

HID: default devices are usb_hid.Device.MOUSE, .KEYBOARD, and .CONSUMER_CONTROL. Gamepad, sys_control, etc, are all dropped, but could be supplied by the user. You can choose any combination of these, or roll your own.

my_device = usb_hid.Device(report_descriptor=bytes(...), , usage_page=1, page=2, in_report_length=5, report_id_index=2)
usb_hid.enable((usb_hid.KEYBOARD, my_device))

All the configuration routines use the general configure_usb(), rather than just, say, enabled()) so that in the future they might take more arguments other than just bools or a list (e.g. interface names, multiple HID devices for a separate boot keyboard, etc.)

Implementation Details

The RAM needed is allocated dynamically according to size, when possible. A few things are small fixed arrays when it wasn't practical or worth it to make it dynamic. The boolean enabled/disabled choices are just remembered in static storage. The defaults are set before boot.py runs.

For the HID device list, it's trickier. The default list is set, and may be changed in boot.py. When boot.py finishes, before the VM goes away, the HID report descriptor is built. Since it might not be safe to leave it on the heap, it's copied up to the main.c stack temporarily, then the VM is stopped, then it's put back in a storage_allocation before the VM and heap are re-created for the REPL or code.py. Before the REPL or code.py VM is started, the other descriptors are built and put in storage_allocations or in static storage if they are fixed length. Then USB is initialized.

The HID devices are remembered in a short static array (up to 8 devices) between VM runs.

We could free the storage_allocations after USB starts, but it's hard to tell when, and I'm not sure the RAM can be made use of anyway, since it's just in a a random place. Perhaps more work could be done on that later, but we're talking a few hundred bytes.

The Python scripts that built the descriptors before are no longer needed. The descriptor fixups that used to be done in Python (put this interface number here and here, etc.) are now in C. The usb_descriptors submodule could be removed but requires some git surgery which I need to look up.

Tested on a Metro M4:

Descriptors compared and vetted: they match the previous fixed descriptors.
CIRCUITPY can be turned on and off
MIDI tested: midi_simpletest.py works and MIDI can be disabled
CDC: REPL only, data only, both
HID: device list varied, and copy of mouse descriptor used for a user-created device and tested

I expect build failures due to size.

Sample boot.py [EDIT: black mangled the comments in the first version of this rather badly. Fixed.]

import storage
import usb_cdc
import usb_midi
import usb_hid

# storage.disable_usb_device()

# usb_cdc.enable(console=True, data=True)

# usb_midi.disable()

# This is the same as the regular mouse descriptor
my_mouse = usb_hid.Device(
    report_descriptor=bytes(
        # Regular mouse
        (0x05, 0x01)  # 0,1 Usage Page (Generic Desktop)
        + (0x09, 0x02)  # 2,3 Usage (Mouse)
        + (0xA1, 0x01)  # 4,5 Collection (Application)
        + (0x09, 0x01)  #   6,7 Usage (Pointer)
        + (0xA1, 0x00) #   8,9 Collection (Physical)
        + (0x85, 0xFF)   #   10,11 Report ID  [11 is SET at RUNTIME]
        + (0x05, 0x09)  #     Usage Page (Button)
        + (0x19, 0x01)  #     Usage Minimum (0x01)
        + (0x29, 0x05)  #     Usage Maximum (0x05)
        + (0x15, 0x00)  #     Logical Minimum (0)
        + (0x25, 0x01)  #     Logical Maximum (1)
        + (0x95, 0x05)  #     Report Count (5)
        + (0x75, 0x01)  #     Report Size (1)
        + (0x81, 0x02)  #     Input (Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position)
        + (0x95, 0x01) #     Report Count (1)
        + (0x75, 0x03)  #     Report Size (8)
        + (0x81, 0x01)  #     Input (Const,Array,Abs,No Wrap,Linear,Preferred State,No Null Position)
        + (0x05, 0x01)  #     Usage Page (Generic Desktop Ctrls)
        + (0x09, 0x30)  #     Usage (X)
        + (0x09, 0x31)  #     Usage (Y)
        + (0x15, 0x81)  #     Logical Minimum (-127)
        + (0x25, 0x7F)  #     Logical Maximum (127)
        + (0x75, 0x08)  #     Report Size (8)
        + (0x95, 0x02)  #     Report Count (2)
        + (0x81, 0x06)  #     Input (Data,Var,Rel,No Wrap,Linear,Preferred State,No Null Position)
        + (0x09, 0x38)  #     Usage (Wheel)
        + (0x15, 0x81)  #     Logical Minimum (-127)
        + (0x25, 0x7F)  #     Logical Maximum (127)
        + (0x75, 0x08)  #     Report Size (8)
        + (0x95, 0x01)  #     Report Count (1)
        + (0x81, 0x06)  #     Input (Data,Var,Rel,No Wrap,Linear,Preferred State,No Null Position)
        + (0xC0,)  #   End Collection  
        + (0xC0,) # End Collection
    ),
    usage_page=1,
    usage=2,
    in_report_length=4,
    report_id_index=11,
)

usb_hid.enable((usb_hid.Device.KEYBOARD, my_mouse))

ThomasAtBBTF · 2021-04-30T05:06:26Z

I like this approach very much.
Will start to test as soon as it is available in a beta build.

Red-M · 2021-04-30T06:11:04Z

Does this allow setting multiple keyboard HID descriptors and their order?
If it does this would also allow a 6+ KRO keyboard HID and a method to allow BIOS mode keyboards.

dhalbert · 2021-04-30T13:10:22Z

Does this allow setting multiple keyboard HID descriptors and their order?
If it does this would also allow a 6+ KRO keyboard HID and a method to allow BIOS mode keyboards.

No, multiple HID device descriptors are not yet supported.

tannewt · 2021-04-30T18:25:02Z

Here are my first API thoughts. I haven't looked at the code yet. Thanks for working on this!

All the configuration routines use the general configure_usb(), rather than just, say, enabled()) so that in the future they might take more arguments other than just bools or a list (e.g. interface names, multiple HID devices for a separate boot keyboard, etc.)

I'm not a fan of the common configure_usb design because it makes them equally vague. I think it'd be better to have interface specific functions like storage.disable_usb_drive(), usb_midi.disable(), and usb_cdc.configure_interfaces(). I think this will make the boot.py code more readable.

Another thing to consider is whether this will work in the longer term where some interfaces are disabled and others are enabled. It might be better to set them all at once so you have one source for errors. What would happen if someone tries to enable too many things at once?

We could free the storage_allocations after USB starts, but it's hard to tell when, and I'm not sure the RAM can be made use of anyway, since it's just in a a random place. Perhaps more work could be done on that later, but we're talking a few hundred bytes.

I don't think you can because USB can restart at any time.

jerryneedell · 2021-05-07T15:01:28Z

@dhalbert I hate to keep bringing the pirkey_m0 up , but this removes pulseio from it again. Was that intentional? Should I give up on it...

jerryneedell · 2021-05-07T15:15:15Z

Ah -- it's more than just that -- I put pulseio back in -- now get an IOError: USB Busy -- I'll look into that but it may be time to move on...

dhalbert · 2021-05-08T02:48:17Z

Ah -- it's more than just that -- I put pulseio back in -- now get an IOError: USB Busy -- I'll look into that but it may be time to move on...

The removal of pulseio was unintentional; I will put it back. If you are getting USB errors, that is something else, and may reflect a larger problem. Try a Trinket M0 or similar; I did test, but not after I did a lot of the build rework.

jerryneedell · 2021-05-08T09:01:11Z

I opened #4724

deshipu · 2021-05-15T13:04:48Z

Works like a charm: https://hackaday.io/project/179496-chocolad-keyboard-hacking/log/192866-dynamic-usb-descriptors

dglaude · 2021-05-16T00:33:49Z

Works like a charm: https://hackaday.io/project/179496-chocolad-keyboard-hacking/log/192866-dynamic-usb-descriptors

Based on the above, I have been trying to improve my mouse jiggler to make the presence of CircuitPython stealth.
But I fail with two error: https://gist.github.com/dglaude/aad3d9e19858b062306b2a514434fd6b

No sure exactly what I did wrong (more than one thing likely)... I tried my best to upgrade my CPX to the latest CP, upgrade all the mpy and run this code as code.py... but It still fail. When using boot.py nothing seems to occurs.

Red-M · 2021-05-16T00:46:02Z

From my understanding, that slide position check and the disabling of USB descriptors needs to be in boot.py after you re-plug the board in.

dglaude · 2021-05-16T13:16:37Z

From my understanding, that slide position check and the disabling of USB descriptors needs to be in boot.py after you re-plug the board in.

So indeed boot.py is needed, and I was confused because the output of boot.py are in boot_out.txt.

It is very hard to troubleshoot and understand what is happening... but I also need to convince Windows 10 that the filesystem is clean. Any ejection when CIRCUITPY is visible make this trick stop to work for me and the drive keep reappering. Is the mass storage temporary visible to windows, and once the popup appear it is not possible to hide the drive???

Here is my minimal boot.py file for CPB board:

import board
import digitalio
import storage
import usb_cdc

switch = digitalio.DigitalInOut(board.SLIDE_SWITCH)
switch.direction = digitalio.Direction.INPUT

if switch.value:
    storage.disable_usb_drive()
    usb_cdc.disable()

dhalbert · 2021-05-16T13:24:12Z

That code looks correct. You should be able to get a clean unmount in Windows by doing an "Eject" from the USB icon in the taskbar. But that should not affect whether this code runs.

boot.py only runs after a hard reset (power-cycle, reset button press, or similar). It does not run after auto-reload due to editing code.py or another file, or if you type ctrl-D in the REPL.

ThomasAtBBTF · 2021-05-16T19:57:39Z

Hi,

this code:

import usb_cdc
import storage
import digitalio
import microcontroller

buttona = digitalio.DigitalInOut(microcontroller.pin.GPIO15)
buttona.direction = digitalio.Direction.INPUT
buttona.pull = digitalio.Pull.UP
thisstat = not buttona.value
buttona.deinit()

if thisstat:
  print("disable_usb_drive")
  storage.disable_usb_drive()
  print("usb_cdc.enable(console=False, data=True)")
  usb_cdc.enable(console=False, data=True)`
else:
  print("nothing changed")
print("boot.py done")

works for me on a MagTag.

What other USB-devices (end-points) can I deactivate to get:
usb_cdc.enable(console=True, data=True)
working?

If I try this the boot process is stuck in a loop!

(I think because "safe-mode" is not working on a ESP32-S2 !?

dhalbert · 2021-05-16T20:06:53Z

Do usb_hid.disable(). I think that will get you the extra endpoint pair you need.

The safe mode problem has an issue at #4766.

(If you put your entire code in a pair of triple-backquotes it will be marked as a single block of code.)

EDIT: and if you add py after the first set of triple-backquotes, it will syntax-color the code as Python. I did that above.

bitboy85 · 2021-05-16T23:25:34Z

I somehow broke the boot.py file which makes the file system read-only.

Adafruit CircuitPython 7.0.0-alpha.2-584-g25a44cb77 on 2021-05-15; Raspberry Pi Pico with rp2040
boot.py output:
OSError: [Errno 5] Input-/Output error

I think it should be the other way round: If something is wrong with boot.py, the filesystem should be always writeable so you are able to fix the error.

Renaming boot.py via REPL didn't work but formating the drive resets it completly.

dhalbert · 2021-05-16T23:32:38Z

I think it should be the other way round: If something is wrong with boot.py, the filesystem should be always writeable so you are able to fix the error.

If the filesystem is corrupted, then the metadata is probably damaged (e.g., the directory entries). It can't be recovered, so we or the host computer make it read-only so no worse damage will happen.

Read about the best way to edit files and avoid corruption here:
https://learn.adafruit.com/welcome-to-circuitpython/creating-and-editing-code#editing-code-2977443-18

bitboy85 · 2021-05-17T01:08:58Z

Ok, because the error got written to boot_out.txt it looked liked an internal behaviour of circuitpython.

Is there an updated sample boot.py from your first post?

dhalbert · 2021-05-17T01:20:51Z

Is there an updated sample boot.py from your first post?

What is your boot.py? That would be the easiest. Also see the documentation: https://circuitpython.readthedocs.io/en/latest/shared-bindings/index.html.

I will be writing a Learn Guide shortly. This is all very new. I am working on boot HID devices at the moment.

ladyada · 2021-05-17T13:57:02Z

FYI tested this last nite with RP2040, very successful!

bitboy85 · 2021-05-17T19:12:15Z

What is your boot.py? That would be the easiest.

I've just copied the sample you gave in this thread in the first post #4689 (comment)

but it outputs an error in boot_out.txt

Adafruit CircuitPython 7.0.0-alpha.2-584-g25a44cb77 on 2021-05-15; Raspberry Pi Pico with rp2040
boot.py output:
back trace (jüngste Aufforderung zuletzt):
file "boot.py", line 48, in
ValueError: out_report_length must be 1-255

Edit: According to this doc, out_report_length is allowed to be 0 https://circuitpython.readthedocs.io/en/latest/shared-bindings/usb_hid/

ThomasAtBBTF · 2021-05-17T19:46:54Z

I somehow broke the boot.py file which makes the file system read-only.

Adafruit CircuitPython 7.0.0-alpha.2-584-g25a44cb77 on 2021-05-15; Raspberry Pi Pico with rp2040
boot.py output:
OSError: [Errno 5] Input-/Output error

I think it should be the other way round: If something is wrong with boot.py, the filesystem should be always writeable so you are able to fix the error.

Renaming boot.py via REPL didn't work but formating the drive resets it completly.

I ran also into errors in boot.py during coding....
To get back a working system I simply downgraded to CP6.2 where these new libraries are not available.
Sure I got errors in boot.py but nothing locked up! And I could edit boot.py

todbot · 2021-05-17T23:28:20Z

I tested disabling CDC and MSD on a QTPy M0, using touchio pins. Works great, thank you!
Gist here: https://gist.github.com/todbot/707e4e3d393313cf31cdab56bf9d4255

Changed line 19 to try to match what's shown in adafruit/circuitpython#4689

Billfred · 2021-07-21T18:24:44Z

I think I'm missing a memo here. Between the information at the top of this pull request and @todbot's Gist from 5/17/21, I got to Billfred/Serpente-iPad-HID@affb16e for boot.py on my Serpente R2. Reset the board, and boot_out.txt reads:

Adafruit CircuitPython 7.0.0-alpha.4 on 2021-07-08; Serpente with samd21e18
boot.py output:
Traceback (most recent call last):
  File "boot.py", line 19, in <module>
AttributeError: 'module' object has no attribute 'KEYBOARD'

Mass storage is also remounting, which is an even clearer sign that something is going off the rails since it sure looks like I'm disabling storage in boot.py. Any insight in what I'm doing wrong?

dhalbert · 2021-07-21T18:40:45Z

@Billfred Please see https://learn.adafruit.com/customizing-usb-devices-in-circuitpython and https://circuitpython.readthedocs.io/en/latest/README.html for a more complete version of the latest information> if you have trouble please ask in https://adafru.it/discord or https://forums.adafruit.com.

dhalbert added 24 commits April 6, 2021 19:15

Add some minimum versions to requirements-dev.txt

9f30f87

wip

3e5bc5a

wip: incorporate new hid descriptor building

2a58b66

wip: latent usb device enabling/disabling

4a7e129

wip: revamp API names

6cb751a

wip: revert usb_descriptor changes; use raw descriptors instead

51ccf8d

wip: working on descriptor templates

6b18a51

wip: descriptor building

64e0958

wip: partial HID, still needs rework

2b4c88d

wip: getting closer

556a126

wip

aea3c4d

merge from main

9d1fcc3

wip: starting to try to compile

c26e49c

very much WIP

7a40b4d

wip: compiles

f98a546

partially working

8500e84

rework storage allocation

587aedd

merge from adafruit

f06d545

MSC, CDC, HID keyboard definitely working

abfb020

working!

71a8cad

delete hid .py; undo some debugging changes

b5efce1

bring protomatter up to date

3189668

undo requirements-dev.txt changes

48dd54c

doc fixes

b8b20fa

dhalbert added 2 commits April 30, 2021 09:40

fix type annotations

cc95b71

uncrustify with newer version of uncrustify

be7b2b0

dhalbert merged commit ebf9dcb into adafruit:main May 6, 2021

dhalbert deleted the dynamic-usb-descriptors branch May 6, 2021 17:51

Neradoc mentioned this pull request May 8, 2021

board disappears when in safe mode #4728

Closed

dhalbert mentioned this pull request May 15, 2021

Allow custom builds to disable CDC, MSC in a better way #3065

Closed

kamtom480 mentioned this pull request May 18, 2021

spresense: Fix USB CDC and MSC #4774

Merged

dhalbert mentioned this pull request May 21, 2021

Disable HID Gamepad by default once dynamic HID descriptors are implemented #4531

Closed

jepler mentioned this pull request Jul 15, 2021

Add support for multiple virtual keyboard devices for N-Key rollover #1439

Closed

Billfred added a commit to Billfred/Serpente-iPad-HID that referenced this pull request Jul 21, 2021

Update boot.py

affb16e

Changed line 19 to try to match what's shown in adafruit/circuitpython#4689

tannewt mentioned this pull request Aug 12, 2021

Teensy 4.1 cannot boot 7.0 #5086

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic USB descriptors #4689

Dynamic USB descriptors #4689

dhalbert commented Apr 30, 2021 •

edited

Loading

ThomasAtBBTF commented Apr 30, 2021

Red-M commented Apr 30, 2021

dhalbert commented Apr 30, 2021

tannewt commented Apr 30, 2021 •

edited

Loading

jerryneedell commented May 7, 2021

jerryneedell commented May 7, 2021

dhalbert commented May 8, 2021

jerryneedell commented May 8, 2021

deshipu commented May 15, 2021

dglaude commented May 16, 2021

Red-M commented May 16, 2021

dglaude commented May 16, 2021

dhalbert commented May 16, 2021

ThomasAtBBTF commented May 16, 2021 •

edited by dhalbert

Loading

dhalbert commented May 16, 2021 •

edited

Loading

bitboy85 commented May 16, 2021

dhalbert commented May 16, 2021

bitboy85 commented May 17, 2021

dhalbert commented May 17, 2021

ladyada commented May 17, 2021

bitboy85 commented May 17, 2021 •

edited

Loading

ThomasAtBBTF commented May 17, 2021

todbot commented May 17, 2021

Billfred commented Jul 21, 2021

dhalbert commented Jul 21, 2021

Dynamic USB descriptors #4689

Dynamic USB descriptors #4689

Conversation

dhalbert commented Apr 30, 2021 • edited Loading

ThomasAtBBTF commented Apr 30, 2021

Red-M commented Apr 30, 2021

dhalbert commented Apr 30, 2021

tannewt commented Apr 30, 2021 • edited Loading

jerryneedell commented May 7, 2021

jerryneedell commented May 7, 2021

dhalbert commented May 8, 2021

jerryneedell commented May 8, 2021

deshipu commented May 15, 2021

dglaude commented May 16, 2021

Red-M commented May 16, 2021

dglaude commented May 16, 2021

dhalbert commented May 16, 2021

ThomasAtBBTF commented May 16, 2021 • edited by dhalbert Loading

dhalbert commented May 16, 2021 • edited Loading

bitboy85 commented May 16, 2021

dhalbert commented May 16, 2021

bitboy85 commented May 17, 2021

dhalbert commented May 17, 2021

ladyada commented May 17, 2021

bitboy85 commented May 17, 2021 • edited Loading

ThomasAtBBTF commented May 17, 2021

todbot commented May 17, 2021

Billfred commented Jul 21, 2021

dhalbert commented Jul 21, 2021

dhalbert commented Apr 30, 2021 •

edited

Loading

tannewt commented Apr 30, 2021 •

edited

Loading

ThomasAtBBTF commented May 16, 2021 •

edited by dhalbert

Loading

dhalbert commented May 16, 2021 •

edited

Loading

bitboy85 commented May 17, 2021 •

edited

Loading