axe is run towards the end of tests so it would be better if we run axe after each action or UI change.

Or at a specific point maybe? I have a view open — add run_axe() to run at that state — maybe passing the selenium instance if we need to... 🤔

axe-selenium-python Released: Nov 12, 2018

It doesn't look that well maintained. The latest version of Python in the test matrix is 3.6.

I'm not sure about having a dependency on this package as it is. 🤔

Yes, nice catch. I went over to axe-selenium-python and the wrapper is fairly simple https://github.com/mozilla-services/axe-selenium-python/blob/master/axe_selenium_python/axe.py, maybe we can write our own logic to inject aXe into our admin page and then run it for results.

I also found out pa11y another accessibility test tool and it seems it has better support for generating better reports and also has an open-source dashboard to see results.

I will take a look into how we could use it and share some ideas soon.

Hey @carltongibson @smithdc1 pa11y seems to be much better than using axe. pa11y can be called directly from CLI on URLs, we can use axe-core as a pa11y runner, it can output a decent formatted result, and have options to customize them even further. Apart from that, I see pa11y-dashboard as well to visualize results if needed.

After every UI change, writing a helper function that calls pa11y on the current selenium instance will not take much effort.

I highly favour using the existing admin tests for accessibility as compared to writing and maintaining accessibility tests.

One issue that I see with using pa11y as described above is redundancy in pa11y results because every time we would call pa11y, it would scan the whole page instead of only the new UI change.

pa11y has an option rootElement to only test a subset of DOM but this would require the use of "#" id tag to identify the elements that I doubt might be present on every UI piece of admin.

Other way than writing a whole accessibility testing suite is to iterate overall the .html files in the project and run pa11y over them to generate results. This might give us a lot of false positives like missing head tag or title tags due to .html file being in jinja template format.

Let me know your thoughts on the above points before I go ahead with something.

Closing per #16372 (comment). Please continue discussing possible solutions in the ticket.