I've noticed that but haven't studied it because it doesn't seem to affect the overall execution status. Does someone have enough GitHub Actions knowledge to tell where to problem is and how to fix it?

I looked at this a bit more, and it seems that the whole Update status with GitHub Status API step is broken. It's only run if the actual execution fails and thus doesn't cause issues when everything is fine otherwise.

I believe that this steps tries to add links to the actual log and report to the status message shown here on GitHub. There's another step that is supposed to upload log and report to Netlify if there are failures, but also that step seems to be broken. The motivation for all this is making log and report visible after execution which GitHub Actions doesn't natively support. That was considered important when GA was taken into use, it replaced our Jenkins setup where that worked, and it certainly would be great if it worked. If it cannot be easily fixed, then it's probably best to remove the whole log/report handling setup altogether.

Would someone be interested to look could the old log/report uploading be fixed or are there better ways to do that nowadays? This setup is from pretty early days of GA and this could nowadays be easier.

I studied options to upload log and report. Here are some findings:

1. Uploading to Netlify is kind of a hack. Fixing and maintaining it is likely quite a lot of work.
2. It's easy to upload log and report as artifacts, we already do that, but they are only downloadable as as a single zip file. That's rather annoying, but it's typically enough to just see failed tests to be able to run them locally, so downloading and extracting the zip isn't that often needed.
3. Also others are annoyed by artifacts only be available as a zip and Upload-Artifact action has several issues related to that. For us these two seem to be most relevant:
   • Display files in browser where appropriate. actions/upload-artifact#14
   • Support artifact URL output actions/upload-artifact#50
4. There are actions for publishing test results that support xunit/junit format (e.g. Publish Test Results and Test Reporter). We could use them for getting overall results shown. As already mentioned, it would then typically be enough to run failed tests locally to see more information.
5. We could also create our own custom summary. I experimented that in my Robot/GA test repo and the SummaryGenerator.py pre-Rebot modifier worked nicely. It could be easily extended for listing all failed tests in addition to total/pass/skip/fail counts.

Based on the above I believe we should do the following:

1. Remove Netlify upload and related status updates.
2. Only use artifacts for storing report and log. They don't need to be downloaded that often and hopefully some day artifacts are enhanced so that they can be viewed directly.
3. Use some existing xunit/junit reporter or create a custom summary report to get the overall status and list of failed tests visible easily.

This isn't relevant anymore.