Skip to content

auto-test documented test programs #186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 tasks done
markusicu opened this issue Dec 16, 2021 · 21 comments
Closed
2 tasks done

auto-test documented test programs #186

markusicu opened this issue Dec 16, 2021 · 21 comments
Assignees
Labels

Comments

@markusicu
Copy link
Member

markusicu commented Dec 16, 2021

In a CI workflow, run unit tests for documented tests.

List of tests:

Tool, not test, moved to #187:

@markusicu
Copy link
Member Author

Josh to find test programs, Steven to JUnit-ify them.

@markusicu
Copy link
Member Author

Example: https://github.com/unicode-org/unicodetools/blob/main/docs/idna.md
--> add an entry in the issue description/checklist above for TestIdna.java

Use judgment; some classes that should be unit tests may not be called *Test*.java.

@markusicu
Copy link
Member Author

Hi, just checking -- any progress? I think "Josh to find test programs" comes first...?

@josh-hadley
Copy link
Collaborator

Thanks for the reminder -- I have been sidetracked but hoping to make some progress on this and #187 this week

@josh-hadley
Copy link
Collaborator

@markusicu @srl295 I went through and added what appeared to be tests ("checks" in some cases).

@markusicu
Copy link
Member Author

markusicu commented Mar 7, 2022

Thanks @josh-hadley ! Over to @srl295 ?

  • TestUnicodeInvariants should be shovel-ready, and it's important because it frequently shows us needing to adjust properties.
    • Ideal would be a parameterized test, I think, but even just initially having it fail (boolean) if there is anything inconsistent would be useful.
  • TestIdna is hopefully easy
  • CheckXmlProperties is definitely a test, not a tool.
    • As a test, it would be useful to JUnit-ify it.
    • Issue: The ucdxml files are usually updated only rarely, maybe only once or twice per release. This means that this test would usually fail, so we can't put it into Continuous Integration.
    • Issue: The docs suggest that the test is failing already, and that one is to ignore those failures. In order to make it pass, it sounds like we would need to improve the XML file parsing.
    • Issue: Ownership of the ucdxml files & spec is up for grabs. Future uncertain.
  • CheckProperties looks more like a tool than a test
    • Most of its actions print useful or interesting information, or export properties data in another format (e.g., JSON), etc.
    • "COMPARE" shows the differences between the latest data and the previous Unicode version
    • "ICU" shows the differences between the current data and the ICU data
    • None of these actions look useful in JUnit form, and especially not for Continuous Integration
    • FYI: The docs show known systematic differences for Age values

@srl295
Copy link
Member

srl295 commented Mar 7, 2022

@markusicu sounds good to me. I can do it along with #146 (similar work, separate PRs)

@markusicu
Copy link
Member Author

@srl295 do you have time to work on TestUnicodeInvariants & TestIdna ?

@srl295
Copy link
Member

srl295 commented Mar 15, 2022

@srl295 do you have time to work on TestUnicodeInvariants & TestIdna ?

Generally yes maybe not today

@srl295
Copy link
Member

srl295 commented Mar 15, 2022

I think #146 is done but need to verify

@markusicu
Copy link
Member Author

friendly ping @srl295 on TestUnicodeInvariants & TestIdna :-) (see #186 (comment))

@markusicu
Copy link
Member Author

It would be great to get these done soon, helping get the Unicode 15 beta ready.

@markusicu
Copy link
Member Author

@srl295 back in town?

@srl295
Copy link
Member

srl295 commented May 10, 2022

@srl295 back in town?

I'll take a look tomorrow.

@markusicu
Copy link
Member Author

For a start, TestUnicodeInvariants does not have to become a parameterized test. The text above says "consider" :-) It would already be useful to just have it run as a test case and fail when either ParseErrorCount or TestFailureCount is not zero.

@markusicu
Copy link
Member Author

It looks like @srl295 turned TestIdna.java into a JUnit test already in PR #209, so I am checking that one off. (I then recently moved it from UnicodeJsps to plain unicodetools, and updated the instructions to say to run it as a JUnit test.)

@srl295
Copy link
Member

srl295 commented May 11, 2022

For a start, TestUnicodeInvariants does not have to become a parameterized test. The text above says "consider" :-) It would already be useful to just have it run as a test case and fail when either ParseErrorCount or TestFailureCount is not zero.

I've scheduled it for my work tomorrow.

@srl295
Copy link
Member

srl295 commented May 11, 2022

TestIdna.java

right, and it will also run along with all others when JUnit tests run.

srl295 added a commit that referenced this issue May 11, 2022
- add a test for SRC_UCD_DIR not containing nulls
- fix a misnamed "outputFile" that was really an input file

#186
srl295 added a commit that referenced this issue May 11, 2022
- read data from classpath on null.

#186
srl295 added a commit that referenced this issue May 12, 2022
- move TestCodeInvariants into the test/ subtree

#186
srl295 added a commit that referenced this issue May 12, 2022
- remove dead code and functions such as main()
- an earlier iteration did not call the GCB tests properly from JUnit

#186
srl295 added a commit that referenced this issue May 12, 2022
- an earlier iteration still did not call the GCB tests properly from JUnit

#186
srl295 added a commit that referenced this issue May 12, 2022
* TestUnicodeInvariants: add tests for TestCodeInvariants and TestUnicodeInvariants

- did not move code into the test subdir..yet

* TestUnicodeInvariants: test improvements

- add a test for SRC_UCD_DIR not containing nulls
- fix a misnamed "outputFile" that was really an input file

#186

* TestUnicodeInvariants: test improvements

- read data from classpath on null.

#186

* TestUnicodeInvariants: test improvements

- move TestCodeInvariants into the test/ subtree

* TestCodeInvariants: test improvements

- remove dead code and functions such as main()
- an earlier iteration did not call the GCB tests properly from JUnit

#186

* TestCodeInvariants: add a @test annotation

- an earlier iteration still did not call the GCB tests properly from JUnit

#186
@srl295
Copy link
Member

srl295 commented May 12, 2022

@markusicu should CheckXmlProperties be JUnit-ified or not? The description and discussion seem to conflict

@markusicu
Copy link
Member Author

@markusicu should CheckXmlProperties be JUnit-ified or not? The description and discussion seem to conflict

Not. After this discussion we decided to skip it, and removed the check box from the description.

I will propose a small docs update before closing this issue.

@srl295
Copy link
Member

srl295 commented May 13, 2022

🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants