Skip to content

Serial flag python #430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Serial flag python #430

wants to merge 17 commits into from

Conversation

sven-oly
Copy link
Collaborator

@sven-oly sven-oly requested review from echeran and sffc March 31, 2025 23:45
@sffc
Copy link
Member

sffc commented Apr 1, 2025 •
edited
Loading

There is an error in CI:

~/work/conformance/conformance
~/work/conformance/conformance/schema ~/work/conformance/conformance
Traceback (most recent call last):
  File "/home/runner/work/conformance/conformance/schema/check_schemas.py", line 153, in <module>
    main(sys.argv)
  File "/home/runner/work/conformance/conformance/schema/check_schemas.py", line 99, in main
    validator.options = SchemaArgs(args).getOptions()
                        ^^^^^^^^^^
NameError: name 'SchemaArgs' is not defined. Did you mean: 'schemaArgs'?

sffc
sffc reviewed Apr 1, 2025
View reviewed changes
@sven-oly
Copy link
Collaborator Author

sven-oly commented Apr 1, 2025

I'll work on this now. I pushed too soon!

@sven-oly
Copy link
Collaborator Author

sven-oly commented Sep 4, 2025

#386

@sven-oly sven-oly mentioned this pull request Sep 4, 2025
Open
sven-oly
sven-oly commented Sep 5, 2025
View reviewed changes
Copy link
Collaborator Author

@sven-oly sven-oly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert the files:

  • PluralRulesTest.java
  • datetime_fmt.py

with @echeran

schema/check_schemas.py Outdated
num_processors = multiprocessing.cpu_count()
logging.info('Schema validation: %s processors for %s schema validations', num_processors, len(file_names))
def validate_all_schema(validator, file_names):
if not validator.options.run_serial:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flip the "then" and "else" branches so that you can avoid using not in your condition test.

schema/check_test_output.py Outdated

# file_base + output_path
test_output_path = schema_options.schema_base
print('!!! TEST OUTPUT PATH %s' % (test_output_path))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no prints. only logging, and if necessary

schema/schema_validator.py Outdated
@@ -129,8 +127,13 @@ def validate_test_data_with_schema(self):

# Check for all the possible files
json_file_pattern = os.path.join(self.test_data_base, '*', '*.json')
logging.info('JSON FILE_PATTERN: %s', json_file_pattern);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for here and elsewhere (above & below): change log level to debug if this is not useful to see in a log file (ex: looking at a CI run)

schema/schema_validator.py Outdated
result = p.map(self.check_test_data_against_schema, schema_test_data)
return result
def check_all_test_data_schema(self, schema_test_data):
if not self.run_serial:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

schema/schema_validator.py Outdated
Comment on lines 189 to 190
for test_data in schema_test_data:
results.append(self.check_test_data_against_schema(test_data))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could turn the instantiation + initialization of results into a FP-style map. Not sure how Python does that.

schema/schema_validator.py Outdated
return results, test_validation_plans
else:
logging.info('JSON test output serially validation on %s files!', len(test_validation_plans))
for test_data in test_validation_plans:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto. also, does results need to be initialized before you call .append on it?

schema/schema_validator.py Outdated
num_processors = mp.cpu_count()
logging.info('JSON test output validation: %s processors for %s plans', num_processors,
len(test_validation_plans))
if not self.run_serial:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

schema/schema_validator.py
pass

if test_data_files_not_found:
logging.info('Note: %d potential test data sets were not found.', len(test_data_files_not_found))

results = self.parallel_check_test_data_schema(schema_test_info)
results = self.check_all_test_data_schema(schema_test_info)

for result_data in results:
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can replace the instantiation, for loop, and if with an FP-style map + filter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sffc sffc sffc left review comments

@echeran echeran Awaiting requested review from echeran

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sven-oly @sffc