Skip to content

Workflow converter #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 88 commits into from
May 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
aa9f4c0
started to implement workflow converter
tclose Mar 26, 2024
95413b0
implemented and tested split statements util
tclose Mar 27, 2024
b1a2193
implementing workflow conversion
tclose Mar 28, 2024
c8b11d9
added local classes and functions to converted workflow files
tclose Apr 2, 2024
f6ff24b
added support for tripple quotes in extract_args
tclose Apr 4, 2024
a6637db
workflow spec generation
tclose Apr 4, 2024
88efbea
unused import
tclose Apr 4, 2024
e4e6be7
unused exception
tclose Apr 4, 2024
508768d
implemented workflow converter
tclose Apr 4, 2024
7eb06c5
added example workflow specs
tclose Apr 4, 2024
4afe3bc
debugging workflow converter
tclose Apr 4, 2024
d798530
reorganised workflow converter code
tclose Apr 5, 2024
3fa6413
sorted out issues with config params
tclose Apr 5, 2024
d9601a4
fixed up node connections in workflow converter
tclose Apr 6, 2024
6e8fd04
sorted out intrapackage module writing
tclose Apr 7, 2024
2651209
fixed up locally defined nested workflows
tclose Apr 7, 2024
d465d3d
fixed up intra_pkg_funcs writing so that only one function is written…
tclose Apr 8, 2024
b0de34e
debugged input_spec + lzin issues
tclose Apr 8, 2024
54048b2
sorted out multiple potential nodes
tclose Apr 8, 2024
f8319da
added external_nested workflows to anat_qc_workflow
tclose Apr 8, 2024
4941d25
task tests pass after import refactor
tclose Apr 10, 2024
6a9da03
fixed up a range of bugs with importing interfaces
tclose Apr 10, 2024
28c46aa
split utils into separate modules
tclose Apr 10, 2024
a9eb426
implementing package translations
tclose Apr 11, 2024
e6fe44b
added in mriqc interfaces
tclose Apr 11, 2024
fa070f2
created package-wide converter for workflow packages
tclose Apr 15, 2024
9d9fd68
ensured that callables imports are absolute
tclose Apr 15, 2024
4ca63e1
added new pkg-generation spec for mriqc
tclose Apr 15, 2024
c8e2951
added in pkg-gen-specs yamls
tclose Apr 16, 2024
8b5c995
working on package generation for workflows
tclose Apr 16, 2024
0ba3031
setting up package gen to work for workflow-predominant packages
tclose Apr 16, 2024
b54c6d9
got interface-only package generation back working again after refactor
tclose Apr 16, 2024
99a9865
debugging wider mriqc workflows
tclose Apr 16, 2024
c5c539d
debugging workflow package conversion
tclose Apr 16, 2024
c83155d
changed gen-code location to use expand user
tclose Apr 17, 2024
1a82e1f
full mriqc package has been written
tclose Apr 17, 2024
b6b02a5
fixing up issues with workflow package conversion
tclose Apr 17, 2024
e8aaeb6
debugging imports
tclose Apr 18, 2024
e6d37e7
debugging import translation
tclose Apr 18, 2024
a351896
fixed up manual import translations
tclose Apr 18, 2024
09b0948
fixed up import locations
tclose Apr 18, 2024
755ef54
reinstated interface generation
tclose Apr 18, 2024
97a67cc
inline nipype non-interface objects
tclose Apr 18, 2024
d124953
refactor task writing modules
tclose Apr 18, 2024
7808c9c
implemented writing of __init__ files
tclose Apr 19, 2024
b4671a1
implemented inlining of nipype objects
tclose Apr 19, 2024
4f33f40
fixed up creating of init files so pydra-mriqc imports
tclose Apr 19, 2024
1188d2a
changed convert CLI to allow explicit inclusion of non-interface/work…
tclose Apr 20, 2024
c423829
fixed up indentation reduction
tclose Apr 20, 2024
586f428
reworked intra_package object handling
tclose Apr 21, 2024
197ae4d
fixed up intra pkg imports and a bug in split statement with inline c…
tclose Apr 22, 2024
7472203
fixed bug with nested workflows local names
tclose Apr 22, 2024
eabf37c
fixed up module omission
tclose Apr 22, 2024
e343633
added empty __init__.py files for pydra.tasks.<pkg> and below up unti…
tclose Apr 22, 2024
28a1aa4
debugging package conversion
tclose Apr 22, 2024
b6ca18a
debugging workflow/package generation
tclose Apr 22, 2024
8d73b4a
implemented ports of interfaces from the nipype core package
tclose Apr 23, 2024
ac49ca9
fixed writing of __init__ in auto package
tclose Apr 23, 2024
8b21444
debugging unittests
tclose Apr 23, 2024
1cd6ea2
deleted old specs
tclose Apr 23, 2024
4d8fb4e
added package conversion test and all unittests pass
tclose Apr 23, 2024
c79d0a9
added conversion from niu.Function to FunctionTask
tclose Apr 24, 2024
697a8ee
fixed up workflow conversions to take workflow inputs as function args
tclose Apr 24, 2024
8af19a1
write conftest for workflow tests
tclose Apr 24, 2024
5b8a7b4
debugging workflow converters
tclose Apr 25, 2024
cda19bd
implemented NestedWorkflowAssignmentConverter
tclose Apr 26, 2024
e56c4d2
moved config defaults into package converter
tclose Apr 26, 2024
af5818d
added test_inputs option
tclose Apr 26, 2024
50a2bad
started implementing NodeFactoryConverter
tclose Apr 26, 2024
521aecf
refactored statements into separate sub-package
tclose Apr 27, 2024
98afe8a
refactored parse_statements to put match/parse logic in respective cl…
tclose Apr 27, 2024
5d625cf
implemented get_imported_object and node factory handling
tclose Apr 27, 2024
abb024d
fixed up handling of node factory methods
tclose Apr 27, 2024
287f4e0
implementing class and function converters
tclose Apr 27, 2024
bfecbc2
debugging workflow and helper function/class generation
tclose Apr 30, 2024
b67b22e
working on assignment statements to get delayed var info
tclose May 9, 2024
e5247d3
handled varfield inputs to nested workflows
tclose May 9, 2024
c32a2b8
reworked nested workflow input/output detection and top-level export …
tclose May 12, 2024
8446187
fixing up regressions
tclose May 13, 2024
28d185b
fixed issue with multiple input conns to same node
tclose May 13, 2024
9667474
debugging input/output mapping
tclose May 13, 2024
2f63f18
finally sorted out nested-workflow input/output propagation
tclose May 14, 2024
28a2828
fixed duplicate output issue caused by replacement mapping
tclose May 14, 2024
c7b274b
check for replaced connections in get_(input|output)_from_conn methods
tclose May 15, 2024
749a805
implemented trimming of unused inputs
tclose May 16, 2024
cc7a270
fixed unittests
tclose May 16, 2024
284458d
added in example specs from mriqc and reworked test_package_complete …
tclose May 17, 2024
451b545
added dep and use '--break-system-packages' in gha
tclose May 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
10 changes: 5 additions & 5 deletions .github/workflows/ci-cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,19 +49,19 @@ jobs:
python-version: ${{ matrix.python-version }}

- name: Update build tools
run: python3 -m pip install --upgrade pip setuptools wheel
run: python3 -m pip install --break-system-packages --upgrade pip setuptools wheel

- name: Install required file-formats packages
run: |
pushd required-fileformats
python3 -m pip install -r requirements.txt
python3 -m pip install --break-system-packages -r requirements.txt
popd

- name: Install Dipy separately as it was causing trouble
run: python3 -m pip install dipy
run: python3 -m pip install --break-system-packages dipy

- name: Install Package
run: python3 -m pip install .[test]
run: python3 -m pip install --break-system-packages .[test]

- name: Pytest
run: pytest -vvs --cov nipype2pydra --cov-config .coveragerc --cov-report xml
Expand Down Expand Up @@ -92,7 +92,7 @@ jobs:
python-version: '3.11'

- name: Install build tools
run: python3 -m pip install build twine
run: python3 -m pip install --break-system-packagesbuild twine

- name: Build source and wheel distributions
run: python3 -m build .
Expand Down
9 changes: 2 additions & 7 deletions conftest.py
Original file line number Diff line number Diff line change
@@ -1,27 +1,22 @@
import os
from pathlib import Path
import traceback
import tempfile
import pytest
from click.testing import CliRunner


PKG_DIR = Path(__file__).parent
EXAMPLE_SPECS_DIR = PKG_DIR / "example-specs"
EXAMPLE_TASKS_DIR = EXAMPLE_SPECS_DIR / "task" / "nipype"
EXAMPLE_INTERFACES_DIR = EXAMPLE_SPECS_DIR / "interface" / "nipype"
EXAMPLE_WORKFLOWS_DIR = EXAMPLE_SPECS_DIR / "workflow"
EXAMPLE_PKG_GEN_DIR = EXAMPLE_SPECS_DIR / "pkg-gen"


@pytest.fixture
def gen_test_conftest():
return PKG_DIR / "scripts" / "pkg_gen" / "resources" / "conftest.py"


@pytest.fixture(params=[str(p.stem) for p in EXAMPLE_WORKFLOWS_DIR.glob("*.yaml")])
def workflow_spec_file(request):
return (EXAMPLE_WORKFLOWS_DIR / request.param).with_suffix(".yaml")


@pytest.fixture
def work_dir():
work_dir = tempfile.mkdtemp()
Expand Down
Loading
Loading