Skip to content

Workflow converter #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 88 commits into from
May 17, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
aa9f4c0
started to implement workflow converter
tclose Mar 26, 2024
95413b0
implemented and tested split statements util
tclose Mar 27, 2024
b1a2193
implementing workflow conversion
tclose Mar 28, 2024
c8b11d9
added local classes and functions to converted workflow files
tclose Apr 2, 2024
f6ff24b
added support for tripple quotes in extract_args
tclose Apr 4, 2024
a6637db
workflow spec generation
tclose Apr 4, 2024
88efbea
unused import
tclose Apr 4, 2024
e4e6be7
unused exception
tclose Apr 4, 2024
508768d
implemented workflow converter
tclose Apr 4, 2024
7eb06c5
added example workflow specs
tclose Apr 4, 2024
4afe3bc
debugging workflow converter
tclose Apr 4, 2024
d798530
reorganised workflow converter code
tclose Apr 5, 2024
3fa6413
sorted out issues with config params
tclose Apr 5, 2024
d9601a4
fixed up node connections in workflow converter
tclose Apr 6, 2024
6e8fd04
sorted out intrapackage module writing
tclose Apr 7, 2024
2651209
fixed up locally defined nested workflows
tclose Apr 7, 2024
d465d3d
fixed up intra_pkg_funcs writing so that only one function is written…
tclose Apr 8, 2024
b0de34e
debugged input_spec + lzin issues
tclose Apr 8, 2024
54048b2
sorted out multiple potential nodes
tclose Apr 8, 2024
f8319da
added external_nested workflows to anat_qc_workflow
tclose Apr 8, 2024
4941d25
task tests pass after import refactor
tclose Apr 10, 2024
6a9da03
fixed up a range of bugs with importing interfaces
tclose Apr 10, 2024
28c46aa
split utils into separate modules
tclose Apr 10, 2024
a9eb426
implementing package translations
tclose Apr 11, 2024
e6fe44b
added in mriqc interfaces
tclose Apr 11, 2024
fa070f2
created package-wide converter for workflow packages
tclose Apr 15, 2024
9d9fd68
ensured that callables imports are absolute
tclose Apr 15, 2024
4ca63e1
added new pkg-generation spec for mriqc
tclose Apr 15, 2024
c8e2951
added in pkg-gen-specs yamls
tclose Apr 16, 2024
8b5c995
working on package generation for workflows
tclose Apr 16, 2024
0ba3031
setting up package gen to work for workflow-predominant packages
tclose Apr 16, 2024
b54c6d9
got interface-only package generation back working again after refactor
tclose Apr 16, 2024
99a9865
debugging wider mriqc workflows
tclose Apr 16, 2024
c5c539d
debugging workflow package conversion
tclose Apr 16, 2024
c83155d
changed gen-code location to use expand user
tclose Apr 17, 2024
1a82e1f
full mriqc package has been written
tclose Apr 17, 2024
b6b02a5
fixing up issues with workflow package conversion
tclose Apr 17, 2024
e8aaeb6
debugging imports
tclose Apr 18, 2024
e6d37e7
debugging import translation
tclose Apr 18, 2024
a351896
fixed up manual import translations
tclose Apr 18, 2024
09b0948
fixed up import locations
tclose Apr 18, 2024
755ef54
reinstated interface generation
tclose Apr 18, 2024
97a67cc
inline nipype non-interface objects
tclose Apr 18, 2024
d124953
refactor task writing modules
tclose Apr 18, 2024
7808c9c
implemented writing of __init__ files
tclose Apr 19, 2024
b4671a1
implemented inlining of nipype objects
tclose Apr 19, 2024
4f33f40
fixed up creating of init files so pydra-mriqc imports
tclose Apr 19, 2024
1188d2a
changed convert CLI to allow explicit inclusion of non-interface/work…
tclose Apr 20, 2024
c423829
fixed up indentation reduction
tclose Apr 20, 2024
586f428
reworked intra_package object handling
tclose Apr 21, 2024
197ae4d
fixed up intra pkg imports and a bug in split statement with inline c…
tclose Apr 22, 2024
7472203
fixed bug with nested workflows local names
tclose Apr 22, 2024
eabf37c
fixed up module omission
tclose Apr 22, 2024
e343633
added empty __init__.py files for pydra.tasks.<pkg> and below up unti…
tclose Apr 22, 2024
28a1aa4
debugging package conversion
tclose Apr 22, 2024
b6ca18a
debugging workflow/package generation
tclose Apr 22, 2024
8d73b4a
implemented ports of interfaces from the nipype core package
tclose Apr 23, 2024
ac49ca9
fixed writing of __init__ in auto package
tclose Apr 23, 2024
8b21444
debugging unittests
tclose Apr 23, 2024
1cd6ea2
deleted old specs
tclose Apr 23, 2024
4d8fb4e
added package conversion test and all unittests pass
tclose Apr 23, 2024
c79d0a9
added conversion from niu.Function to FunctionTask
tclose Apr 24, 2024
697a8ee
fixed up workflow conversions to take workflow inputs as function args
tclose Apr 24, 2024
8af19a1
write conftest for workflow tests
tclose Apr 24, 2024
5b8a7b4
debugging workflow converters
tclose Apr 25, 2024
cda19bd
implemented NestedWorkflowAssignmentConverter
tclose Apr 26, 2024
e56c4d2
moved config defaults into package converter
tclose Apr 26, 2024
af5818d
added test_inputs option
tclose Apr 26, 2024
50a2bad
started implementing NodeFactoryConverter
tclose Apr 26, 2024
521aecf
refactored statements into separate sub-package
tclose Apr 27, 2024
98afe8a
refactored parse_statements to put match/parse logic in respective cl…
tclose Apr 27, 2024
5d625cf
implemented get_imported_object and node factory handling
tclose Apr 27, 2024
abb024d
fixed up handling of node factory methods
tclose Apr 27, 2024
287f4e0
implementing class and function converters
tclose Apr 27, 2024
bfecbc2
debugging workflow and helper function/class generation
tclose Apr 30, 2024
b67b22e
working on assignment statements to get delayed var info
tclose May 9, 2024
e5247d3
handled varfield inputs to nested workflows
tclose May 9, 2024
c32a2b8
reworked nested workflow input/output detection and top-level export …
tclose May 12, 2024
8446187
fixing up regressions
tclose May 13, 2024
28d185b
fixed issue with multiple input conns to same node
tclose May 13, 2024
9667474
debugging input/output mapping
tclose May 13, 2024
2f63f18
finally sorted out nested-workflow input/output propagation
tclose May 14, 2024
28a2828
fixed duplicate output issue caused by replacement mapping
tclose May 14, 2024
c7b274b
check for replaced connections in get_(input|output)_from_conn methods
tclose May 15, 2024
749a805
implemented trimming of unused inputs
tclose May 16, 2024
cc7a270
fixed unittests
tclose May 16, 2024
284458d
added in example specs from mriqc and reworked test_package_complete …
tclose May 17, 2024
451b545
added dep and use '--break-system-packages' in gha
tclose May 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
started to implement workflow converter
  • Loading branch information
tclose committed Mar 26, 2024
commit aa9f4c0c944f6cb1cc8672dc5626d9f10f5d7a4c
44 changes: 44 additions & 0 deletions nipype2pydra/cli/workflow.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
from pathlib import Path
import click
import yaml
import nipype2pydra.workflow
from .base import cli


@cli.command(
name="workflow",
help="""Port Nipype task interface code to Pydra

YAML_SPEC is a YAML file which defines the workflow function to be imported

PACKAGE_ROOT is the path to the root directory of the packages in which to generate the
converted workflow
""",
)
@click.argument("yaml-spec", type=click.File())
@click.argument("package-root", type=click.Path(path_type=Path))
@click.option(
"--output-module",
"-m",
type=str,
default=None,
help=(
"the output module to store the converted task into relative to the `pydra.tasks` "
"package. If not provided, then the path relative to base package in the "
"source function will be used instead"
),
)
def workflow(yaml_spec, package_root, callables, output_module):

spec = yaml.safe_load(yaml_spec)

converter = nipype2pydra.workflow.WorkflowConverter(
output_module=output_module, **spec
)
converter.generate(package_root)


if __name__ == "__main__":
import sys

workflow(sys.argv[1:])
69 changes: 69 additions & 0 deletions nipype2pydra/workflow.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
from importlib import import_module
from functools import cached_property
import inspect
from types import ModuleType
from pathlib import Path
import attrs


@attrs.define
class WorkflowConverter:
"""Specifies how the semi-automatic conversion from Nipype to Pydra should
be performed

Parameters
----------
name: str
name of the workflow to generate
nipype_name: str, optional
the name of the task in the nipype module, defaults to the output task_name
nipype_module: str or ModuleType
the nipype module or module path containing the Nipype interface
output_module: str
the output module to store the converted task into relative to the `pydra.tasks` package
input_struct: tuple[str, type]
a globally accessible structure containing inputs to the workflow, e.g. config.workflow.*
tuple consists of the name of the input and the type of the input
"""

name: str
nipype_name: str
nipype_module: ModuleType = attrs.field(
converter=lambda m: import_module(m) if not isinstance(m, ModuleType) else m
)
output_module: str = attrs.field()
input_struct: str = None
inputnode: str = "inputnode"
outputnode: str = "outputnode"
nested_workflow_funcs: list[str] = None
omit_nodes: list[str] = None

@output_module.default
def _output_module_default(self):
return f"pydra.tasks.{self.nipype_module.__name__}"

@cached_property
def nipype_function(self):
return getattr(self.nipype_module, self.nipype_name)

@cached_property
def generate(self, package_root: Path):
"""Generate the Pydra task module

Parameters
----------
package_root: str
the root directory of the package to write the module to
"""

output_module = package_root.joinpath(
self.output_module.split(".")
).with_suffix(".py")
output_module.parent.mkdir(parents=True, exist_ok=True)

src = inspect.getsource(self.nipype_function)

code_str = ""

with open(output_module, "w") as f:
f.write(code_str)