Skip to content

Relative template_dir path causes TemplateNotFound error in monorepo configuration #845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
deme3 opened this issue Feb 23, 2024 · 13 comments
Labels
bug Something isn't working properly confirmed Prevent from becoming stale

Comments

@deme3
Copy link

deme3 commented Feb 23, 2024

The problem

TemplateNotFound error thrown by Jinja2 when generating changelog in a subfolder. The directory structure is as follows:

- main_program
+-- templates
    +-- main_program
        +-- CHANGELOG.md.j2
+-- CHANGELOG.md
+-- pyproject.toml
+-- ...
- lib1
- lib2

To be clear, the templates are discovered by semantic-release, but when they're passed to Jinja, a relative path is used, and Jinja is not able to find them.

Expected behavior

I expect a generated CHANGELOG.md in the same level as pyproject.toml by using the templates provided in the configuration.

Environment

  • Python Version: v3.8.18
  • Pip Version: v23.0.1
  • Semantic Release Version: v9.1.0
pip freeze
annotated-types==0.6.0
antlr4-python3-runtime==4.13.1
antlr4-tools==0.2
argcomplete==3.2.2
argparse-addons==0.12.0
asn1tools==0.166.0
bcrypt==4.0.1
Beaker==1.12.1
beautifulsoup4==4.12.2
bitstruct==8.17.0
build==1.0.3
cantools==39.0.0
capstone==5.0.0rc2
certifi==2024.2.2
cffi==1.15.1
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
colored-traceback==0.3.0
crccheck==1.3.0
cryptography==40.0.2
cssselect==1.2.0
decli==0.6.1
decorator==5.1.1
diskcache==5.6.1
docutils==0.20.1
dotty-dict==1.3.1
exceptiongroup==1.1.3
filebytes==0.10.2
FormEncode==2.1.0
future==0.18.3
gitdb==4.0.11
GitPython==3.1.42
idna==3.6
importlib-metadata==7.0.1
importlib-resources==6.1.1
iniconfig==2.0.0
install-jdk==1.1.0
intervaltree==3.1.0
jaraco.classes==3.3.1
jeepney==0.8.0
Jinja2==3.1.3
keyring==24.3.0
lark==1.1.8
-e git+<redacted>
lxml==4.9.3
Mako==1.2.4
markdown-it-py==3.0.0
MarkupSafe==2.1.2
mdurl==0.1.2
mikai==4.0.1
more-itertools==10.2.0
msgpack==1.0.5
-e git+<redacted>
nh3==0.2.15
nose==1.3.7
numpy==1.24.4
packaging==23.1
paramiko==3.1.0
Paste==3.7.1
PasteDeploy==3.1.0
PasteScript==3.3.0
pkginfo==1.9.6
pluggy==1.3.0
plumbum==1.8.1
prompt-toolkit==3.0.36
psutil==5.9.5
pwntools==4.9.0
pycparser==2.21
pydantic==2.6.1
pydantic_core==2.16.2
pyelftools==0.29
Pygments==2.16.1
Pylons==1.0.3
PyNaCl==1.5.0
pyparsing==3.1.0
pyproject_hooks==1.0.0
pyserial==3.5
PySide6==6.6.1
PySide6-Addons==6.6.1
PySide6-Essentials==6.6.1
PySocks==1.7.1
pytest==7.4.2
python-can==4.2.2
python-dateutil==2.8.2
python-gitlab==4.4.0
python-semantic-release==9.1.0
pytz==2023.3
PyYAML==6.0.1
questionary==2.0.1
readme-renderer==42.0
repoze.lru==0.7
requests==2.31.0
requests-toolbelt==1.0.0
rfc3986==2.0.0
rich==13.7.0
ROPGadget==7.3
ropper==1.13.8
Routes==2.5.1
rpyc==5.3.1
SecretStorage==3.3.3
shellingham==1.5.4
shiboken6==6.6.1
simplejson==3.19.2
six==1.16.0
smmap==5.0.1
sortedcontainers==2.4.0
soupsieve==2.5
Tempita==0.5.2
termcolor==2.4.0
textparser==0.24.0
tomli==2.0.1
tomlkit==0.12.3
twine==5.0.0
typing_extensions==4.7.1
tzdata==2023.3
unicorn==2.0.1.post1
urllib3==2.2.1
waitress==2.1.2
wcwidth==0.2.9
WebError==0.13.1
WebHelpers==1.3
WebOb==1.8.7
WebTest==3.0.0
wrapt==1.15.0
zipp==3.17.0

Using build==1.0.3 and setuptools=56.0.0.

Configuration

[tool.semantic_release]
assets = []
commit_message = "chore: bumped version to {version}\n\nAutomatically generated by python-semantic-release"
commit_parser = "angular"
logging_use_named_masks = false
major_on_zero = true
tag_format = "main_program-v{version}"
version_variables = [
  "main_program/__init__.py:__version__"
]


[tool.semantic_release.changelog]
template_dir = "main_program/templates"
changelog_file = "CHANGELOG.md"
exclude_commit_patterns = []

Logs

GitHub Attachment to Logs

Additional context

There's something wrong here:

def recursive_render(
template_dir: Path,
environment: Environment,
_root_dir: str | os.PathLike[str] = ".",
) -> list[str]:
rendered_paths: list[str] = []
for root, file in (
(Path(root), file)
for root, _, files in os.walk(template_dir)
for file in files
# we slice relpath.parts[1:] to allow the top-level
# template folder to have a dot prefix.
# e.g. to permit ".github/psr-templates" to contain the templates,
# rather than enforcing a top-level, non-hidden directory
if not any(
elem.startswith(".")
for elem in Path(root).relative_to(template_dir).parts[1:]
)
and not file.startswith(".")
):
output_path = (_root_dir / root.relative_to(template_dir)).resolve()
log.info("Rendering templates from %s to %s", root, output_path)
output_path.mkdir(parents=True, exist_ok=True)
if file.endswith(".j2"):
# We know the file ends with .j2 by the filter in the for-loop
output_filename = file[:-3]
# Strip off the template directory from the front of the root path -
# that's the output location relative to the repo root
src_file_path = str((root / file).relative_to(template_dir))
output_file_path = str((output_path / output_filename).resolve())
log.debug("rendering %s to %s", src_file_path, output_file_path)
stream = environment.get_template(src_file_path).stream()
with open(output_file_path, "wb+") as output_file:
stream.dump(output_file, encoding="utf-8")
rendered_paths.append(output_file_path)
else:
src_file = str((root / file).resolve())
target_file = str((output_path / file).resolve())
log.debug(
"source file %s is not a template, copying to %s", src_file, target_file
)
shutil.copyfile(src_file, target_file)
rendered_paths.append(target_file)
return rendered_paths

specifically, line 109

log.debug("rendering %s to %s", src_file_path, output_file_path)
stream = environment.get_template(src_file_path).stream()

@deme3 deme3 added the bug Something isn't working properly label Feb 23, 2024
@codejedi365
Copy link
Contributor

@deme3 Thanks for the bug report and the initial troubleshooting you have accomplished. I have been working an extensive rewrite of the changelog testing and this will definitely be one of the new tests to add along with a fix.

@deme3
Copy link
Author

deme3 commented Feb 23, 2024

@codejedi365 thanks for letting me know. :)

Until we get the new release with the changelog rewrite, I can suggest the following patch for whoever is using this tool in a monorepo like me (note that I have no idea if this is good, I haven't looked much into the codebase, I just needed a quick fix for now):

unchanged:
--- a/cli/commands/changelog.py
+++ b/cli/commands/changelog.py
@@ -3,6 +3,7 @@ from __future__ import annotations
 import logging
 import os
 from typing import TYPE_CHECKING
+from pathlib import Path
 
 import click
 
@@ -76,7 +77,8 @@ def changelog(ctx: click.Context, release_tag: str | None = None) -> None:
                 f"{template_dir!r} relative to {repo.working_dir!r}"
             )
         else:
-            recursive_render(template_dir, environment=env, _root_dir=repo.working_dir)
+            project_root = Path(runtime.global_cli_options.config_file).absolute().parent
+            recursive_render(template_dir, environment=env, _root_dir=project_root)
 
     if release_tag:
         if runtime.global_cli_options.noop:
@@ -116,4 +118,4 @@ def changelog(ctx: click.Context, release_tag: str | None = None) -> None:
                 )
             except Exception as e:
                 log.exception(e)
-                ctx.fail(str(e))
\ No newline at end of file
+                ctx.fail(str(e))
unchanged:
--- a/cli/config.py
+++ b/cli/config.py
@@ -395,9 +395,10 @@ class RuntimeContext:
         )
 
         # changelog_file
+        project_root = Path(global_cli_options.config_file).absolute().parent
         changelog_file = Path(raw.changelog.changelog_file).resolve()
 
-        template_dir = Path(repo.working_tree_dir or ".") / raw.changelog.template_dir
+        template_dir = Path(project_root or ".") / raw.changelog.template_dir
 
         template_environment = environment(
             template_dir=raw.changelog.template_dir,
@@ -434,4 +435,4 @@ class RuntimeContext:
         # credential masker
         self.apply_log_masking(self.masker)
 
-        return self
\ No newline at end of file
+        return self
only in patch2:
unchanged:
--- a/cli/commands/version.py
+++ b/cli/commands/version.py
@@ -5,6 +5,7 @@ import os
 import subprocess
 from contextlib import nullcontext
 from datetime import datetime
+from pathlib import Path
 from typing import TYPE_CHECKING, ContextManager, Iterable
 
 import click
@@ -402,6 +403,7 @@ def version(  # noqa: C901
         hvcs_client=hvcs_client, release_history=rh
     )
     changelog_context.bind_to_environment(env)
+    project_root = Path(runtime.global_cli_options.config_file).absolute().parent
 
     updated_paths: list[str] = []
     if update_changelog:
@@ -409,13 +411,13 @@ def version(  # noqa: C901
             if opts.noop:
                 noop_report(
                     f"would have recursively rendered the template directory "
-                    f"{template_dir!r} relative to {repo.working_dir!r}. "
+                    f"{template_dir!r} relative to {project_root!r}. "
                     "Paths which would be modified by this operation cannot be "
                     "determined in no-op mode."
                 )
             else:
                 updated_paths = recursive_render(
-                    template_dir, environment=env, _root_dir=repo.working_dir
+                    template_dir, environment=env, _root_dir=project_root
                 )
 
         else:
@@ -425,13 +427,13 @@ def version(  # noqa: C901
             if opts.noop:
                 noop_report(
                     "would have written your changelog to "
-                    + str(changelog_file.relative_to(repo.working_dir))
+                    + str(changelog_file.relative_to(project_root))
                 )
             else:
                 changelog_text = render_default_changelog_file(env)
                 changelog_file.write_text(changelog_text, encoding="utf-8")
 
-            updated_paths = [str(changelog_file.relative_to(repo.working_dir))]
+            updated_paths = [str(changelog_file.relative_to(project_root))]
 
         if commit_changes and opts.noop:
             noop_report(

which instead of using the repository root directory as reference, uses the folder where the config is located.

@codejedi365
Copy link
Contributor

@deme3 I hope to have this in next week. My PR has been pending since the weekend.

I was a bit concerned that this could break other changelog designs but we didn't have any testing in place to validate either scenario

@codejedi365 codejedi365 added this to the Monorepo support milestone Mar 30, 2024
@codejedi365 codejedi365 added the confirmed Prevent from becoming stale label May 5, 2024
Copy link

github-actions bot commented Jul 5, 2024

It has been 60 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

@github-actions github-actions bot added the needs-update Needs status update from maintainers label Jul 5, 2024
@codejedi365
Copy link
Contributor

Need to circle back around to this issue. It slipped my mind, unfortunately the patch above will not work anymore given the refactor I did in v9.8.4 but if you port it to the changelog_writer.py file it should be easy enough to get back to your fix. I don't have a good pythonic monorepo example to go off of which is been part of the delay in support of monorepos. If you have an example open source project I can mirror for testing that would be great.

@github-actions github-actions bot removed the needs-update Needs status update from maintainers label Jul 7, 2024
@JonZeolla
Copy link

@codejedi365 https://github.com/langchain-ai/langchain/tree/master/libs is a pretty good open source python monorepo

@codejedi365
Copy link
Contributor

@JonZeolla, thank you!

Copy link

It has been 60 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

@github-actions github-actions bot added the needs-update Needs status update from maintainers label Dec 10, 2024
@codejedi365
Copy link
Contributor

Still in backlog

@github-actions github-actions bot removed the needs-update Needs status update from maintainers label Dec 11, 2024
Copy link

It has been 60 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

@github-actions github-actions bot added the needs-update Needs status update from maintainers label Feb 10, 2025
@codejedi365
Copy link
Contributor

Still in backlog

@github-actions github-actions bot removed the needs-update Needs status update from maintainers label Feb 11, 2025
Copy link

It has been 60 days since the last update on this confirmed issue. @python-semantic-release/team can you provide an update on the status of this issue?

@github-actions github-actions bot added the needs-update Needs status update from maintainers label Apr 12, 2025
@codejedi365
Copy link
Contributor

Still in backlog

@github-actions github-actions bot removed the needs-update Needs status update from maintainers label Apr 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working properly confirmed Prevent from becoming stale
Projects
None yet
Development

No branches or pull requests

3 participants