Skip to content

Custom changelog seems to ignore update mode #1132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
marc-at-brightnight opened this issue Jan 5, 2025 · 35 comments
Closed

Custom changelog seems to ignore update mode #1132

marc-at-brightnight opened this issue Jan 5, 2025 · 35 comments
Labels

Comments

@marc-at-brightnight
Copy link

marc-at-brightnight commented Jan 5, 2025

Question

How do I make sure the changelog entries operate in 'update' mode, even with a custom changelog? I noticed that the update mode was working perfectly until I switched to a custom template.

My end goal is simply to include the body of the commit in the changelog entry, hence why I have a custom template.

Configuration

Semantic Release Configuration
[tool.semantic_release]
version_variables = ["__init__.py:__version__"]
upload_to_pypi = false
tag_format = "@[app]/[service]-v{version}"
commit_message = "chore(release): @[app]/[service]-v{version} [skip ci]"

[tool.semantic_release.changelog]
template_dir = "../../../documentation/templates"
mode = "update"

[tool.semantic_release.changelog.default_templates]
changelog_file = "../../../documentation/docs/[service]/changelog.md"

[tool.semantic_release.branches.test-semantic-release]
match = "test-semantic-release"
prerelease = false

Additional context

my template (if there is an easier way to do this, please let me know):

{# <root>/documentation/templates/documentation/docs/[service]/changelog.md.j2 #}

{% for version, release in ctx.history.released.items() %}
{{ "## %s (%s)" | format(version.as_semver_tag(), release.tagged_date.strftime("%Y-%m-%d")) }}
{% for type_, commits in (release.elements | dictsort) if type_ != "unknown" %}
{% set filtered_commits = [] %}
{% for commit in commits %}
{% if "[skip ci]" not in commit.message %}
{% set _ = filtered_commits.append(commit) %}
{% endif %}
{% endfor %}
{% if filtered_commits %}
{{ "### %s" | format(type_ | title) }}
{% for commit in commits %}
{{ "* %s ([`%s`](%s))" | format(
    commit.descriptions[0] | capitalize,
    commit.hexsha[:7],
    commit.hexsha | commit_hash_url
) }}
{% set lines = commit.message.split('\n') %}
{% set bullet_lines = [] %}
{% for line in lines[1:] %}
{% if line and (line.startswith('-') or line.startswith('*')) %}
{% set _ = bullet_lines.append(line) %}
{% endif %}
{% endfor %}
{% if bullet_lines %}
{% for bline in bullet_lines %}
    {{ bline }}
{% endfor %}
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}
{% endfor %}
# cicd.yml
# ... 
      - name: Create [service] github release
        if: needs.changes.outputs.[service] == 'true'
        uses: python-semantic-release/python-semantic-release@v9.15.2
        with:
          directory: poweralpha/services/[service]
          changelog: true
          commit: true
          push: true
          tag: true
          vcs_release: true
          github_token: ${{ secrets.GITHUB_TOKEN }}
          root_options: "-vv"
@marc-at-brightnight marc-at-brightnight added question triage waiting for initial maintainer review labels Jan 5, 2025
@codejedi365 codejedi365 removed the triage waiting for initial maintainer review label Jan 5, 2025
@codejedi365
Copy link
Contributor

codejedi365 commented Jan 5, 2025

Hello @marc-at-brightnight,

Did you happen to look at the Incrementally Updating Changelog Template section of the documentation? Was this helpful or not understandable?

Secondly, if I understand your desire, you would like the commit body to be a bulleted list rather than how the default template displays the full commit body? I think I first read it as you wanted only the subject line which is what most people ask me about, but by the look of the template it is more of a template formatting change. Either way I have provided instructions on what components to modify for handling the commit body within issue #1122. There is a comment for how to only display the commit subject line but in your case it points out the section you should modify for formatting details. The issue also includes more detail than the above documentation of what to change from the default to a custom template which would preserve other features (ex updating) that PSR has provided.

Also, note that we already split the commit messages up by newlines and include them in the commit.descriptions list so I don't think you would need to do that part yourself.

Lastly, I would recommend using the changelog.exclude_commit_patterns setting to enable PSR to remove those undesirable commits rather than you having to do that inside of the template.

@codejedi365 codejedi365 added the awaiting-reply Waiting for response label Jan 5, 2025
@codejedi365
Copy link
Contributor

Separately, may I ask what information are you trying to preserve and include from the commit body?

Over the past few months and for a few more in the future, I have been significantly changing the default changelog contents and on that roadmap is to reduce down to only displaying the commit subject line. The precursor to this is the support for parsing squash commits so that they can be each presented per version. Both are very close to publish. I have also added the features to parse out PR/MR numbers and parse out issue resolution numbers which are available to consumers in each ParsedCommit object. I have been systematically changing the default template to display these components in pretty formats. My reason for asking the question above is to understand the users of PSR and what other info they may desire that I have not already considered. Thanks!

@marc-at-brightnight
Copy link
Author

Did you happen to look at the Incrementally Updating Changelog Template section of the documentation? Was this helpful or not understandable?

I overlooked it. I think maybe changing the title might help, perhaps "Using a changelog template in update mode"?

Secondly, if I understand your desire, you would like the commit body to be a bulleted list rather than how the default template displays the full commit body? I think I first read it as you wanted only the subject line which is what most people ask me about, but by the look of the template it is more of a template formatting change. Either way I have provided instructions on what components to modify for handling the commit body within issue #1122. There is a comment for how to only display the commit subject line but in your case it points out the section you should modify for formatting details. The issue also includes more detail than the above documentation of what to change from the default to a custom template which would preserve other features (ex updating) that PSR has provided.

If I use the default template, this is what I see in the changelog:

### Features

- Add changelog template x3
  ([`780e9f4`](https://github.com/BrightNight-Energy/[app]/commit/780e9f407664d4050caea2acecdd1da2f7045f99))

This was my commit:

feat: Add changelog template x3
- this is a test
- hello

[PA-232]

Instead, I would like my changelog entry to read:

### Features

- Add changelog template x3
  ([`780e9f4`](https://github.com/BrightNight-Energy/[app]/commit/780e9f407664d4050caea2acecdd1da2f7045f99))
  - this is a test
  - hello

Which is why I turned to a custom template. It seems like the default template only includes the subject header. For our use case, if the user wants to provide more details in the form of bullets in the commit message, those bullets should be reflected in the changelog.

Also, note that we already split the commit messages up by newlines and include them in the commit.descriptions list so I don't think you would need to do that part yourself.

Got it 👍🏼

Lastly, I would recommend using the changelog.exclude_commit_patterns setting to enable PSR to remove those undesirable commits rather than you having to do that inside of the template.

Very good 👍🏼

Thank you very much @codejedi365 for the extremely good work on this repo and diligence on provided A+ user support. It is refreshing to encounter such professionalism on an open-source project and it does not go unnoticed 🫡

@marc-at-brightnight
Copy link
Author

marc-at-brightnight commented Jan 5, 2025

Separately, may I ask what information are you trying to preserve and include from the commit body?

See my previous response. If that doesn't answer your question fully, happy to expound.

@codejedi365
Copy link
Contributor

codejedi365 commented Jan 5, 2025

Did you happen to look at the Incrementally Updating Changelog Template section of the documentation? Was this helpful or not understandable?

I overlooked it. I think maybe changing the title might help, perhaps "Using a changelog template in update mode"?

Thanks, I'll consider it. That specific section is for updating a custom changelog previously made to now be an updating changelog, not exactly to explain update mode. Update mode is explained at the top of the page under the default templates. I'll try to add some clarity there.

This was my commit:

feat: Add changelog template x3
- this is a test
- hello

[PA-232]

It seems like the default template only includes the subject header.

I believe you are hitting an unexpected edge case of the parser due to your commit format to give you a result that only includes the commit subject. From the conventional commit standard (i.e. the newer variant of the angular guidelines), It is expected for there to be a double newline (a blank line) between the commit subject and the commit body. Since your commit does not have a double newline, somehow the bullets are being stripped during parsing.

If you amend your commit to be the following then I suspect you will get a different result (and maybe the one you desire):

feat: Add changelog template x3

- this is a test
- hello

[PA-232]

Also, note that we already split the commit messages up by newlines and include them in the commit.descriptions list so I don't think you would need to do that part yourself.

Got it 👍🏼

I should probably clarify now that I have discussed commit formatting above. The commit.descriptions list is a product of the following logic:

  1. Reverse word-wrapping (i.e. strip out single newlines) unless the following line does not start with an alphanumeric
  2. Reverse hyphen-wrapped words
  3. Split the resulting commit message body paragraphs into an array (i.e. split on double newlines)
  4. Insert the commit summary as the very first item in the array

Thank you very much @codejedi365 for the extremely good work on this repo and diligence on provided A+ user support. It is refreshing to encounter such professionalism on an open-source project and it does not go unnoticed 🫡

That means a lot, as user support takes a lot more time than expected. Thank you!

@marc-at-brightnight
Copy link
Author

marc-at-brightnight commented Jan 5, 2025

If you amend your commit to be the following then I suspect you will get a different result (and maybe the one you desire):

Interesting. I tried that and got the following:

- Let's try this with a double new line, see what we get
  ([`5f5eadb`](https://github.com/BrightNight-Energy/[app]/commit/5f5eadb5c60f64e2d94583007382591100b180c5))

- double new line above - very good

[PA-1231]

with commit:

feat: let's try this with a double new line, see what we get

- double new line above
- very good

[PA-1231]

Not exactly what I want so will stick with custom template for now.

I should probably clarify now that I have discussed commit formatting above. The commit.descriptions list is a product of the following logic:

  1. Reverse word-wrapping (i.e. strip out single newlines) unless the following line does not start with an alphanumeric
  2. Reverse hyphen-wrapped words
  3. Split the resulting commit message body paragraphs into an array (i.e. split on double newlines)
  4. Insert the commit summary as the very first item in the array

Ah, that explains why .descriptions doesn't work for me. I need it split up by single newlines, not double. No matter, easy fix.

I think I have what I need (mostly just needed the pointer to the right section of the documentation but the extra context and help has been super helpful), except I have one more question, if you don't mind:
I have multiple services and would like to avoid duplicating the changelog template in different folders that mirrors our documentation structure. is there a way I can have multiple services use the same changelog?

My latest template, in case it is helpful for others encountering this thread: (sorry for bad formatting, my first jinja2 template)
{%- set prev_changelog_contents = ctx.prev_changelog_file | read_file | safe -%}
{%- set insertion_flag = "<!-- version list -->" -%}
{%- set changelog_parts = prev_changelog_contents.split(insertion_flag, maxsplit=1) -%}
{%- set prev_changelog_top = changelog_parts[0] | trim -%}
{{ "%s\n\n%s\n" | format(prev_changelog_top, insertion_flag | trim) }}
{% for version, release in ctx.history.released.items() %}
{{ "## %s (%s)" | format(version.as_semver_tag(), release.tagged_date.strftime("%Y-%m-%d")) }}
{% for type_, commits in (release.elements | dictsort) if type_ != "unknown" %}
{{ "### %s" | format(type_ | title) }}
{% for commit in commits %}
{{ "* %s ([`%s`](%s))" | format(
    commit.descriptions[0] | capitalize,
    commit.hexsha[:7],
    commit.hexsha | commit_hash_url
) }}
{%- set lines = commit.message.split('\n') -%}
{%- set bullet_lines = [] -%}
{%- for line in lines[1:] -%}
{%- if line and (line.startswith('-') or line.startswith('*')) -%}
{%- set _ = bullet_lines.append(line) -%}
{%- endif -%}
{%- endfor -%}
{%- if bullet_lines -%}
{% for bline in bullet_lines %}
    {{ bline }}
{%- endfor -%}
{%- endif -%}
{% endfor %}
{% endfor %}
{%- endfor -%}
{%- set previous_changelog_bottom = changelog_parts[1] | trim -%}
{%- if previous_changelog_bottom | length > 0 -%}
{{ "\n%s\n" | format(previous_changelog_bottom) }}
{%- endif -%}

@codejedi365
Copy link
Contributor

For our use case, if the user wants to provide more details in the form of bullets in the commit message, those bullets should be reflected in the changelog.
See my previous response. If that doesn't answer your question fully, happy to expound.

Unfortunately, I don't think the provided example helps illustrate a real life use of what the bulleted information would be. Currently PSR's parsers support extracting additional paragraphs in the form of the prefix BREAKING CHANGE: which then in the default template will be consolidated and placed in a bulleted list at the bottom in a ### BREAKING CHANGES section.

Another feature that is not yet released (but will be soon) is the additional support of an additional paragraph prefix NOTICE: (reference issue: #223) which will similarly create a separate section ### ADDITIONAL RELEASE INFORMATION with a bulleted list of additional instructions/details for product consumers. The intent was to provide deprecation notices which are not breaking changes within themselves.

I recognize that the standard although detailed does not comment as much on what the content should be within commit sections. I personally use commitizen when I started out which helped frame what information should be described in a commit message. With those tools as a base and many personal experiences after, I have come to believe that the subject line is written from a user perspective (how does this change affect me [the end user]) and the commit body paragraph is written for the programmer that comes after you with a hint of how the specific change maps to the summary line. Then if you have additional info to give the users, use one of our two prefixes (or manually write/update the CHANGELOG). And lastly include any git footer prefixes that close issues, add references, or enrich the commit details (signed-off-by, etc.).

With my approach above, I don't find any additional information that needs to be provided to users that is not captured in these ways. What additional information are you trying to provide to users that wouldn't fit in this approach?


Lastly, something I forgot to mention earlier, If you are intending for the JIRA issue reference to be parsed as a linked issue then in its current format it will not be detected by PSR. I implemented PSR to mirror the 4 VCS's PSR supports with the caveat that it must be specified in the colon-defined git footer syntax (referenced by conventional commit standard). See Common Issue Identifier Detection

feat: Add changelog template x3

- this is a test
- hello

Closes: PA-232

@codejedi365
Copy link
Contributor

I should probably clarify now that I have discussed commit formatting above. The commit.descriptions list is a product of the following logic:

  1. Reverse word-wrapping (i.e. strip out single newlines) unless the following line does not start with an alphanumeric
  2. Reverse hyphen-wrapped words
  3. Split the resulting commit message body paragraphs into an array (i.e. split on double newlines)
  4. Insert the commit summary as the very first item in the array

Ah, that explains why .descriptions doesn't work for me. I need it split up by single newlines, not double. No matter, easy fix.

Partially. The first problem was from the double newline from the subject. I recently added support to detect single newline bulleted lists and it will not collapse those single newlines. In fact, it actually normalizes to insert an additional newline to meet the markdown style guidelines which then allow step 3 to properly separate into separate entries in commit.descriptions

@codejedi365
Copy link
Contributor

codejedi365 commented Jan 5, 2025

Not exactly what I want so will stick with custom template for now.

Understandable, I didn't put much effort in to this as it was a temporary fix and will be changed in the future. There was some weirdness with additional indents that I tried to fix with the word wrap filter and the indent parameter but it was hard to solve generically.

I have multiple services and would like to avoid duplicating the changelog template in different folders that mirrors our documentation structure. is there a way I can have multiple services use the same changelog?

Unfortunately not. Primarily because Jinja has a poor path resolution algorithm that attempts to prevent path traversal attacks but errors out if it ever detects a .. parent directory path even if it is valid within its sandboxed directory. One day I plan to open a PR for it but I haven't gotten around to it. I believe #845 matches your situation and I haven't gotten around to that fix yet either.

PSR does not yet support monorepos fully and this is one of those situations I believe.

@codejedi365
Copy link
Contributor

@marc-at-brightnight, I looked over your template. I think line 6 will cause you issues on the next release. You are looping through the entire list of releases again upon each new release rather than just the latest release. All the previous releases will be in the changelog contents as part of the update.

@marc-at-brightnight
Copy link
Author

With my approach above, I don't find any additional information that needs to be provided to users that is not captured in these ways. What additional information are you trying to provide to users that wouldn't fit in this approach?

Yeah I think I agree with that. For us, the purpose is exclusively for the programmer, not so much the user. Sometimes it is helpful to include additional detail beyond the commit header. The idea is to make sure this added detail is included in the changelog entry.

@marc-at-brightnight
Copy link
Author

@marc-at-brightnight, I looked over your template. I think line 6 will cause you issues on the next release. You are looping through the entire list of releases again upon each new release rather than just the latest release. All the previous releases will be in the changelog contents as part of the update.

Ouch! Ok I'll see how I fix that.

@codejedi365
Copy link
Contributor

codejedi365 commented Jan 5, 2025

Ouch! Ok I'll see how I fix that.

If you do a raw copy of the default template directory as I described in #1122, you should make the following change to meet your format:

diff a/templates/.components/changes.md.j2 b/templates/.components/changes.md.j2

  #}{%    set commit_descriptions = []
  %}{#
  #}{%    for commit in ns.commits
  %}{#      # Update the first line with reference links and if commit description
            # has more than one line, add the rest of the lines
-           # NOTE: This is specifically to make sure to not hide contents
-           # of squash commits (until parse support is added)
  #}{%      set description = "- %s" | format(format_commit_summary_line(commit))
  %}{%      if commit.descriptions | length > 1
+ %}{%        set lines = commit.message.split('\n')
+ %}{%        set bullet_lines = []
+ %}{%        for line in lines[1:]
+ %}{%          if line and (line.startswith('-') or line.startswith('*'))
+ %}{%            set _ = bullet_lines.append(line)
+ %}{%          endif
+ %}{%        endfor
+ %}{%        if bullet_lines
+ %}{%          set description = "%s\n    %s" | format(
+                 description,
+                 bullet_lines | join("\n    ")
+               )
+ %}{%        endif
  %}{%      endif
- %}{%      set description = description | autofit_text_width(max_line_width, hanging_indent)
  %}{{      commit_descriptions.append(description) | default("", true)
  }}{%    endfor

I have not yet tested this but I'm pretty sure its fairly close.


For reference this is recommended from #1122:

VIRTUAL_ENV=".venv"

cp -r "$VIRTUAL_ENV/lib/python3*/site-packages/semantic-release/data/templates/angular/md/ ./templates/

First change to porting the default templates to user defined templates is to hardcode the changelog file name into the changelog template if you want to support an updating changelog.

diff a/templates/CHANGELOG.md.j2 b/templates/CHANGELOG.md.j2

  #}{%  set insertion_flag = ctx.changelog_insertion_flag
+ %}{%. set this_file = "CHANGELOG.md"
  %}{%  set unreleased_commits = ctx.history.unreleased | dictsort
  %}{%  set releases = ctx.history.released.values() | list
  %}{#
  #}{%  if ctx.changelog_mode == "init"
  %}{%    include ".components/changelog_init.md.j2"
  %}{#
  #}{%  elif ctx.changelog_mode == "update"
+ %}{%    set prev_changelog_file = this_file
- %}{%    set prev_changelog_file = ctx.prev_changelog_file
  %}{%    include ".components/changelog_update.md.j2"
  %}{#
  #}{%  endif

@codejedi365
Copy link
Contributor

codejedi365 commented Jan 5, 2025

I have multiple services and would like to avoid duplicating the changelog template in different folders that mirrors our documentation structure. is there a way I can have multiple services use the same changelog?

Unfortunately not. Primarily because Jinja has a poor path resolution algorithm that attempts to prevent path traversal attacks but errors out if it ever detects a .. parent directory path even if it is valid within its sandboxed directory. One day I plan to open a PR for it but I haven't gotten around to it. I believe #845 matches your situation and I haven't gotten around to that fix yet either.

Since this is unavailable, I would modify my CI to copy a "template" directory into the proper locations prior to running PSR. PSR is not affected by a dirty working directory and unless you git add the files it will not commit any changes that it did not make.

./poweralpha/
├── docs/
│   ├── svrc1/
│   │   └── READMD.md
│   └── svrc2/
│       └── READMD.md
├── services/
│   ├── svrc1/
│   │   └── __main__.py
│   └── svrc2/
│       └── __main__.py
├── templates/
│   └── docs/
│       └── .base_changelog_template/
│           ├── .components/
│           │   ├── changelog_header.md.j2
│           │   ├── changelog_init.md.j2
│           │   ├── changelog_update.md.j2
│           │   ├── changes.md.j2                              <-- Update me with above diff
│           │   ├── first_release.md.j2
│           │   ├── macros.md.j2
│           │   ├── unreleased_changes.md.j2
│           │   └── versioned_changes.md.j2
│           ├── .release_notes.md.j2
│           └── CHANGELOG.md.j2                                <-- Update me with above diff
└── pyproject.toml

Then have my CI run the following:

cp -r templates/docs/.base_changelog_template templates/docs/srvc1
cp -r templates/docs/.base_changelog_template templates/docs/srvc2

semantic-release -v version

This would result in a (committed) directory structure like this after semantic-release version.

./poweralpha/
├── docs/
│   ├── svrc1/
│   │   ├── CHANGELOG.md                                  <-- Newly Created
│   │   └── READMD.md
│   └── svrc2/
│       ├── CHANGELOG.md                                  <-- Newly Created
│       └── READMD.md
├── services/
│   ├── svrc1/
│   │   └── __main__.py
│   └── svrc2/
│       └── __main__.py
├── templates/
│   └── docs/
│       └── .base_changelog_template/
│           ├── .components/
│           │   ├── changelog_header.md.j2
│           │   ├── changelog_init.md.j2
│           │   ├── changelog_update.md.j2
│           │   ├── changes.md.j2
│           │   ├── first_release.md.j2
│           │   ├── macros.md.j2
│           │   ├── unreleased_changes.md.j2
│           │   └── versioned_changes.md.j2
│           ├── .release_notes.md.j2
│           └── CHANGELOG.md.j2
└── pyproject.toml

@marc-at-brightnight
Copy link
Author

marc-at-brightnight commented Jan 5, 2025

Thanks for the great suggestions! Unfortunately, I can't seem to get the update mode to work. Here's what I have:

# cicd.yml

      - name: Reset [service] release check
        if: steps.release-check.outputs.[service]== 'true'
        run: |
          yq e -i '.[service] = false' semantic-release.yaml  # set the "[service]" entry to false
          git status
          git diff
          git add .
          mkdir -p documentation/templates/documentation/docs/[service]
          rsync -a documentation/templates/.base-changelog-template/ documentation/templates/documentation/docs/[service]/
          git status
      - name: Create [service] github release
        if: steps.release-check.outputs.[service] == 'true'
        uses: python-semantic-release/python-semantic-release@v9.15.2
        with:
          directory: poweralpha/services/[service]
          changelog: true
          commit: true
          push: true
          tag: true
          vcs_release: true
          github_token: ${{ secrets.GITHUB_TOKEN }}
# <root>/poweralpha/services/[service]/pyproject.toml
...
[tool.semantic_release.changelog]
template_dir = "../../../documentation/templates"
mode = "update"
exclude_commit_patterns = [".*\\[skip ci\\].*"]

[tool.semantic_release.changelog.default_templates]
changelog_file = "../../../documentation/docs/[service]/changelog.md"
...
<!-- <root>/documentation/docs/[service]/changelog.md

# CHANGELOG

<!-- version list -->

## v2.13.0 (2025-01-05)


## v0.1.0 (2025-01-05)

### Features

- Yet another feature ([`2dc5089`](https://github.com/BrightNight-Energy/poweralpha/commit/2dc508972c7fc0e98f200d1e7d0a3c0a9155897f))


## v2.12.0 (2025-01-05)

### Features

- Add changelog template x3 ([`780e9f4`](https://github.com/BrightNight-Energy/poweralpha/commit/780e9f407664d4050caea2acecdd1da2f7045f99))

- Include this commit as well in 2.12 ([`b61a717`](https://github.com/BrightNight-Energypoweralpha/commit/b61a7174a687cf81dac655188e47c62b9d43ec2d))

- Let's try this with a double new line, see what we get ([`5f5eadb`](https://github.com/BrightNight-Energypoweralpha/commit/5f5eadb5c60f64e2d94583007382591100b180c5))
    - double new line above
    - very good


## v2.11.1 (2025-01-03)

### Bug Fixes

- Add changelog template ([`7a4bf9f`](https://github.com/BrightNight-Energy/poweralpha/commit/7a4bf9f36495403fe5d4aceaf2062a1381f38c88))


## v2.11.0 (2024-12-31)

### Features

- Check semantic release is working ([`249fee2`](https://github.com/BrightNight-Energy/poweralpha/commit/249fee24bdd31f5cf6da2d8ab7687854ace3fc37))


## v2.10.0 (2024-12-23)

### Chores

- Add semantic release to each individual service ([`0058872`](https://github.com/BrightNight-Energy/poweralpha/commit/0058872e3bb032b52fe511bd14cb5127ece7a369))

### Continuous Integration

- Add Azure Static Web Apps workflow file
 ([`734d1d7`](https://github.com/BrightNight-Energy/poweralpha/commit/734d1d71d875216d60bd647dc4202b7cc09581e0))

3 issues I've found:

  1. It definitely seems to be in init mode, since there should be entries under v2.10.0.
  2. I made your suggested edits to CHANGELOG.md and changes.md. weirdly enough though, it seems like not having a double newline under the commit header seems to omit the bullets, as bullet 1 under v2.12.0 should have sub-bullets.
  3. I have no idea why the v0.1.0 is slipping in there under v2.13.0. That line should not be there.

Let me know if I need to create repro somewhere of a public monorepo

@codejedi365
Copy link
Contributor

3 issues I've found:

  1. It definitely seems to be in init mode, since there should be entries under v2.10.0.

It will default to init mode if there is no existing file. Do you have an existing file with an insertion flag at the defined prev_changelog_file? You can see this logic at the beginning of .components/changelog_update.md.j2.

  1. I made your suggested edits to CHANGELOG.md and changes.md. weirdly enough though, it seems like not having a double newline under the commit header seems to omit the bullets, as bullet 1 under v2.12.0 should have sub-bullets.

That was expected as we discussed above. The edge case of not having a double newline after the commit summary line. But I would say the template changes were a success due to sub-bullet 3 under v2.12.0. Do you want it like that? Just note that from the Markdown style guides (although it will render the same regardless) it should have a double newline between items in a bulleted list.

  1. I have no idea why the v0.1.0 is slipping in there under v2.13.0. That line should not be there.

That is weird, however the tag date is very recent. We sort tags by semver value so this is still weird. Did you run it more than once and were you at the correct HEAD (generally top of main/master)?

Let me know if I need to create repro somewhere of a public monorepo

At this point, it is probably required as that is what I was going to do next. PSR is impacted by your branching and merging strategy so that should be replicated as well. PSR is tested repeatedly on Git Flow, GitHub Flow, and Trunk Based Development with various release branch configurations to include merge & squash commit variants. ReleaseFlow and Monorepos are not part of the current test suite. Generally I ask for just the git graph output to get a visual of that.

git log --decorate --oneline --graph --all --topo-order

@codejedi365
Copy link
Contributor

So I started building a repo to simulate this. One thing I did not mention above is that in order to handle the release notes in a common way. You will need to make a few changes I didn't specify above.

# 1. Move .release_notes.md.j2 to root of templates directory
mv documentation/templates/.base_changelog_templates/.release_notes.md.j2 \
   documentation/templates/.release_notes.md.j2
  1. Make the following changes in the file to ensure files can be located
diff a/documentation/templates/.release_notes.md.j2 b/documentation/templates/.release_notes.md.j2

  #}{%  if releases | length == 1 and mask_initial_release
  %}{#    # On a first release, generate our special message
- #}{%    include ".components/first_release.md.j2"
+ #}{%    include ".base_changelog_templates/.components/first_release.md.j2"
  %}{%  else
  %}{#    # Not the first release so generate notes normally
- #}{%    include ".components/versioned_changes.md.j2"
+ #}{%    include ".base_changelog_templates/.components/versioned_changes.md.j2"
  -%}{#
  #}{%    if detailed_changes_link is defined
  %}{{      "\n"

@codejedi365
Copy link
Contributor

codejedi365 commented Jan 6, 2025

@marc-at-brightnight, I have messed around with a fake monorepo and I have effectively had PSR initialize a changelog and update it in the file structure you defined. Repo: https://github.com/codejedi365/psr-monorepo-poweralpha. I used a simple trunk based development as it was the fastest.

There are still a lot of not-so-great results from it because PSR does not have full support for monorepos at this time.

Problems I see:

  1. All components (aka your services) must all be versioned the same regardless of it there is a change that effects the service. Solution: implement a custom parser to filter by a scope prefix for that service.

  2. Changelog exclusion patterns are ignored if the commit by the other service causes a version bump. I recently caused this because I needed to prevent when a breaking change that normally was ignored triggered a bump. The resulting changelog would be blank for the reason it was bumped. This change effects monorepos more signifcantly where my simple exclude_commit_pattern of any not the current service did not remove the commit from the changelog. Solution: a custom parser will need to just throw a ParseError object instead

  3. Configuration uses the changelog.default_templates settings which really are not intended for custom changelogs but in this case it is the best way to handle it.

I ran it all locally using some custom scripts to simulate github actions

bash ./scripts/release-srvc1.sh && bash ./scripts/release-srvc2.sh

You can also test the release notes configuration via:

cd poweralpha/services/srvc1
semantic-release --noop changelog --post-to-release-tag srvc1-v1.2.0

@marc-at-brightnight
Copy link
Author

So I started building a repo to simulate this. One thing I did not mention above is that in order to handle the release notes in a common way. You will need to make a few changes I didn't specify above.

# 1. Move .release_notes.md.j2 to root of templates directory
mv documentation/templates/.base_changelog_templates/.release_notes.md.j2 \
   documentation/templates/.release_notes.md.j2
  1. Make the following changes in the file to ensure files can be located
diff a/documentation/templates/.release_notes.md.j2 b/documentation/templates/.release_notes.md.j2

  #}{%  if releases | length == 1 and mask_initial_release
  %}{#    # On a first release, generate our special message
- #}{%    include ".components/first_release.md.j2"
+ #}{%    include ".base_changelog_templates/.components/first_release.md.j2"
  %}{%  else
  %}{#    # Not the first release so generate notes normally
- #}{%    include ".components/versioned_changes.md.j2"
+ #}{%    include ".base_changelog_templates/.components/versioned_changes.md.j2"
  -%}{#
  #}{%    if detailed_changes_link is defined
  %}{{      "\n"

Not sure this change makes sense to me. If I have the following structure, wouldn't the relative path of .components/first_release.md.j2 be valid from .release_notes.md.j2?

documentation/templates
├── .base-changelog-template
│   ├── .components
│   │   ├── changelog_header.md.j2
│   │   ├── changelog_init.md.j2
│   │   ├── changelog_update.md.j2
│   │   ├── changes.md.j2
│   │   ├── first_release.md.j2
│   │   ├── macros.md.j2
│   │   ├── unreleased_changes.md.j2
│   │   └── versioned_changes.md.j2
│   ├── .release_notes.md.j2
│   └── changelog.md.j2
└── documentation
    └── docs
        └── svc1
            ├── .components
            │   ├── changelog_header.md.j2
            │   ├── changelog_init.md.j2
            │   ├── changelog_update.md.j2
            │   ├── changes.md.j2
            │   ├── first_release.md.j2
            │   ├── macros.md.j2
            │   ├── unreleased_changes.md.j2
            │   └── versioned_changes.md.j2
            ├── .release_notes.md.j2
            └── changelog.md.j2

@marc-at-brightnight
Copy link
Author

Appreciate the hard work on this 👍🏼

  1. All components (aka your services) must all be versioned the same regardless of it there is a change that effects the service. Solution: implement a custom parser to filter by a scope prefix for that service.

Yep, we initially tried to use dorny/paths-filter to figure out which services had changes. We realized quickly the issue with that is if you are developing a feature for svc1 but you have small changes in svc2, you will cause a version bump in 2 services, as opposed to just one. So we went with defining a top level yaml file for each service, where if set to true, trigger a release.

# semantic-release.yaml
svc1: true
svc2: false

In ci/cd, we parse the yaml file in a step and if true, we reset the flag and then release:

      - name: Reset svc1 release check
        if: steps.release-check.outputs.svc1 == 'true'  # this is the parsed value from the yaml to trigger the release
        run: |
          yq e -i '.svc1 = false' semantic-release.yaml  # set the "svc1" entry to false
          git add .

If the commit parser deems no release should be made based on the type of commit, the boolean flags are reset to false at the end of the workflow and the changes are committed.

This solution has the added benefit of working when you merge a new feature but you forget to deploy (we have manual deployment triggers) and someone else merges another feature on a different service. When they deploy, both services will be triggered for release, as the boolean flags for both services will be true.

  1. Changelog exclusion patterns are ignored if the commit by the other service causes a version bump. I recently caused this because I needed to prevent when a breaking change that normally was ignored triggered a bump. The resulting changelog would be blank for the reason it was bumped. This change effects monorepos more signifcantly where my simple exclude_commit_pattern of any not the current service did not remove the commit from the changelog. Solution: a custom parser will need to just throw a ParseError object instead

Mhm not sure I totally follow but this hasn't been an issue for us.

@codejedi365
Copy link
Contributor

Not sure this change makes sense to me. If I have the following structure, wouldn't the relative path of .components/first_release.md.j2 be valid from .release_notes.md.j2?

In order for PSR to detect that you have a custom release notes template, PSR only looks for if $TEMPLATE_DIR/.release_notes.md.j2 exists. Your changelog.template_dir is only documentation/templates. This is the reason for step 1: the file move. The second change of the include paths is dependent on the first, now that the file has moved up. You are also safe to use the base template directory because the release notes have no effect on the file location of the changelog nor is there any specifics related to which service it is.

@marc-at-brightnight
Copy link
Author

marc-at-brightnight commented Jan 6, 2025

Not sure this change makes sense to me. If I have the following structure, wouldn't the relative path of .components/first_release.md.j2 be valid from .release_notes.md.j2?

In order for PSR to detect that you have a custom release notes template, PSR only looks for if $TEMPLATE_DIR/.release_notes.md.j2 exists. Your changelog.template_dir is only documentation/templates. This is the reason for step 1: the file move. The second change of the include paths is dependent on the first, now that the file has moved up. You are also safe to use the base template directory because the release notes have no effect on the file location of the changelog nor is there any specifics related to which service it is.

Mhm interesting. I forked your repo and it seemed to work, even without moving .release_notes.md.j2 up one level. marc-at-brightnight/psr-monorepo-poweralpha@52cff2c (I invited you to be a collaborator on the repo, feel free to burn some GHA minutes)

EDIT: strange, I tried your way on another PR, and it didn't make a changelog entry: marc-at-brightnight/psr-monorepo-poweralpha@6b1041f

@codejedi365
Copy link
Contributor

codejedi365 commented Jan 6, 2025

Mhm interesting. I forked your repo and it seemed to work, even without moving .release_notes.md.j2 up one level.

Well since we copied the default templates the only way you would see a difference is if you specifically ran it against a version that had a commit body that has a bullet list in it (I forgot about this when creating my example project). The reason you have to move the release notes template is so that PSR detects your template and doesn't automatically fall back to the internal one. Your custom release notes templates import your custom changes template that displays commit bodies with hanging indented bullets

marc-at-brightnight/psr-monorepo-poweralpha@52cff2c (I invited you to be a collaborator on the repo, feel free to burn some GHA minutes)

Got the invite, I can try it out tonight.

EDIT: strange, I tried your way on another PR, and it didn't make a changelog entry: marc-at-brightnight/psr-monorepo-poweralpha@6b1041f

Will have to look into what didn't happen later. I was running everything locally.

@marc-at-brightnight
Copy link
Author

marc-at-brightnight commented Jan 7, 2025

Ok SO I gave it another shot and you were definitely right, it was using the default changelog template on the first PR.

The second PR seems to work wonderfully. What wasn't working earlier is resolved now, looks like the tags on the repo were behind what was already in the changelog so the changelog writer was intelligently omitting those entries so I just didn't give it enough credit.

The bad news is that I can't seem to replicate this success in my bigger repo. I have to deep dive tomorrow on why the sample monorepo is working and why its not working in my prod monorepo and what the differences might be. It's totally weird: it's writing the changelog in init mode (previous changelog is gettting wiped out so that's one problem), the changelog and version changes are getting committed to the repo BUT the github job fails with the following traceback. So I'm not sure why that's happening when the changelog entry is coming through, it is bizarre.

I'll follow up with what I find, just wanted to give a quick update...

ci/cd traceback in prod poweralpha repo
TemplateNotFound: '.base-changelog-templates/.components/versioned_changes.md.j2' not found in search path: '/github/workspace/documentation/templates'

Traceback (most recent call last):
  File "/psr/.venv/lib/python3.13/site-packages/semantic_release/__main__.py", line 15, in main
    cli_main(args=sys.argv[1:])
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/psr/.venv/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/psr/.venv/lib/python3.13/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/psr/.venv/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/psr/.venv/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/psr/.venv/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/psr/.venv/lib/python3.13/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/psr/.venv/lib/python3.13/site-packages/semantic_release/cli/commands/version.py", line 694, in version
    release_notes = generate_release_notes(
        hvcs_client,
    ...<4 lines>...
        mask_initial_release=runtime.changelog_mask_initial_release,
    )
  File "/psr/.venv/lib/python3.13/site-packages/semantic_release/cli/changelog_writer.py", line 270, in generate_release_notes
    return render_release_notes(
        release_notes_template_file=release_notes_tpl_file,
        template_env=release_notes_env,
    )
  File "/psr/.venv/lib/python3.13/site-packages/semantic_release/cli/changelog_writer.py", line 102, in render_release_notes
    release_notes = template.render().rstrip() + os.linesep
                    ~~~~~~~~~~~~~~~^^
  File "/psr/.venv/lib/python3.13/site-packages/jinja2/environment.py", line 1295, in render
    self.environment.handle_exception()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/psr/.venv/lib/python3.13/site-packages/jinja2/environment.py", line 942, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "/github/workspace/documentation/templates/.release_notes.md.j2", line 49, in top-level template code
    #}{%    include ".base-changelog-templates/.components/versioned_changes.md.j2"
  File "/psr/.venv/lib/python3.13/site-packages/jinja2/loaders.py", line 209, in get_source
    raise TemplateNotFound(
    ...<2 lines>...
    )

@codejedi365
Copy link
Contributor

codejedi365 commented Jan 7, 2025

This is a totally random thought but I see you chose kebab-case for the base templates directory. Since Jinja uses Python compiler type things under the hood, maybe switch to snake_case because Python fails to import kebab-case file paths (probably one of the dumbest pythonic rules ever).

@codejedi365
Copy link
Contributor

codejedi365 commented Jan 7, 2025

Yep, we initially tried to use dorny/paths-filter to figure out which services had changes. We realized quickly the issue with that is if you are developing a feature for svc1 but you have small changes in svc2, you will cause a version bump in 2 services, as opposed to just one. So we went with defining a top level yaml file for each service, where if set to true, trigger a release.

I didn't realize you were using a path filter so now that makes a bit more sense to how you were handling which service to release rather than solely relying on PSR to figure it out based on commits. The latter being how the open issues related to monorepos expect PSR to function.

  1. Changelog exclusion patterns are ignored if the commit by the other service causes a version bump. I recently caused this because I needed to prevent when a breaking change that normally was ignored triggered a bump. The resulting changelog would be blank for the reason it was bumped. This change effects monorepos more significantly where my simple exclude_commit_pattern of any not the current service did not remove the commit from the changelog. Solution: a custom parser will need to just throw a ParseError object instead

Mhm not sure I totally follow but this hasn't been an issue for us.

Its probably because I'm super picky about the content of changelogs but since you had different changelogs (one for each service) I figured you would only want relevant commits to service 1 included in the service 1 changelog and no service 2 commits included in the service 1 changelog. Without a custom parser to differentiate when a commit is only relevant to a specific service, all commits that cause a change will be included into either changelog. You can see this in the example repo I made:

<!-- documentation/docs/srvc1/changelog.md -->

## v1.2.0 (2025-01-06)

### Chores

- Fix comments in configuration ([`0a874b5`](https://github.com/codejedi365/psr-monorepo-poweralpha/commit/0a874b5b7a86f5c10b0403d670457d895c1411ea))

### Documentation

- **srvc1-changelog**: Add custom header to changelog ([`f194cb3`](https://github.com/codejedi365/psr-monorepo-poweralpha/commit/f194cb335b2bf8c1b8391b9f379735e422e4ea34))

### Features

- **srvc1-version**: Add version variable to service module ([`5d24983`](https://github.com/codejedi365/psr-monorepo-poweralpha/commit/5d24983d88b76e2d1c51706bb1fd5c24f66baf88))

- **srvc2-version**: Add version variable to service module ([`a526b84`](https://github.com/codejedi365/psr-monorepo-poweralpha/commit/a526b84af2e2138abd2545b04cf5bb331bf20079))

The above changelog excerpt came from the following unreleased commits.

* 8ad336e (HEAD -> main, tag: srvc2-v1.2.0, origin/main) chore(release): Release `service2@1.2.0` [skip ci]
* 8ea4e1d (tag: srvc1-v1.2.0) chore(release): Release `service1@1.2.0` [skip ci]
* a526b84 feat(srvc2-version): add version variable to service module       <-- Included but should NOT be
* 5d24983 feat(srvc1-version): add version variable to service module       <-- Included b/c I'm service 1 scoped
* 0db541d docs(srvc2-changelog): add custom header to changelog             <-- NOT Included as Im not the right scope
* f194cb3 docs(srvc1-changelog): add custom header to changelog
* 0a874b5 chore: fix comments in configuration                              <-- Included b/c I have no scope
* 761c1ba (tag: srvc2-v1.1.0) chore(release): Release `service2@1.1.0` [skip ci]

Note that I tried to solve this problem with exclude_commit_patterns using a negative lookahead regular expression and it works for any commit that does not cause a version bump (i.e. not feat, fix, or perf). The regular expression will attempt to match any conventional commit type with a scope prefix that is not the service itself (i.e. srvc1- for service 1).

@marc-at-brightnight
Copy link
Author

It ended up being really silly. Turns out I had a reference to ".base-changelog-template" and the folder was named ".base-changelog-templates". RE your comment regarding kebab-case, I have replaced this folder named with underscores.

Thanks again for all the superb support, definitely would have been quite lost without all the help. Cheers!

@marc-at-brightnight
Copy link
Author

just saw this comment.

Note that I tried to solve this problem with exclude_commit_patterns using a negative lookahead regular expression and it works for any commit that does not cause a version bump (i.e. not feat, fix, or perf). The regular expression will attempt to match any conventional commit type with a scope prefix that is not the service itself (i.e. srvc1- for service 1).

Ah fair enough, will give this a shot.

@marc-at-brightnight
Copy link
Author

I just observed another behavior that's a bit strange. I tested a double-release, where 2 services are marked for release at the same time. In this scenario, I would expect both services to receive the same changelog entry, but with their respective versions. However, what seemed to happen is that srvc1 (the first service to release based on the ordering in cicd.yml) had its changelog overwritten by srvc2's changelog. You can see this behavior in this commit.

For now, I will take measures to prevent a double-release, since it's not a common scenario and generally we would want unique changelog entries. However, if you have any idea why this is happening, I would be curious but if not, we can put it as a parking lot item, as you continue to further support for monorepos.

@codejedi365
Copy link
Contributor

@marc-at-brightnight, I'm not sure how the double-release differs from my offline test that calls both shell scripts sequentially. I feel this is part of your setup because for one it reinserted an "initial release" statement which means it didn't detect any previous tags?

I can't explain what happened in the other changelog. I do think the recursive copy of the base templates directory must only make one copy at a time otherwise it will write two changelogs when it releases a single service.

@marc-at-brightnight
Copy link
Author

I do think the recursive copy of the base templates directory must only make one copy at a time otherwise it will write two changelogs when it releases a single service.

This was the right instinct. I added rm -rf documentation/templates/documentation to each step before release and this solved the problem.

This repo is now setup very similar to how prod poweralpha is setup, if its helpful for future development: https://github.com/marc-at-brightnight/psr-monorepo-poweralpha

@codejedi365 codejedi365 removed the awaiting-reply Waiting for response label Jan 7, 2025
@codejedi365
Copy link
Contributor

I did work on a custom parser for monorepos yesterday (in line with path filtering in #614, and scoped variations discussed above), will try to do some testing now that I have an artificial repository. If this is of use or desire for you, then please let me know and it would be helpful if you are able to test it out further for your environment.

@marc-at-brightnight
Copy link
Author

marc-at-brightnight commented Jan 7, 2025

I did work on a custom parser for monorepos yesterday (in line with path filtering in #614, and scoped variations discussed above), will try to do some testing now that I have an artificial repository. If this is of use or desire for you, then please let me know and it would be helpful if you are able to test it out further for your environment.

I think it could certainly be of use to monorepos with strict separation between packages. However, we have a libraries folder where services use common functions within. Changing the name or arguments, for example, of one of these library functions would result in changes in many services. This could cause releases in multiple services if we relied on path filtering only. Therefore, we opted for a more manual solution, where the developer has to specify which services to release, as I described earlier.

@codejedi365
Copy link
Contributor

I think it could certainly be of use to monorepos with strict separation between packages. However, we have a libraries folder where services use common functions within. Changing the name or arguments, for example, of one of these library functions would result in changes in many services. This could cause releases in multiple services if we relied on path filtering only. Therefore, we opted for a more manual solution, where the developer has to specify which services to release, as I described earlier.

I think we are talking past each other. Yes, #614 wants path filtering, and yes with strict separation of packages they would not need your elegant solution to solve service release determination. I was not necessarily expecting to replace your release determination because of the unknown complexities of your system. If I could simplify it, I would, but ultimately I was only trying to solve the changelog issue described above which I had indicated would require a custom parser. I also added the scope relevancy as a filter and/or alternative to the path filtering variant, neither which would necessarily work for a shared library (thats new to me).

What is your opinion of the following?

only relevant commits to service 1 should be included in the service 1 changelog and no service 2 commits included in the service 1 changelog

@marc-at-brightnight
Copy link
Author

I think we are talking past each other. Yes, #614 wants path filtering, and yes with strict separation of packages they would not need your elegant solution to solve service release determination. I was not necessarily expecting to replace your release determination because of the unknown complexities of your system. If I could simplify it, I would, but ultimately I was only trying to solve the changelog issue described above which I had indicated would require a custom parser. I also added the scope relevancy as a filter and/or alternative to the path filtering variant, neither which would necessarily work for a shared library (thats new to me).

understood 👍🏼

What is your opinion of the following?

only relevant commits to service 1 should be included in the service 1 changelog and no service 2 commits included in the service 1 changelog

💯

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants