-
Notifications
You must be signed in to change notification settings - Fork 255
Custom changelog seems to ignore update mode #1132
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
Hello @marc-at-brightnight, Did you happen to look at the Incrementally Updating Changelog Template section of the documentation? Was this helpful or not understandable? Secondly, if I understand your desire, you would like the commit body to be a bulleted list rather than how the default template displays the full commit body? I think I first read it as you wanted only the subject line which is what most people ask me about, but by the look of the template it is more of a template formatting change. Either way I have provided instructions on what components to modify for handling the commit body within issue #1122. There is a comment for how to only display the commit subject line but in your case it points out the section you should modify for formatting details. The issue also includes more detail than the above documentation of what to change from the default to a custom template which would preserve other features (ex updating) that PSR has provided. Also, note that we already split the commit messages up by newlines and include them in the Lastly, I would recommend using the |
Separately, may I ask what information are you trying to preserve and include from the commit body? Over the past few months and for a few more in the future, I have been significantly changing the default changelog contents and on that roadmap is to reduce down to only displaying the commit subject line. The precursor to this is the support for parsing squash commits so that they can be each presented per version. Both are very close to publish. I have also added the features to parse out PR/MR numbers and parse out issue resolution numbers which are available to consumers in each |
I overlooked it. I think maybe changing the title might help, perhaps "Using a changelog template in update mode"?
If I use the default template, this is what I see in the changelog:
This was my commit:
Instead, I would like my changelog entry to read:
Which is why I turned to a custom template. It seems like the default template only includes the subject header. For our use case, if the user wants to provide more details in the form of bullets in the commit message, those bullets should be reflected in the changelog.
Got it 👍🏼
Very good 👍🏼 Thank you very much @codejedi365 for the extremely good work on this repo and diligence on provided A+ user support. It is refreshing to encounter such professionalism on an open-source project and it does not go unnoticed 🫡 |
See my previous response. If that doesn't answer your question fully, happy to expound. |
Thanks, I'll consider it. That specific section is for updating a custom changelog previously made to now be an updating changelog, not exactly to explain update mode. Update mode is explained at the top of the page under the default templates. I'll try to add some clarity there.
I believe you are hitting an unexpected edge case of the parser due to your commit format to give you a result that only includes the commit subject. From the conventional commit standard (i.e. the newer variant of the angular guidelines), It is expected for there to be a double newline (a blank line) between the commit subject and the commit body. Since your commit does not have a double newline, somehow the bullets are being stripped during parsing. If you amend your commit to be the following then I suspect you will get a different result (and maybe the one you desire):
I should probably clarify now that I have discussed commit formatting above. The
That means a lot, as user support takes a lot more time than expected. Thank you! |
Interesting. I tried that and got the following:
with commit:
Not exactly what I want so will stick with custom template for now.
Ah, that explains why I think I have what I need (mostly just needed the pointer to the right section of the documentation but the extra context and help has been super helpful), except I have one more question, if you don't mind: My latest template, in case it is helpful for others encountering this thread: (sorry for bad formatting, my first jinja2 template){%- set prev_changelog_contents = ctx.prev_changelog_file | read_file | safe -%}
{%- set insertion_flag = "<!-- version list -->" -%}
{%- set changelog_parts = prev_changelog_contents.split(insertion_flag, maxsplit=1) -%}
{%- set prev_changelog_top = changelog_parts[0] | trim -%}
{{ "%s\n\n%s\n" | format(prev_changelog_top, insertion_flag | trim) }}
{% for version, release in ctx.history.released.items() %}
{{ "## %s (%s)" | format(version.as_semver_tag(), release.tagged_date.strftime("%Y-%m-%d")) }}
{% for type_, commits in (release.elements | dictsort) if type_ != "unknown" %}
{{ "### %s" | format(type_ | title) }}
{% for commit in commits %}
{{ "* %s ([`%s`](%s))" | format(
commit.descriptions[0] | capitalize,
commit.hexsha[:7],
commit.hexsha | commit_hash_url
) }}
{%- set lines = commit.message.split('\n') -%}
{%- set bullet_lines = [] -%}
{%- for line in lines[1:] -%}
{%- if line and (line.startswith('-') or line.startswith('*')) -%}
{%- set _ = bullet_lines.append(line) -%}
{%- endif -%}
{%- endfor -%}
{%- if bullet_lines -%}
{% for bline in bullet_lines %}
{{ bline }}
{%- endfor -%}
{%- endif -%}
{% endfor %}
{% endfor %}
{%- endfor -%}
{%- set previous_changelog_bottom = changelog_parts[1] | trim -%}
{%- if previous_changelog_bottom | length > 0 -%}
{{ "\n%s\n" | format(previous_changelog_bottom) }}
{%- endif -%} |
Unfortunately, I don't think the provided example helps illustrate a real life use of what the bulleted information would be. Currently PSR's parsers support extracting additional paragraphs in the form of the prefix Another feature that is not yet released (but will be soon) is the additional support of an additional paragraph prefix I recognize that the standard although detailed does not comment as much on what the content should be within commit sections. I personally use With my approach above, I don't find any additional information that needs to be provided to users that is not captured in these ways. What additional information are you trying to provide to users that wouldn't fit in this approach? Lastly, something I forgot to mention earlier, If you are intending for the JIRA issue reference to be parsed as a linked issue then in its current format it will not be detected by PSR. I implemented PSR to mirror the 4 VCS's PSR supports with the caveat that it must be specified in the colon-defined git footer syntax (referenced by conventional commit standard). See Common Issue Identifier Detection
|
Partially. The first problem was from the double newline from the subject. I recently added support to detect single newline bulleted lists and it will not collapse those single newlines. In fact, it actually normalizes to insert an additional newline to meet the markdown style guidelines which then allow step 3 to properly separate into separate entries in |
Understandable, I didn't put much effort in to this as it was a temporary fix and will be changed in the future. There was some weirdness with additional indents that I tried to fix with the word wrap filter and the indent parameter but it was hard to solve generically.
Unfortunately not. Primarily because Jinja has a poor path resolution algorithm that attempts to prevent path traversal attacks but errors out if it ever detects a PSR does not yet support monorepos fully and this is one of those situations I believe. |
@marc-at-brightnight, I looked over your template. I think line 6 will cause you issues on the next release. You are looping through the entire list of releases again upon each new release rather than just the latest release. All the previous releases will be in the changelog contents as part of the update. |
Yeah I think I agree with that. For us, the purpose is exclusively for the programmer, not so much the user. Sometimes it is helpful to include additional detail beyond the commit header. The idea is to make sure this added detail is included in the changelog entry. |
Ouch! Ok I'll see how I fix that. |
If you do a raw copy of the default template directory as I described in #1122, you should make the following change to meet your format: diff a/templates/.components/changes.md.j2 b/templates/.components/changes.md.j2
#}{% set commit_descriptions = []
%}{#
#}{% for commit in ns.commits
%}{# # Update the first line with reference links and if commit description
# has more than one line, add the rest of the lines
- # NOTE: This is specifically to make sure to not hide contents
- # of squash commits (until parse support is added)
#}{% set description = "- %s" | format(format_commit_summary_line(commit))
%}{% if commit.descriptions | length > 1
+ %}{% set lines = commit.message.split('\n')
+ %}{% set bullet_lines = []
+ %}{% for line in lines[1:]
+ %}{% if line and (line.startswith('-') or line.startswith('*'))
+ %}{% set _ = bullet_lines.append(line)
+ %}{% endif
+ %}{% endfor
+ %}{% if bullet_lines
+ %}{% set description = "%s\n %s" | format(
+ description,
+ bullet_lines | join("\n ")
+ )
+ %}{% endif
%}{% endif
- %}{% set description = description | autofit_text_width(max_line_width, hanging_indent)
%}{{ commit_descriptions.append(description) | default("", true)
}}{% endfor I have not yet tested this but I'm pretty sure its fairly close. For reference this is recommended from #1122: VIRTUAL_ENV=".venv"
cp -r "$VIRTUAL_ENV/lib/python3*/site-packages/semantic-release/data/templates/angular/md/ ./templates/ First change to porting the default templates to user defined templates is to hardcode the changelog file name into the changelog template if you want to support an updating changelog. diff a/templates/CHANGELOG.md.j2 b/templates/CHANGELOG.md.j2
#}{% set insertion_flag = ctx.changelog_insertion_flag
+ %}{%. set this_file = "CHANGELOG.md"
%}{% set unreleased_commits = ctx.history.unreleased | dictsort
%}{% set releases = ctx.history.released.values() | list
%}{#
#}{% if ctx.changelog_mode == "init"
%}{% include ".components/changelog_init.md.j2"
%}{#
#}{% elif ctx.changelog_mode == "update"
+ %}{% set prev_changelog_file = this_file
- %}{% set prev_changelog_file = ctx.prev_changelog_file
%}{% include ".components/changelog_update.md.j2"
%}{#
#}{% endif |
Since this is unavailable, I would modify my CI to copy a "template" directory into the proper locations prior to running PSR. PSR is not affected by a dirty working directory and unless you
Then have my CI run the following: cp -r templates/docs/.base_changelog_template templates/docs/srvc1
cp -r templates/docs/.base_changelog_template templates/docs/srvc2
semantic-release -v version This would result in a (committed) directory structure like this after
|
Thanks for the great suggestions! Unfortunately, I can't seem to get the update mode to work. Here's what I have:
3 issues I've found:
Let me know if I need to create repro somewhere of a public monorepo |
It will default to init mode if there is no existing file. Do you have an existing file with an insertion flag at the defined prev_changelog_file? You can see this logic at the beginning of
That was expected as we discussed above. The edge case of not having a double newline after the commit summary line. But I would say the template changes were a success due to sub-bullet 3 under v2.12.0. Do you want it like that? Just note that from the Markdown style guides (although it will render the same regardless) it should have a double newline between items in a bulleted list.
That is weird, however the tag date is very recent. We sort tags by semver value so this is still weird. Did you run it more than once and were you at the correct HEAD (generally top of main/master)?
At this point, it is probably required as that is what I was going to do next. PSR is impacted by your branching and merging strategy so that should be replicated as well. PSR is tested repeatedly on Git Flow, GitHub Flow, and Trunk Based Development with various release branch configurations to include merge & squash commit variants. ReleaseFlow and Monorepos are not part of the current test suite. Generally I ask for just the git graph output to get a visual of that. git log --decorate --oneline --graph --all --topo-order |
So I started building a repo to simulate this. One thing I did not mention above is that in order to handle the release notes in a common way. You will need to make a few changes I didn't specify above. # 1. Move .release_notes.md.j2 to root of templates directory
mv documentation/templates/.base_changelog_templates/.release_notes.md.j2 \
documentation/templates/.release_notes.md.j2
diff a/documentation/templates/.release_notes.md.j2 b/documentation/templates/.release_notes.md.j2
#}{% if releases | length == 1 and mask_initial_release
%}{# # On a first release, generate our special message
- #}{% include ".components/first_release.md.j2"
+ #}{% include ".base_changelog_templates/.components/first_release.md.j2"
%}{% else
%}{# # Not the first release so generate notes normally
- #}{% include ".components/versioned_changes.md.j2"
+ #}{% include ".base_changelog_templates/.components/versioned_changes.md.j2"
-%}{#
#}{% if detailed_changes_link is defined
%}{{ "\n" |
@marc-at-brightnight, I have messed around with a fake monorepo and I have effectively had PSR initialize a changelog and update it in the file structure you defined. Repo: https://github.com/codejedi365/psr-monorepo-poweralpha. I used a simple trunk based development as it was the fastest. There are still a lot of not-so-great results from it because PSR does not have full support for monorepos at this time. Problems I see:
I ran it all locally using some custom scripts to simulate github actions bash ./scripts/release-srvc1.sh && bash ./scripts/release-srvc2.sh You can also test the release notes configuration via: cd poweralpha/services/srvc1
semantic-release --noop changelog --post-to-release-tag srvc1-v1.2.0 |
Not sure this change makes sense to me. If I have the following structure, wouldn't the relative path of
|
Appreciate the hard work on this 👍🏼
Yep, we initially tried to use dorny/paths-filter to figure out which services had changes. We realized quickly the issue with that is if you are developing a feature for svc1 but you have small changes in svc2, you will cause a version bump in 2 services, as opposed to just one. So we went with defining a top level yaml file for each service, where if set to
In ci/cd, we parse the yaml file in a step and if true, we reset the flag and then release:
If the commit parser deems no release should be made based on the type of commit, the boolean flags are reset to This solution has the added benefit of working when you merge a new feature but you forget to deploy (we have manual deployment triggers) and someone else merges another feature on a different service. When they deploy, both services will be triggered for release, as the boolean flags for both services will be
Mhm not sure I totally follow but this hasn't been an issue for us. |
In order for PSR to detect that you have a custom release notes template, PSR only looks for if |
Mhm interesting. I forked your repo and it seemed to work, even without moving EDIT: strange, I tried your way on another PR, and it didn't make a changelog entry: marc-at-brightnight/psr-monorepo-poweralpha@6b1041f |
Well since we copied the default templates the only way you would see a difference is if you specifically ran it against a version that had a commit body that has a bullet list in it (I forgot about this when creating my example project). The reason you have to move the release notes template is so that PSR detects your template and doesn't automatically fall back to the internal one. Your custom release notes templates import your custom changes template that displays commit bodies with hanging indented bullets
Got the invite, I can try it out tonight.
Will have to look into what didn't happen later. I was running everything locally. |
Ok SO I gave it another shot and you were definitely right, it was using the default changelog template on the first PR. The second PR seems to work wonderfully. What wasn't working earlier is resolved now, looks like the tags on the repo were behind what was already in the changelog so the changelog writer was intelligently omitting those entries so I just didn't give it enough credit. The bad news is that I can't seem to replicate this success in my bigger repo. I have to deep dive tomorrow on why the sample monorepo is working and why its not working in my prod monorepo and what the differences might be. It's totally weird: it's writing the changelog in init mode (previous changelog is gettting wiped out so that's one problem), the changelog and version changes are getting committed to the repo BUT the github job fails with the following traceback. So I'm not sure why that's happening when the changelog entry is coming through, it is bizarre. I'll follow up with what I find, just wanted to give a quick update... ci/cd traceback in prod poweralpha repo
|
This is a totally random thought but I see you chose kebab-case for the base templates directory. Since Jinja uses Python compiler type things under the hood, maybe switch to snake_case because Python fails to import kebab-case file paths (probably one of the dumbest pythonic rules ever). |
I didn't realize you were using a path filter so now that makes a bit more sense to how you were handling which service to release rather than solely relying on PSR to figure it out based on commits. The latter being how the open issues related to monorepos expect PSR to function.
Its probably because I'm super picky about the content of changelogs but since you had different changelogs (one for each service) I figured you would only want relevant commits to service 1 included in the service 1 changelog and no service 2 commits included in the service 1 changelog. Without a custom parser to differentiate when a commit is only relevant to a specific service, all commits that cause a change will be included into either changelog. You can see this in the example repo I made: <!-- documentation/docs/srvc1/changelog.md -->
## v1.2.0 (2025-01-06)
### Chores
- Fix comments in configuration ([`0a874b5`](https://github.com/codejedi365/psr-monorepo-poweralpha/commit/0a874b5b7a86f5c10b0403d670457d895c1411ea))
### Documentation
- **srvc1-changelog**: Add custom header to changelog ([`f194cb3`](https://github.com/codejedi365/psr-monorepo-poweralpha/commit/f194cb335b2bf8c1b8391b9f379735e422e4ea34))
### Features
- **srvc1-version**: Add version variable to service module ([`5d24983`](https://github.com/codejedi365/psr-monorepo-poweralpha/commit/5d24983d88b76e2d1c51706bb1fd5c24f66baf88))
- **srvc2-version**: Add version variable to service module ([`a526b84`](https://github.com/codejedi365/psr-monorepo-poweralpha/commit/a526b84af2e2138abd2545b04cf5bb331bf20079)) The above changelog excerpt came from the following unreleased commits.
Note that I tried to solve this problem with |
It ended up being really silly. Turns out I had a reference to ".base-changelog-template" and the folder was named ".base-changelog-templates". RE your comment regarding kebab-case, I have replaced this folder named with underscores. Thanks again for all the superb support, definitely would have been quite lost without all the help. Cheers! |
just saw this comment.
Ah fair enough, will give this a shot. |
I just observed another behavior that's a bit strange. I tested a double-release, where 2 services are marked for release at the same time. In this scenario, I would expect both services to receive the same changelog entry, but with their respective versions. However, what seemed to happen is that srvc1 (the first service to release based on the ordering in cicd.yml) had its changelog overwritten by srvc2's changelog. You can see this behavior in this commit. For now, I will take measures to prevent a double-release, since it's not a common scenario and generally we would want unique changelog entries. However, if you have any idea why this is happening, I would be curious but if not, we can put it as a parking lot item, as you continue to further support for monorepos. |
@marc-at-brightnight, I'm not sure how the double-release differs from my offline test that calls both shell scripts sequentially. I feel this is part of your setup because for one it reinserted an "initial release" statement which means it didn't detect any previous tags? I can't explain what happened in the other changelog. I do think the recursive copy of the base templates directory must only make one copy at a time otherwise it will write two changelogs when it releases a single service. |
This was the right instinct. I added This repo is now setup very similar to how prod poweralpha is setup, if its helpful for future development: https://github.com/marc-at-brightnight/psr-monorepo-poweralpha |
I did work on a custom parser for monorepos yesterday (in line with path filtering in #614, and scoped variations discussed above), will try to do some testing now that I have an artificial repository. If this is of use or desire for you, then please let me know and it would be helpful if you are able to test it out further for your environment. |
I think it could certainly be of use to monorepos with strict separation between packages. However, we have a libraries folder where services use common functions within. Changing the name or arguments, for example, of one of these library functions would result in changes in many services. This could cause releases in multiple services if we relied on path filtering only. Therefore, we opted for a more manual solution, where the developer has to specify which services to release, as I described earlier. |
I think we are talking past each other. Yes, #614 wants path filtering, and yes with strict separation of packages they would not need your elegant solution to solve service release determination. I was not necessarily expecting to replace your release determination because of the unknown complexities of your system. If I could simplify it, I would, but ultimately I was only trying to solve the changelog issue described above which I had indicated would require a custom parser. I also added the scope relevancy as a filter and/or alternative to the path filtering variant, neither which would necessarily work for a shared library (thats new to me). What is your opinion of the following? only relevant commits to service 1 should be included in the service 1 changelog and no service 2 commits included in the service 1 changelog |
understood 👍🏼
💯 |
Question
How do I make sure the changelog entries operate in 'update' mode, even with a custom changelog? I noticed that the update mode was working perfectly until I switched to a custom template.
My end goal is simply to include the body of the commit in the changelog entry, hence why I have a custom template.
Configuration
Semantic Release Configuration
Additional context
my template (if there is an easier way to do this, please let me know):
The text was updated successfully, but these errors were encountered: