Auto-generator doesn't choose most current release as base diff

[jbergstroem]

While trying out the new release-notes auto-generation feature, I entered a new tag (patch-level; 1.8.1, previous being 1.8.0) and the generator chose to diff against an older tag: v1.7.1. If there's too much complexity in the magic of figuring out what $previous means, perhaps allow the choice of tag to diff against?

Relevant repo: https://github.com/jbergstroem/hadolint-gh-action

Aside: not a big fan of the author being part of the changelog; but I'm probably a minority here.

[MylesBorins]

oh hey @jbergstroem been a minute! This definitely seems like a bug, I wonder if it is at all related to the v1 tag that seems to have been published on the same date as v1.8.0. Could you try deleting that tag (if it isn't being used for anyting) and let me know if that helps?

[PromoFaux]

I'm seeing this too, Just to copy what I posted in the announce thread:

---

I'm a little confused about the auto-generate release notes feature (appreciate it's still in preview)

Playing about with it to see what it does, I am not really sure how it's deciding what to put into the release notes, take for example on pi-hole/pi-hole, we have these tags/releases:

(Ignore the top one, it's a rolling draft)

When I add a tag of, say v5.6, and click the auto-generate button, it generates notes for contributions since 5.3.1 - Should it not be taking the most recent tag before it and going from there? (or perhaps we could have an option to manually enter the "since" tag)

Now, new stuff for release is not yet in the master branch, rather development, but the same thing happens no matter which branch I choose. Would like to use this feature, as it can be quite tedious if we forget to update the rolling draft before release, so any insight would be appreciated... thanks!

[jbergstroem]

oh hey @jbergstroem been a minute!

It has!

This definitely seems like a bug, I wonder if it is at all related to the v1 tag that seems to have been published on the same date as v1.8.0. Could you try deleting that tag (if it isn't being used for anyting) and let me know if that helps?

I unfortunately can't. The way versions for github actions is usually by maintaining major tags in sync with latest feature/patch (another example of this pattern at actions/checkout here: https://github.com/actions/checkout/tags).

Happy to debug in a separate repo, can look at that later.

[PromoFaux]

I've been able to reproduce it on a play repo, too, but can't see anything obvious that would cause it. I did have a hunch that it might have been something to do with having a tag that wasn't connected to a release, after seeing one in each of mine and @jbergstroem's screenshots

https://github.com/PromoFaux/TestRepo/

And actually, even after publishing the v4 release, if I draft a v5, I get the same thing

Happy to grant access to anyone who wants to debug it if needed

[MylesBorins]

we released a fix for version resolution. Can you please try again and let me know if you see improvements?

[MylesBorins]

I'm not 100% but I think that moving from using pep440 to determine previous release instead of semver might fix some of the issues we are seeing. I'm not 100% that it will fix the issue but it does seem like part of the weirdness is a mix of semver and non semver compliant tags.

In the case of the action with the v1 tag there might be even more weirdness in the UI and we might need to do some additional work to figure this out.

[jbergstroem]

Thanks for elaborating. Point me to how I can assist!

[PromoFaux]

I think that moving from using pep440 to determine previous release instead of semver might fix some of the issues we are seeing

Please forgive a very naive question, but I'm a big fan of the ol' work smarter not harder philosophy... Is there anything stopping just taking the previous release/tag chronologically?

[MylesBorins]

@PromoFaux the last tag chronologically may be a pre-release.

[PromoFaux]

Hmm, yeah OK that makes sense I think. Is the mechanism in the background looking at tags or at Github releases? If the latter, are pre-releases not marked as such?

[MylesBorins]

As a quick update we are going to be doing a spike on the current algorithm and examine the various edge cases identified here. We hope to find a solution to fix as many of these edge cases as possible in the coming weeks

[MylesBorins]

Hey all. We just rolled out what should hopefully be a fix for many edge cases folks have identified. Can you please give release note generation another go and let us know how it works out. Thank you!

[jbergstroem]

@MylesBorins my use case is fixed.

[PromoFaux]

@MylesBorins - Still not working for us on (at least):

pi-hole/AdminLTE (prev tag = v5.8)

pi-hole/FTL (prev tag = v5.11)

pi-hole/pi-hole (prev tag = v5.6)

Something I'm noting between all three here is that the auto generation appears to be going back to the last tag that had a three part version number - we tend not to add the .0 patch part on new minor versions (though we do include .0 on a new major, interestingly... not really given it much though before)

[MylesBorins]

Thanks for sharing the examples folks. Escalating to egnineering

[jimAScott]

Hi there,

I'm also facing the same issue. The last 6 releases I have created all try and Diff between the release I made on the 27 Oct 2021?
I am incrementing the the following way (Major.Minor.Hotfix version)
25.4
25.3
25.3
25.2.1
25.2
25.1
24.7
24.6.1
24.6

It would be good to have an update.

[MylesBorins]

While those version ranges are all valid pep440, they are not valid semver which confuses our current algorithm. We are exploring switching to pep440 to see if we can make a better experience for lots of folks and at the same time giving a clear warning when we were not able to accurately determine the range.

[planetf1]

I have been trying to create a release for https://github.com/odpi/egeria. We've been doing monthly released with very fixed names (.... ...2.7,2.8,2.9,2.10,2.11,3.0,3.1,3.2 etc) and I'm now doing 3.3 & see the release notes diff goes back to 2.8. I should add, each tag has a 'V' (uppercase) as a prefix. The schedule is pretty fixed and I can't see inconsistencies.

I tried this just now -- with the latest fix, and am still getting an incorrect diff

[MylesBorins]

that's for sharing, escalating to engineering

[muvaf]

Hey! We're hitting a similar issue in https://github.com/crossplane/provider-aws . I want to generate release note for v0.21.0 which I expect to work with the diff between v0.21.0 and v0.20.3 but it takes v0.13.0 for some reason. The following is the diff line at the end of the generated release notes:

**Full Changelog**: https://github.com/crossplane/provider-aws/compare/v0.13.0...v0.21.0

The latest release is v0.20.3 and the following is the list of the latest tags. They follow a very similar pattern and we were able to get the diff logic work correct in the earlier releases. Happy to help debugging in any way.

My two cents, it'd be great if we can choose the base tag as configuration in the UI for the generation so that it doesn't have to know which tag it should be based on.

[muvaf]

@MylesBorins We are using a bot to backport commits from master to release branches but that can't keep the commit hash as the same. Is there a way to make release branches compatible with the note generation or even having release branches are not OK for the tool?

[achimnol]

I have the exact same problem with @muvaf. Support for release branches with cherry-picked backports would be great! Backported commits usually preserve the issue number reference to the original PR in the commit message's title, so I think it should be possible to associate the related PR for them.

I'm also leaving my own git trailer like Backported-From: main (21.09) and Backported-To: 21.03 to ensure future mechanical analysis or automation to distinguish cherry-picked backport commits. Maybe supporting a custom git trailer like Associated-Pull-Request: 123 would also be a good trade-off for cases where backported commit's titles do not contain the PR number.

For reference, here are some of my repositories that use the "cherry-picked backports with release branches" policy are:

• lablup/backend.ai-manager
• lablup/backend.ai-agent
• lablup/backend.ai-common

[cyx2]

Hi there, we have recently changed our naming strategy from v0.11.1 to v2021-11-29.0. Our auto-generate release notes always refers back to the last release of the old format. Is there any way to change this other than deleting all of the old tags?

Thank you 🙏

@MylesBorins

[MylesBorins]

Hey @cyx2. It seems like you have moved from semver to a calendar based versioning scheme. There are a variety of projects using calver that you might want to explore using rather than what you are using right now. Particularly most project use . separators between dates rather than -. We are working on a change based on pep440 to support any arbitrary number of dot separated integers, so moving to a scheme like v2021.11.29.0 would work after we roll out the change.

In the mean time you are indeed kind of stuck. Our algorithm right now looks for previous valid semver associated with a release, but perhaps we should short circuit that and look for the previous release in the case that the proposed new tag itself is not valid.

Thanks for reporting

[cyx2]

Very helpful response, thank you! Is there a way I can subscribe to notifications for the pep440 change?

[pelletier]

If it helps troubleshooting it, I'm facing the same issue on https://github.com/pelletier/go-toml. I am trying to generate the changelog for a new tag v2.0.0-beta.5 on the v2 branch. I am expecting the generated changelog to range from the latest commit on the v2 branch, up to and excluding the last released commit on the same branch, in that case v2.0.0-beta.4. Instead, the changelog is generated all the way back to v1.8.1, for no reason that I can tell. Note that I was able to successfully use this feature when releasing v2.0.0-beta.4, so it looks like a regression.

[MylesBorins]

Our algorithm does not look at RCs as a previous release. If we did that it wouldn't be possible to create a new release properly when ready to promote from RC

[pelletier]

Thank, good to know. Any plan on allowing to pick a "since" manually? No problem if not, I'll just look at something else to generate release notes.

[MylesBorins]

The generate notes API allows you to manually set the range. We are scoping something for the UI

[MylesBorins]

Hey all,

We have a new feature flagged version determination algorithm we'd love to test out. If you are still having issues with getting the correct previous release we'd love to flag in your repo and see if it fixes things. Please respond here with the repo you'd like us to turn it on for

[mourner]

@MylesBorins thanks for the thorough response! Totally understand the issue with cherry-picking, but it shouldn't be a big problem in practice (any PRs that get cherry-picked in the release branch usually also land on the main branch, and we can do light editing of generated PRs afterwards anyway to fix any inconsistencies between the two).

In the case of Leaflet, the detached commits were a legacy consequence of supporting Bower many years ago (our release script created a new commit that added the built files, and then tagged that). We just got rid of this. However, it's still possible to determine the correct list of PRs between the two tags — if you go through a chain of this commit's parents, you eventually land on the main branch, and can use that as a base point.

We have a similar issue in https://github.com/mapbox/mapbox-gl-js — there's no "detached commit" problem, but we do use release branches, and currently the button calculates the list to v2.7.0 starting from v1.1.1, even though we had a ton of releases in between. I assume the same solution (looping through parents until you land on a common merge base) would work here too.

It is possible that you might be able to get everything working by using our generate notes API directly and manually setting the version range. We are in the process of trying to surface this in the UI right now.

Yes, definitely. I couldn't derive the right query parameters in the notes request yet, but perhaps they're documented somewhere? (The UI doesn't specify the base tag on request).

[mourner]

E.g. this git command gives a correct list of PRs that landed to the main branch since the release, so it should be possible with the release-notes tool as well:

git rev-list --pretty=oneline v1.7.1..master

[cyx2]

Hi @MylesBorins - this is working well for us and we'd like to turn it on for 10gen/leafylink - is this possible

[MylesBorins]

@cyx2 added! lmk if it works

[lkiesow]

I've just tried the feature in two different repositories and both fail to recognize the previous release correctly:

• In https://github.com/opencast/pyCA creating a new v4.5 (v4.4 is the latest release), the algorithm recognizes v1.0.1 as the previous release.
• In https://github.com/opencast/opencast creating 10.6 (10.5 is the latest release), the algorithm recognizes 1.6.0 as the previous release

[MylesBorins]

Hey all,

We've just rolled out the new release regex to all repositories. Please take a look and see if it fixes your issues.

One of the issues folks have been running into here are repositories that are using advanced release strategies with release branches. The new regex is not going to help in those cases.

Our release generating tool is built on top of Pull-Requests and requires that your PRs target the same target branch that the release is created from.

I'm going to mark this thread as answered. Please let me know if you run into issues and we can open new discussions for additional bugs or features

[lkiesow]

• Works now for github.com/opencast/pyCA 🎉
• No luck for github.com/opencast/opencast. But that's using release branches, which does make things a bit more complex. I think you could get a solution which works for that as well. E.g. picking the merge base between the latest two releases and then picking all PRs based on that but I understand if you want to keep the algorithm simple for now.

[mourner]

@MylesBorins would it be possible to expose / document the API endpoint where you could specify two commits of the main branch between which the changelog would be generated?

[MylesBorins]

https://docs.github.com/en/rest/reference/repos#generate-release-notes-content-for-a-release

[WillHalto]

👋 hey @PromoFaux ! The release note generation won't use the same commit as both the start and end of a release range - it will always look for something before the current targeted commit to use as the previous version. In your https://github.com/pi-hole/pi-hole example, the current target of the new tag v5.7 is master, whose latest commit b5e0f14 is also referenced by tag v5.6. So both v5.7 and v5.6 are pointing to the same commit. I'd think that once any new commits are added to master, this will behave how you expect!

[PromoFaux]

Ah.. thanks @WillHalto that makes a sort of sense, I think because we are so used to just keeping a draft release as we merge into development branches, I was expecting to be able to generate the release notes before development was then merged into master... but thanks for the explanation... I'll give it a go when next we release!

[ross-nordstrom]

@MylesBorins has the team considered a prompt for the base tag? It seems like that would give more flexibility and avoid confusion about the default behavior?

[shoskensMagics]

We're using a year-based version strategy (v2022.6.0 > v2022.7.0) and this doesn't work at all it seems. The v2022.7.0 release does not compare to any previous release (it just generates a link to all commits). This shouldn't be too different from a v1-2-3 scheme I would think. Anything that would hinder the recognition of the base tag here?

[KhizerRehan]

Hi, @PromoFaux How to get Automated release notes from just 1 previous tag i can see that release notes are created but it is using develop branch latest tag. Which is incorrect i want to choose latest tag of master branch although the last tag was created from master branch. Any help would be appreciated.

[jimmcslim]

Where do we find this 'release regex'? I've been looking but cannot find it in Settings.

Update: if its this there's no mention about regular expression support.

[jimmcslim]

Raised a new issue that will hopefully get clarified #12873

[planetf1]

Not working for our https://github.com/odpi/egeria releases (which we do monthly)- I end up having to edit the list manually as it insists on going back many many releases. Most commits get done on master. We do branch to a release of the form egeria-release-36 & tag as 'V3.6' format on a release - we've adopted a similar pattern for the last 1-2 years. On occasion we backport from master to the release if we find a regression in testing.

[MylesBorins]

Hey all,

we are working on a feature to allow you to select the previous tag manually. If you would like to try it out please comment in this thread with the repository you would like us to enable the preview feature on

[SchrodingersGat]

@MylesBorins hey Myles, we would love to try out this feature on https://github.com/inventree/inventree

Please ping me if you can enable it for us to try! Cheers

[dalevfenton]

@MylesBorins we would like to try the beta on this repo https://github.com/castingnetworks/apps-monorepo

[MylesBorins]

@dalevfenton the feature is now available for all dot-com customers so you should already have it

[SampsonCrowley]

@MylesBorins is there a reason we're limited to previous release tag instead of using the same option for target: a previous_commitish ? it's really limiting in generating notes for specific tags to only be able to reference other tags in "what's changed" being able to just pass a plain commit hash would be extremely useful for workflows that have dev builds "mini releases" for QA to be able to see exactly what has changed in their image agains the current development sprint

[anthony-langford]

We'd like to try it on all our repositories. https://github.com/SecondCloset/silver

[kevin-wise]

Selecting it manually is a good workaround, but I'm confused as to why it's not finding the prior releases automatically. I have several repos. All of them use semantic versioning beginning with a v. Nothing fancy at all. Yet it almost never gets the previous release correct. Would be happy to help you troubleshoot. There are certainly no shortage of examples.

Auto-generated release notes are a great idea, but getting the right previous release is pretty key to making it useful.

[LyleDavis]

I'm slightly confused by the behaviour here, what does it select by default as the previous release?

I was expecting it to ignore pre-releases but that doesn't seem to be the case.

We have a fairly simple versioning strategy, by year. For example, v2022.1.1 then v2022.2.1 then v2022.3.1. Whenever we create a release we mark it as "pre-release" until it has gone to production, at which point it is marked as the latest release. If we find a bug in the release, we fix it in master and then cut another, this time incrementing the final number. E.g. v2022.1.1 would be the first attempt at the first release of 2022, v2022.1.2 would be the second attempt at the first release of 2022.

Unfortunately it seems that when the second attempt at the release is cut, it uses the previous pre-release as the basis for the release notes despite it not being considered the latest release.

[ross-nordstrom]

See this thread and add your repo to try their beta feature letting you pick a previous release manually when generating your release notes: #5975 (comment)

[LyleDavis]

I considered it but that doesn't really help with the enforcement side of things really. It is very easy for people to click the big green button 😅

I think that by default, pre-releases should be disregarded when generating release notes. I believe it does do that for semantic versioning strategies, but not when your versioning strategy is slightly different.

[ldurazo]

Bump

No way to select the last tag automatically instead of manually yet?

[fumieval]

At the moment, GitHub picks the most recent tag when "Previous tag: auto" is chosen. However, this causes confusion in our usecase because we create nightly tags; the release note should be comparing against the most recent release, not a nightly tag.

While it's not a critical issue because there's a dropdown list to pick a previous tag manually, using the latest release instead of the latest tag seems like a no-brainer improvement to me.

[pcharlan]

We're seeing "auto" make very strange choices.

What I want: choose the most recent tag whose commit is an ancestor of the current tag's commit. (We use annotated tags.)

For bonus points, limit to semver tags, but what we're seeing is seemingly randomly skipping past 5 or so tags in history, or using something from many dozens of tags ago.

(Furthermore, once the tag is selected, the changelog doesn't even list the changes since that tag. Sometimes it will be one change that was from that chosen tag, and nothing since. And even when selecting the tag I want from the drop-down instead of using auto, the changelog is wrong, but I get that this particular issue is about the choice of the "from" tag, not the changelog that is produced from that.

We have no release.yml configured in this repo.