Dependabot should not do major updates by default

[MattIPv4]

Over the last day, dependabot has been going around GitHub and creating bump PRs for node-fetch. For a lot of folks, this has been bumping from 2.x to 3.x, a semver-major jump.

For those who aren't familiar with node-fetch, 3.x includes a really major breaking change for a lot of projects in that it only supports ESM-style Node.js, whereas many projects still use the CJS-style of writing Node.js. This makes the new major version completely breaking and incompatible for them.

This wave of PRs that are completely breaking for many projects has resulted in the update from dependabot now having a compatibility score of less than 25%.

It feels really odd to me that dependabot's default behavior is to automatically create PRs for breaking, major changes to libraries on projects across all of GitHub, unless you specifically have a config file in each of your repos to tell it not to.

I would kindly like to suggest that dependabot's default behaviour is changed to be minor and patch updates only, unless you specifically opt-in to having it automate major version updates. This to me would be more in line with behaviour in the ecosystem, such as with npm update which will only do minor and patch updates for you to give the best chance at non-breaking changes.

---

Edit: Maybe it's learning? After creating the breaking 3.x bump PR on one of my projects earlier today and me closing it, it has just opened the PR I would've expected it to open from the start, performing a minor version bump. 3.x bump PR, 2.x bump PR.

[jurre]

Thanks for opening this issue @MattIPv4, I am actually surprised by the initial PR that was opened, because for security updates Dependabot's default behavior is to update to the lowest version that fixes the vulnerability, and that is compatible with the existing dependencies.

When I try to replay the update, that's also what I'm seeing:

    ± package.json
    ~~~
    19c19
    <     "node-fetch": "^2.6.1",
    ---
    >     "node-fetch": "^2.6.7",
    ~~~

I am not quite sure why an update was opened for v3.x 🤔 I will dig into things further, but this definitely isn't the default behavior.

Edit: Ok, I have a theory on what happened:

We use GitHub's advisory DB as the source of truth for fixing vulnerable dependencies, and at the time the first PR was opened, 2.6.7 wasn't marked as "fixed". I see an update to the advisory was made between those two dates, and I am now figuring out if this is indeed the data that was changed.

[jurre]

Yes, I'm fairly sure that this is what happened, because of how we enqueue and process these updates I don't have rock solid evidence, but the fixed 2.6.7 version was added to the advisory shortly before this PR was opened and wasn't present before that, and since we enqueue a lot of these jobs for processing (especially with such a widely used dependency), it will take a while for them to get picked up, but the job will include the advisory info from when it was enqueued.

Combined with the fact that I can't reproduce the result with the advisory data that includes 2.6.7 as fixed but I can when I remove it, I think this is a pretty safe assumption.

TL;DR it was a race condition with an update to the advisory record.

[travi]

i want to add another perspective since i think PRs for major updates are incredibly valuable, whether for a vulnerability patch or for a normal update. PRs from dependabot are simply suggestions to update to a newer version that has become available. these PRs help a team understand that there is a newer version available and help to motivate a team to update. it is up to the recipient of the PR to determine if accepting the PR is going to break their project or not. ideally, this is handled with automated test coverage, but that is not a reality in many cases. in cases where automation is not thorough enough, the PR recipient can evaluate the contribution as they would any other to determine if it is acceptable.

This wave of PRs that are completely breaking for many projects has resulted in the update from dependabot now having a compatibility score of less than 25%.

i see this as the compatibility score working as designed. if upgrading projects past breaking changes means that the upgrade is incompatible with that amount of projects that attempted, that is useful information to others considering acceptance of the suggestion.

It feels really odd to me that dependabot's default behavior is to automatically create PRs for breaking, major changes

i agree that it would be best to suggest an upgrade to a non-major version that fixes a vulnerability in cases where such a fix is available, which sounds like is the existing behavior according to #10441 (comment). however, in the case where a non-breaking update is not available, opening the PR provides very useful information that a fix is available for the vulnerability. it is still on the recipient to decide what to do with the information, but without that information (even if an alert informs the team about the vulnerability existing), it can be unclear that a resolution is even available yet. many OSS project only maintain a single release line, so it may be the case that the fix is never backported to a non-breaking version, so moving forward may be the only way to fix. even if it takes some work to get the project ready to accept the suggestion, knowing that a path forward is available can be very valuable.

[jurre]

however, in the case where a non-breaking update is not available, opening the PR provides very useful information that a fix is available for the vulnerability.

Yep, this is how things work currently. We open a PR with whatever the smallest upgrade possible is, even if the only option is a breaking change.