test: fix flaky key pair generation test

[tniessen]

There is a very small chance (about 0.4%) that OpenSSL will successfully decrypt a key without the correct passphrase and will then fail while parsing its ASN.1 structure. In those rare cases, the error message will be different.

Fixes: #22978

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• commit message follows commit guidelines

[nodejs-github-bot]

@tniessen build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/950/pipeline

[tniessen]

Suggesting to fast-track, please add 👍 here to approve.

[addaleax]

CI: https://ci.nodejs.org/job/node-test-pull-request/17344/

[Trott]

Would it make sense to repeat the encryption/decryption when that happens rather than allow the test to pass? (Not blocking. Just a suggestion. Like, if the decryption fails 100% of the time, we want the test to fail. That sort of thing.)

[tniessen]

@Trott Short answer: No.

Longer answer: We expect the signing / decryption operation using the generated private key to fail since the private key was encrypted with a custom passphrase.
We don't give the passphrase to OpenSSL, so OpenSSL will attempt to decrypt the private key without a password. When that happens, there are two possible outcomes:

1. OpenSSL detects that the decrypted message has an invalid padding (that's how people usually distinguish between "correctly decrypted" and "incorrectly decrypted") and throws the bad decrypt error, or
2. the padding is valid by chance (said 0.4%). The default padding is valid if the last n bytes of the decrypted plaintext have a value of n (for 1 <= n <= 16). So in those cases where the padding is considered correct by OpenSSL, it invokes the ASN1 parser which attempts to parse the (supposedly decrypted) private key. The parser fails, however, because the private key was not decrypted correctly, thus resulting in the asn1 encoding routines error.

Both results meet our expectations, but one is much more likely than the other.

[tniessen]

Thanks for reviewing! Landed in bad670c.