[CLOUDP-352109] Run MCK E2E tests against OCI published helm chart

[viveksinghggits]

Summary

As part of our work to move towards OCI compatible container registries for our helm chart, we are also planning to run our E2E tests against the helm chart that we publish to OCI. This will make sure that we are testing in our E2E what we are providing to our customers.
As part of this effort we have already raised a PR that starts publishing our helm chart to the OCI container registry during dev/staging workflows.
This PR goes and changes our E2E tests to start consuming the helm chart from OCI registry instead of the local helm chart repo.

Additional unrelated change is to fix kubectl-mongodb location that is used for gke code snippets. The fix is to add missing workdir path suffix.

Proof of Work

Successful CI on this PR.
I also ran the test e2e_replica_set_migration locally and it was successful. The logs are here.

publish_helm_chart example output:

[2025/11/05 22:11:19.857] Running command 'subprocess.exec' in function 'publish_helm_chart' (step 7 of 7).
[2025/11/05 22:11:19.919] INFO     2025-11-05 21:11:19,919 [publish_helm_chart]  Packaging chart: mongodb-kubernetes with Version: 0.0.0+690bbc0f836fbf0007154375
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Successfully executed: helm package --version 0.0.0+690bbc0f836fbf0007154375 helm_chart
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Successfully packaged chart and saved it to: /data/mci/55da5d1be5488b41a9ec4e7d02041a86/src/github.com/mongodb/mongodb-kubernetes/mongodb-kubernetes-0.0.0+690bbc0f836fbf0007154375.tgz
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Determined OCI Registry: oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts
[2025/11/05 22:11:19.959] INFO     2025-11-05 21:11:19,959 [publish_helm_chart]  Pushing chart to registry: oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts
[2025/11/05 22:11:20.848] INFO     2025-11-05 21:11:20,848 [publish_helm_chart]  Successfully executed: helm push mongodb-kubernetes-0.0.0+690bbc0f836fbf0007154375.tgz oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts
[2025/11/05 22:11:20.848] INFO     2025-11-05 21:11:20,848 [publish_helm_chart]  Helm Chart mongodb-kubernetes:0.0.0+690bbc0f836fbf0007154375 was published successfully!
[2025/11/05 22:11:20.858] Finished command 'subprocess.exec' in function 'publish_helm_chart' (step 7 of 7) in 1.001053371s.

Downloading the chart with helm pull oci://268558157000.dkr.ecr.us-east-1.amazonaws.com/dev/mongodb/helm-charts/mongodb-kubernetes --untar --version 0.0.0+690bbc0f836fbf0007154375 you can see that the Chart.yaml version was properly updated:

apiVersion: v2
description: 'MongoDB Controllers for Kubernetes translate the human knowledge of
  creating a MongoDB instance into a scalable, repeatable, and standardized method. '
home: https://github.com/mongodb/mongodb-kubernetes
icon: https://mongodb-images-new.s3.eu-west-1.amazonaws.com/leaf-green-dark.png
keywords:
  - mongodb
  - database
  - nosql
kubeVersion: '>=1.16-0'
maintainers:
  - email: support@mongodb.com
    name: MongoDB
name: mongodb-kubernetes
type: application
version: 0.0.0+690bbc0f836fbf0007154375

Also run release test and e2e smoke tests are also passing (ignore ibm_power, they are fixed in #573)

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you added changelog file?
  • use skip-changelog label if not needed
  • refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

[github-actions]

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.6.0 Release Notes

New Features

• MongoDBCommunity: Added support to configure custom cluster domain via newly introduced spec.clusterDomain resource field. If spec.clusterDomain is not set, environment variable CLUSTER_DOMAIN is used as cluster domain. If the environment variable CLUSTER_DOMAIN is also not set, operator falls back to cluster.local as default cluster domain.
• Helm Chart: Introduced two new helm fields operator.podSecurityContext and operator.securityContext that can be used to configure securityContext for Operator deployment through Helm Chart.
• MongoDBSearch: Switch to gRPC and mTLS for internal communication
  Since MCK 1.4 the mongod and mongot processess communicated using the MongoDB Wire Protocol and used keyfile authentication. This release switches that to gRPC with mTLS authentication. gRPC will allow for load-balancing search queries against multiple mongot processes in the future, and mTLS decouples the internal cluster authentication mode and credentials among mongod processes from the connection to the mongot process. The Operator will automatically enable gRPC for existing and new workloads, and will enable mTLS authentication if both Database Server and MongoDBSearch resource are configured for TLS.
• MongoDBSearch: MongoDB deployments using X509 internal cluster authentication are now supported. Previously MongoDB Search required SCRAM authentication among members of a MongoDB replica set. Note: SCRAM client authentication is still required, this change merely relaxes the requirements on internal cluster authentication.
• MongoDBSearch: Updated the default mongodb/mongodb-search image version to 0.55.0. This is the version MCK uses if .spec.version is not specified.

Bug Fixes

• Fixed parsing of the customEnvVars Helm value when values contain = characters.
• ReplicaSet: Blocked disabling TLS and changing member count simultaneously. These operations must now be applied separately to prevent configuration inconsistencies.

Other Changes

• Simplified MongoDB Search setup: Removed the custom Search Coordinator polyfill (a piece of compatibility code previously needed to add the required permissions), as MongoDB 8.2.0 and later now include the necessary permissions via the built-in searchCoordinator role.
• kubectl-mongodb plugin: cosign, the signing tool that is used to sign kubectl-mongodb plugin binaries, has been updated to version 3.0.2. With this change, released binaries will be bundled with .bundle files containing both signature and certificate information. For more information on how to verify signatures using new cosign version please refer to -> https://github.com/sigstore/cosign/blob/v3.0.2/doc/cosign_verify-blob.md

[MaciejKaras]

q: how can we instruct this function to use local helm charts for local tests? Ideally we should be able to set this somewhere in the context files and override it by default locally, for example by specifying it in local-defaults-context.

[viveksinghggits]

Yes, that's the plan. I am working on the change that will make sure that we are able to run the tests locally. Will push the changes soon.

[mircea-cosbuc]

nit: Mixed casing

[mircea-cosbuc]

nit: Prefer explicit import for code navigation.