How to adjust persistent volume claim size? (Type: ReplicaSet)

[hi-usui]

I looked at the YAML reference at https://docs.mongodb.com/kubernetes-operator/master/tutorial/deploy-replica-set/ and have been trying to tweak the following YAML:

apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: rocketchat-mdb
spec:
  members: 3
  type: ReplicaSet
  version: "4.2.6"
  persistent: true
  podSpec:
    memory: 512M
    persistence:
      multiple:
        data:
          storage: 1Gi
        journal:
          storage: 500M
        logs:
          storage: 500M

However, each member of the replica set still has 10Gi per PVC. How can I correctly change it?

[bznein]

Hello @uberkelium, you might be referring to the enterprise operator.

If so, the documentation on how to achieve what you are asking can be found here:
https://docs.mongodb.com/kubernetes-operator/stable/tutorial/resize-pv-storage/

[hi-usui]

Hello @uberkelium, you might be referring to the enterprise operator.

If so, the documentation on how to achieve what you are asking can be found here:
https://docs.mongodb.com/kubernetes-operator/stable/tutorial/resize-pv-storage/

Oh, sorry. I mixed up the documentation between the two. No, I'm not using the MongoDB Enterprise Kubernetes Operator, but I was looking at that documentation by mistake. Now that you mention it though, I am a little bit confused on the differences between the two. Community can do everything that Enterprise can, except you have to manage replica sets and sharding yourself, correct? This is the only difference I can see.

Anyway, I'm trying to resize the replica set PVCs from this project.

[bznein]

From Release 0.1.0 we added the possibility to specify arbitrary configurations from the stateful set through spec.statefulSet.spec. We are still in the process of creating examples but you should be able to achieve what you are trying by adding your custom requirements to spec.statefulSet.spec.volumeClaimTemplates.spec.resources. (I believe it is the same issue that was raised here: #66)

Regarding the differences between Community and Enterprise: Enterprise operator has support for many more features, such as, but not limited to:

• Sharded Clusters
• Standalones
• LDAP authentication and authorization
• Full support of Ops manager, which provides automation, monitoring and backup

Feel free to reach out if you have further questions!

[hi-usui]

Thanks for the advice @bznein

From Release 0.1.0 we added the possibility to specify arbitrary configurations from the stateful set through spec.statefulSet.spec. We are still in the process of creating examples but you should be able to achieve what you are trying by adding your custom requirements to spec.statefulSet.spec.volumeClaimTemplates.spec.resources. (I believe it is the same issue that was raised here: #66)

Regarding the differences between Community and Enterprise: Enterprise operator has support for many more features, such as, but not limited to:

* Sharded Clusters

* Standalones

* LDAP authentication and authorization

* Full support of Ops manager, which provides automation, monitoring and backup

Feel free to reach out if you have further questions!

I think it is not working as intended. There are 2 outcomes that I tried:

1. Use name "data-volume" for the pvc template
2. Use a different name than "data-volume"

For # 1, the mongodb-kubernetes-operator crashes
For # 2, two pvc templates are used, one "data-volume" and the other that is specified in MongoDB spec.
So, my understanding is that the operator is adamant about adding data-volume, and that any spec.statefulSet.spec.volumeClaimTemplates.spec.resources specified will only be an addition to the default data-volume

Can you let me know if I am following your advice correctly?
My MongoDB spec:

apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: rocketchat-mdb
spec:
  members: 3
  type: ReplicaSet
  version: "4.2.6"
  persistent: true
  statefulSet:
    spec:
      volumeClaimTemplates:
        - metadata:
            name: rocketchat
          spec:
            accessModes: [ "ReadWriteOnce" ]
            resources:
              requests:
                storage: 1Gi

Output

root@primary:~/mongodb-kubernetes-operator# k get pvc
NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
data-volume-rocketchat-mdb-0   Bound    pvc-2aa31773-7a37-487d-a056-a0096a36a8c2   10Gi       RWO            rook-ceph-block   50s
rocketchat-rocketchat-mdb-0    Bound    pvc-efb9ca59-6bfd-4380-9214-3360f53b5185   1Gi        RWO            rook-ceph-block   50s
rocketchat-srv                 Bound    pvc-ffe3a59b-310a-4389-80e3-8d3472ee782c   1Gi        RWO            rook-ceph-block   51s
root@primary:~/mongodb-kubernetes-operator# k describe statefulset/rocketchat-mdb
Name:               rocketchat-mdb
Namespace:          default
CreationTimestamp:  Thu, 20 Aug 2020 21:49:08 +0000
Selector:           app=rocketchat-mdb-svc
Labels:             app=rocketchat-mdb-svc
Annotations:        <none>
Replicas:           3 desired | 1 total
Update Strategy:    RollingUpdate
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app=rocketchat-mdb-svc
  Service Account:  mongodb-kubernetes-operator
  Init Containers:
   mongod-posthook:
    Image:      quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.2
    Port:       <none>
    Host Port:  <none>
    Command:
      cp
      version-upgrade-hook
      /hooks/version-upgrade
    Environment:  <none>
    Mounts:
      /hooks from hooks (rw)
  Containers:
   mongodb-agent:
    Image:      quay.io/mongodb/mongodb-agent:10.15.1.6468-1
    Port:       <none>
    Host Port:  <none>
    Command:
      agent/mongodb-agent
      -cluster=/var/lib/automation/config/automation-config
      -skipMongoStart
      -noDaemonize
      -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
      -serveStatusPort=5000
    Readiness:  exec [/var/lib/mongodb-mms-automation/probes/readinessprobe] delay=5s timeout=1s period=10s #success=1 #failure=240
    Environment:
      AGENT_STATUS_FILEPATH:     /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
      MANAGED_SECURITY_CONTEXT:  true
    Mounts:
      /data from data-volume (rw)
      /var/lib/automation/config from automation-config (ro)
      /var/log/mongodb-mms-automation/healthstatus from healthstatus (rw)
   mongod:
    Image:      mongo:4.2.6
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sh
      -c

      # run post-start hook to handle version changes
      /hooks/version-upgrade

      # wait for config to be created by the agent
      while [ ! -f /data/automation-mongod.conf ]; do sleep 3 ; done ; sleep 2 ;

      # start mongod with this configuration
      exec mongod -f /data/automation-mongod.conf ;

    Environment:
      AGENT_STATUS_FILEPATH:     /healthstatus/agent-health-status.json
      MANAGED_SECURITY_CONTEXT:  true
    Mounts:
      /data from data-volume (rw)
      /healthstatus from healthstatus (rw)
      /hooks from hooks (rw)
  Volumes:
   healthstatus:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
   hooks:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
   automation-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rocketchat-mdb-config
    Optional:  false
Volume Claims:
  Name:          data-volume
  StorageClass:
  Labels:        <none>
  Annotations:   <none>
  Capacity:      10G
  Access Modes:  [ReadWriteOnce]
  Name:          rocketchat
  StorageClass:
  Labels:        <none>
  Annotations:   <none>
  Capacity:      1Gi
  Access Modes:  [ReadWriteOnce]
Events:
  Type    Reason            Age   From                    Message
  ----    ------            ----  ----                    -------
  Normal  SuccessfulCreate  53s   statefulset-controller  create Claim rocketchat-rocketchat-mdb-0 Pod rocketchat-mdb-0 in StatefulSet rocketchat-mdb success
  Normal  SuccessfulCreate  53s   statefulset-controller  create Claim data-volume-rocketchat-mdb-0 Pod rocketchat-mdb-0 in StatefulSet rocketchat-mdb success
  Normal  SuccessfulCreate  53s   statefulset-controller  create Pod rocketchat-mdb-0 in StatefulSet rocketchat-mdb successful

[hi-usui]

Hack solution: If it is currently not possible to set PVC size for MongoDB in statefulSet, there is an awkward solution which requires manual intervention.

1. Create the replicaSet the ordinary way
2. Copy the statefulSet info: k get statefulset -o=yaml
3. Save the statefulSet info to statefulset.yaml
4. Modify persistentVolumeClaim template with the desired size
5. Delete everything and apply the statefulSet: k delete statefulset.apps/<statefulSet name>; k delete --force pod -l app=<pod label>; k delete pvc $(k get pvc | grep data-volume | awk '{print $1}'); k delete pv $(k get pv | grep data-volume | awk '{print $1}'); k apply -f statefulset.yaml

Example:
k get statefulset -l app=rocketchat-mdb-svc -o=yaml > statefulset.yaml; sed -i "s/10G/1Gi/g" statefulset.yaml; k delete statefulset.apps/rocketchat-mdb; k delete --force pod -l app=rocketchat-mdb-svc; k delete pvc $(k get pvc | grep data-volume | awk '{print $1}'); k delete pv $(k get pv | grep data-volume | awk '{print $1}'); k apply -f statefulset.yaml

The statefulSet's modification will not survive if it is deleted.

Output:

root@primary:~/mongodb-kubernetes-operator# k delete statefulset.apps/rocketchat-mdb; k delete --force pod -l app=rocketchat-mdb-svc; k delete pvc $(k get pvc | grep data-volume | awk '{print $1}'); k delete pv $(k get pv | grep data-volume | awk '{print $1}'); k apply
statefulset.apps "rocketchat-mdb" deleted
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "rocketchat-mdb-0" force deleted
pod "rocketchat-mdb-1" force deleted
pod "rocketchat-mdb-2" force deleted
persistentvolumeclaim "data-volume-rocketchat-mdb-0" deleted
persistentvolumeclaim "data-volume-rocketchat-mdb-1" deleted
persistentvolumeclaim "data-volume-rocketchat-mdb-2" deleted
persistentvolume "pvc-6fc55ddf-b72e-419a-b867-23db4f2a871c" deleted
persistentvolume "pvc-c19b0f2c-5c79-4048-9564-193eaf3a4a3f" deleted
persistentvolume "pvc-d9c183cf-e73a-4011-9302-28a73ee182a7" deleted
statefulset.apps/rocketchat-mdb created
root@primary:~/mongodb-kubernetes-operator# k get pvc
NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
data-volume-rocketchat-mdb-0   Bound    pvc-97f0ac84-b5b7-4084-ae19-6d80fddb7e0b   1Gi        RWO            rook-ceph-block   4s
rocketchat-srv                 Bound    pvc-2d27b088-7530-4594-bd58-16287b98e000   1Gi        RWO            rook-ceph-block   30m

[gubjanos]

@uberkelium I have run into the same issue as you. I did some research and what I see currently is there is no way with the community operator to overwrite the data-volume with a custom-tailored one. Let me elaborate on this:
The replicaset controller will always create a pvc named data-volume, with 10G store and some other default settings, regardless of your MongoDB custom resource as you can see here:

mongodb-kubernetes-operator/pkg/controller/mongodb/replica_set_controller.go

Line 628 in ab14dc5

statefulset.WithVolumeClaim(dataVolumeName, defaultPvc()),

Also here you can see, that the container's /data will be mounted to this pvc thus there is no easy way to bypass this. If you specify the volume again:

 statefulSet:
    spec:
      volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            storageClassName: "local-storage" 

The operator is going to crash (see the log below).

E0826 11:41:01.739841       1 runtime.go:78] Observed a panic: &reflect.ValueError{Method:"reflect.Value.Elem", Kind:0x19} (reflect: call of reflect.Value.Elem on struct Value)
goroutine 180 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1392500, 0xc00000ff60)
        /root/go/pkg/mod/k8s.io/apimachinery@v0.17.5/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /root/go/pkg/mod/k8s.io/apimachinery@v0.17.5/pkg/util/runtime/runtime.go:48 +0x82
panic(0x1392500, 0xc00000ff60)
        /usr/local/go/src/runtime/panic.go:969 +0x166
reflect.Value.Elem(0x145f880, 0xc0000dce00, 0x99, 0x0, 0x0, 0x0)
        /usr/local/go/src/reflect/value.go:827 +0x1a4
github.com/imdario/mergo.resolveValues(0x145f880, 0xc0000dce00, 0x145f880, 0xc0000dcfc0, 0x0, 0x0, 0x0, 0x8, 0x0, 0x0, ...)
        /root/go/pkg/mod/github.com/imdario/mergo@v0.3.9/mergo.go:66 +0xb7
github.com/imdario/mergo.merge(0x145f880, 0xc0000dce00, 0x145f880, 0xc0000dcfc0, 0xc000559b90, 0x2, 0x2, 0xc0005045b8, 0xc000559b78)
        /root/go/pkg/mod/github.com/imdario/mergo@v0.3.9/merge.go:314 +0xc9
github.com/imdario/mergo.Merge(...)
        /root/go/pkg/mod/github.com/imdario/mergo@v0.3.9/merge.go:260
github.com/mongodb/mongodb-kubernetes-operator/pkg/kube/statefulset.mergeVolumeClaimTemplates(0xc0000dc8c0, 0x1, 0x1, 0xc0000dc700, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
        /go/pkg/kube/statefulset/statefulset.go:300 +0x4dc
github.com/mongodb/mongodb-kubernetes-operator/pkg/kube/statefulset.mergeStatefulSetSpecs(0xc0005041c0, 0xc00000e820, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/pkg/kube/statefulset/statefulset.go:276 +0x1ca
github.com/mongodb/mongodb-kubernetes-operator/pkg/kube/statefulset.WithCustomSpecs.func1(0xc000838500)
        /go/pkg/kube/statefulset/statefulset.go:260 +0x11d
github.com/mongodb/mongodb-kubernetes-operator/pkg/kube/statefulset.Apply.func1(0xc000838500)
        /go/pkg/kube/statefulset/statefulset.go:180 +0x4d
github.com/mongodb/mongodb-kubernetes-operator/pkg/controller/mongodb.(*ReplicaSetReconciler).createOrUpdateStatefulSet(0xc00040ffb0, 0x129bb77, 0x7, 0xc0003be6b0, 0xe, 0xc0001bc330, 0x9, 0x0, 0x0, 0xc0001bc339, ...)
        /go/pkg/controller/mongodb/replica_set_controller.go:307 +0x1ed

I think a solution could be to modify the operator code not to include the default pvc in case a volumeclaimtemplate is defined with the name "data-volume" in the MongoDB crd definition. @bznein Do you think it is something worth working on? Should I come up with a PR for this? Or could you suggest a way to bypass this limitation?

Thanks in advance

[gubjanos]

Or maybe it is just an issue with the merging introduced in #118? @fabianlindfors

[gubjanos]

Isn't an & missing from here?

mongodb-kubernetes-operator/pkg/kube/statefulset/statefulset.go

Line 300 in 3b8ea12

if err := mergo.Merge(defaultMountsMap[idx], overrideMount, mergo.WithAppendSlice, mergo.WithOverride); err != nil {

The correct one should be:

if err := mergo.Merge(&defaultMountsMap[idx], overrideMount, mergo.WithAppendSlice, mergo.WithOverride); err != nil {

Mergo doc states that dst must be a pointer to a struct in Merge: https://godoc.org/github.com/imdario/mergo

Also it is used like this on volume mounts:

mongodb-kubernetes-operator/pkg/kube/podtemplatespec/podspec_template.go

Line 296 in 3b8ea12

if err := mergo.Merge(&defaultMount, overrideMount, mergo.WithAppendSlice); err != nil { //nolint

[bznein]

Hello @gubjanos and @uberkelium, and thanks for your investigation!

There was indeed a bug in our implementation (partially is what @gubjanos pointed out in their last comment) which has now been found. A fix will be merged soon, and will be available in the next release!

[williamtrelawny]

Sorry to necro this thread, but I was facing the same issue in 2025 and I figure I'd leave my final solution that worked for me here for any that come across this. Note that since the OP there has been the addition of a separate logs-volume pvc as well.

apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: test-mongodb
  namespace: mongodb
spec:
...
  statefulSet:
    spec:
      volumeClaimTemplates:
      - metadata:
          name: data-volume
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 1Gi
      - metadata:
          name: logs-volume
        spec:
          accessModes: [ "ReadWriteOnce" ]
          resources:
            requests:
              storage: 1Gi