What is the best approach for PersistentVolumeClaim

General Discussions
1

Cluster information:

Kubernetes version: 1.24.7-00
Cloud being used: bare-metal
Installation method: terraform + ansible
Host OS: Ubuntu 20.04
CNI and version: weave-net v2.8.1
CRI and version: containerd v1.6.12

Hello,

To make it clearer I will explain with a case. We have a service that synchronizes user data with google workspace. I set up an image with some pre-configured scripts

/opt/app/scripts

and then I deploy it to the cluster using PersistentVolumeClaim to keep the data between pod updates.

It turns out that the deployment is overwriting the initial data in /opt/app/scripts

One approach I used was to copy the /opt/app/scripts folder to /tmp and then via post hook, dump the files to the pv created.

Is this the best approach? If not, what is the best approach for this?

2

There’s really not enough information here to give advice yet.

What kind of PV (NFS, iscsi, something else)?

Is it read-write-once?

How many replicas in your deployment?

Where are you mounting it?

What does “the deployment is overwriting” mean? Deployments don’t do that, it would have to be the code inside the container.

Posting YAML would help.

3

longhorn on local storage

ReadWriteMany

One replica

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: data-files
  name: data-files
  namespace: gcds
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: longhorn
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: gcds
  name: gcds-deploy
  namespace: gcds
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gcds
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: gcds
    spec:
      containers:
        - name: gcds
          image: IMG_WEB_NAME
          imagePullPolicy: IfNotPresent
          env:
            - name: GCDS_ROOT_PATH
              value: "/opt/gcds"
            - name: GCDS_CONFIG
              value: "/opt/gcds/data/config.xml"
            - name: GCDS_KEY
              value: "/opt/gcds/data/gcds.key"
            - name: DRY_RUN
              value: "TRUE"
            - name: CRON_EXP
              value: "*/5 * * * 1-5"
          ports:
            - containerPort: 80
          volumeMounts:
            - mountPath: "/opt/gcds/data"
              name: data-vol
            - mountPath: "/opt/gcds/scripts"
              name: scripts-vol
            - mountPath: "/opt/gcds/log"
              name: logs-vol
          lifecycle:
            postStart:
              exec:
                command:
                - /bin/bash
                - -c
                - |
                  cp -r /tmp/data/. /opt/gcds/data \
                  && cp -r /tmp/scripts/. /opt/gcds/scripts \
                  && chmod +x /opt/gcds/scripts/* \
                  && /opt/gcds/scripts/setup.sh
      hostname: gcds
      restartPolicy: Always
      imagePullSecrets:
        - name: gitlab-registry
      volumes:
        - name: data-vol
          persistentVolumeClaim:
            claimName: data-files
        - name: scripts-vol
          persistentVolumeClaim:
            claimName: scripts-files
        - name: logs-vol
          persistentVolumeClaim:
            claimName: logs-files
4

So it looks like the data-files volume is mounted at /opt/gcds/data.

It turns out that the deployment is overwriting the initial data in /opt/app/scripts

Can you help me parse that? Is it “one of the scripts is overwriting my data in /opt/gcds/data” or “something is overwriting my scripts in /opt/app/scripts” ? English is ambiguous. :slight_smile:

Looking at the postStart I see:

cp -r /tmp/data/. /opt/gcds/data \
  && cp -r /tmp/scripts/. /opt/gcds/scripts \
  && chmod +x /opt/gcds/scripts/* \
  && /opt/gcds/scripts/setup.sh

I can’t say what is in /tmp when it starts up, but this hook is nuking your volumes (well, merging static data into them, I guess) every time the container starts. That seems to be on purpose, but I don’t know why.

5

Correct.

The correct thought is: The deployment is overwriting “with nothing” the data in the volumes data-files, scripts-files and logs-files. :slight_smile:

This was the approach I found to get the files in the folders after the deployment.

Initially I didn’t have this postStart.

The files that I copied via Dockerfile to the image build, in my mind, were enough to be in the PVs afterwards.

FROM debian:bullseye-slim

ENV GCDS_ROOT_PATH "/opt/gcds"
ENV GCDS_CONFIG "config.xml"
ENV GCDS_KEY "gcds.key"
ENV DRY_RUN "TRUE"
ENV CRON_EXP "* * * * *"

WORKDIR /opt/gcds

RUN mkdir -p data scripts log

ADD config /tmp/data
ADD app app
ADD scripts /tmp/scripts
COPY requirements.txt .

RUN pip install -r requirements.txt

EXPOSE 5000

CMD ["gunicorn", "--bind", "0.0.0.0:5000", "--pythonpath", "/opt/gcds/app", "app:app"]
6

One of three things must be true:

  1. It’s not actually mounting your volumes and you are just seeing the underlying empty directories
  2. Something in your app is wiping those dirs
  3. Something in the sotrage driver is wiping those dirs

Kubernetes itself doesn’t do anything here.

Here’s what I would do to figure out whether #1 is true.

In your Dockerfile:

RUN mkdir -p data scripts log && touch data/this_is_the_image_data scripts/this_is_the_image_scripts logs/this_is_th_image_log

Make sure the volumes have some real data in them (even just a file named “this_is_the_volume_data”.

Then run your deployment and see what you see in those dirs.

If you see “this_is_the_volume_data”, then you know it is being mounted.
If you see “this_is_the_image_data”, then you know it is not being mounted.
If you see anything else, then you know either it is being wiped or something else is mounted

If it is the last case, check the volume - does it still contain “this_is_the_volume_data” or was it wiped?

Further experiments:

Replace the image and command with something that just logs the contents of those dirs and then sleep forever. This will rule out your own app.

Mount the volumes as readOnly: true and see if someone complains.

Mount the volumes to a different dir and see if the behavior is different.