Hi team,
I am facing an issue where I created a Kubernetes secret containing two password files. In my deployment, I have two containers. I mounted the above secret into one of the containers using defaultMode: 0400 and readOnly: true. I expected the secret files to be mounted with 0400 permissions (i.e., readable only by the owner). However, when the secret is mounted into the specified directory, the file permissions are actually 0440 (i.e., readable by both the owner and the group).
I tried multiple methods to resolve this:
- Using an init container to create an
emptyDirvolume and change the file permissions. - Using the
cpcommand to copy the files to a writable directory and modify the permissions.
However, I do not want to use these workarounds in production.
Below is the actual YAML file:
apiVersion: apps/v1
kind: Deployment
metadata:
namespace:
labels:
app: pulse-grpc
name: pulse-grpc
spec:
replicas: 1
selector:
matchLabels:
app: pulse-grpc
strategy:
rollingUpdate:
maxSurge: 2
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
app: pulse-grpc
spec:
securityContext:
fsGroup: 5000
initContainers:
- name: init-sysctl
image: busybox
# Updating the kernal properties
command:
- /bin/sh
- -c
- |
sysctl -w net.core.somaxconn=10000
sysctl -w net.ipv4.tcp_max_syn_backlog=4096
securityContext:
privileged: true
containers:
- name: prometheus-jmx-exporter
securityContext:
runAsUser: 5000
runAsGroup: 5000
image: <jmx-image>
ports:
- containerPort: 8081
name: prometheus-jmx
resources:
limits:
cpu: 100m
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
command:
- /usr/bin/java
- -Djavax.net.ssl.trustStore=/usr/lib/pulse/ssl/client.truststore.jks
- -Djavax.net.ssl.trustStorePassword=Ks56EEYFLFc3Ng19ZG
- -Djavax.net.ssl.keyStore=/usr/lib/pulse/ssl/client.keystore.jks
- -Djavax.net.ssl.keyStorePassword=Ks56EEYFLFc3Ng19ZG
- -jar
- /usr/lib/jmx_exporter/jmx_prometheus_exporter.jar
- 0.0.0.0:8081
- /usr/lib/jmx_exporter/pulse_prom_metrics.yaml
volumeMounts:
- name: pulse-config-secret
mountPath: /usr/lib/jmx_exporter/pulse_prom_metrics.yaml
subPath: pulse_prom_metrics.yaml
- name: jmx-client-ssl
mountPath: /usr/lib/pulse/ssl/
readOnly: true
- name: pulse-grpc
securityContext:
runAsUser: 5000
runAsGroup: 5000
image: <grpc-image>
imagePullPolicy: Always
command:
- /usr/bin/java
- -Dcom.sun.management.jmxremote
- -Dcom.sun.management.jmxremote.ssl=true
- -Djavax.net.ssl.keyStore=/usr/lib/pulse/ssl/server.keystore.jks
- -Djavax.net.ssl.keyStorePassword=Ks56EEYFLFc3Ng19ZG
- -Djavax.net.ssl.trustStore=/usr/lib/pulse/ssl/server.keystore.jks
- -Djavax.net.ssl.trustStorePassword=Ks56EEYFLFc3Ng19ZG
- -Dcom.sun.management.jmxremote.registry.ssl=true
- -Dcom.sun.management.jmxremote.port=9999
- -Dcom.sun.management.jmxremote.rmi.port=9999
- -Djava.rmi.server.hostname=127.0.0.1
- -Dcom.sun.management.jmxremote.authenticate=true
- -Dcom.sun.management.jmxremote.password.file=/usr/lib/pulse/jmx-auth/jmxremote.password
- -Dcom.sun.management.jmxremote.access.file=/usr/lib/pulse/jmx-auth/jmxremote.access
- -Dvertx.metrics.options.enabled=true
- -Dvertx.metrics.options.jmxEnabled=true
- -Dvertx.metrics.options.jmxDomain=vertx
- -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory
- -Dlog4j.configurationFile=/usr/lib/pulse/conf/log4j2.xml
- -XX:+UseG1GC
- -XX:MaxGCPauseMillis=20
- -XX:G1HeapRegionSize=1M
- -Xms2g
- -Xmx2g
- -XX:+AlwaysPreTouch
- -XX:+UnlockExperimentalVMOptions
- -XX:G1NewSizePercent=15
- -XX:G1ReservePercent=15
- -XX:+DisableExplicitGC
- -XX:+ParallelRefProcEnabled
- -XX:MaxRAMFraction=1
- -XX:+UseCGroupMemoryLimitForHeap
- -server
- -jar
- /usr/lib/pulse/pulse.jar
- -conf
- /usr/lib/pulse/conf/pulse.json
env:
- name: LOG4J_APPLEVEL
value: ERROR
- name: LOG4J_KAFKALEVEL
value: ERROR
ports:
- containerPort: 8080
name: grpc
protocol: TCP
- containerPort: 9999
name: jmx
resources:
limits:
cpu: "4"
memory: 6000Mi
requests:
cpu: "4"
memory: 6000Mi
readinessProbe:
exec:
command: [ "./grpc_health_probe", "-addr=:8080" ]
initialDelaySeconds: 10
timeoutSeconds: 1
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
livenessProbe:
exec:
command: [ "./grpc_health_probe", "-addr=:8080" ]
failureThreshold: 6
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
- name: pulse-config-secret
mountPath: /usr/lib/pulse/conf/log4j2.xml
subPath: log4j2.xml
- name: pulse-config-secret
mountPath: /usr/lib/pulse/conf/pulse.json
subPath: pulse.json
- name: pulse-kafka-certificate
mountPath: /usr/lib/pulse/client-ssl/
- name: jmx-auth-secret
mountPath: /usr/lib/pulse/jmx-auth/
readOnly: true
- name: grpc-server-ssl
mountPath: /usr/lib/pulse/ssl/
readOnly: true
terminationGracePeriodSeconds: 60
volumes:
- name: pulse-kafka-certificate
secret:
secretName: pulse-kafka-certificate
defaultMode: 0400
- name: pulse-config-secret
secret:
secretName: pulse-config-secret
defaultMode: 0400
- name: jmx-auth-secret
secret:
secretName: pulse-jmx-auth
defaultMode: 0400
- name: grpc-server-ssl
secret:
secretName: grpc-server-secret
defaultMode: 0400
- name: jmx-client-ssl
secret:
secretName: jmx-client-secret
defaultMode: 0400
inside the pod the file permission after mounting
bash-4.2$ pwd
/usr/lib/pulse/jmx-auth
bash-4.2$ ls -al
total 8
drwxrwsrwt 3 root pulse 120 May 17 01:51 .
drwxr-x— 1 pulse pulse 4096 May 17 01:51 ..
drwxr-sr-x 2 root pulse 80 May 17 01:51 ..2025_05_17_01_51_55.1328754605
lrwxrwxrwx 1 root pulse 32 May 17 01:51 ..data → ..2025_05_17_01_51_55.1328754605
lrwxrwxrwx 1 root pulse 23 May 17 01:51 jmxremote.access → ..data/jmxremote.access
lrwxrwxrwx 1 root pulse 25 May 17 01:51 jmxremote.password → ..data/jmxremote.password
bash-4.2$ cd ..data/
bash-4.2$ ls -al
total 8
drwxr-sr-x 2 root pulse 80 May 17 01:51 .
drwxrwsrwt 3 root pulse 120 May 17 01:51 ..
-r–r----- 1 root pulse 26 May 17 01:51 jmxremote.access
-r–r----- 1 root pulse 33 May 17 01:51 jmxremote.password
bash-4.2$
Logs observed in the pod:
[root@pre-prod-manager new]# kubectl logs pulse-grpc-ebay-ssl-849897d7d7-chsg2 -n radware-botmanager -c pulse-grpc
Error: Password file read access must be restricted: /usr/lib/pulse/jmx-auth/jmxremote.password
Inside the pod, I see a JMX error stating that the password file must have “owner read permission only”.
My questions:
- I specified
0400permissions while mounting the secret, but the permissions are not being reflected correctly inside the pod — it is actually showing as0440. - How can I solve this issue properly in production pods without using workarounds like init containers or
cp?
Please, can someone help me resolve this issue?
Thanks.