PFA
Looking for an assist with my NFS volumes
Cluster information:
Kubernetes version: 1.20
Cloud being used: bare-metal
Installation method: Cluster installed to 4x 8GB raspberry pi4s, booting directly from 20GB SATA ssds. NFS hosted by another raspberry pi running ubuntu also running ubuntu 20.10
Host OS: Ubunutu 20.10
CNI and version: Calico
CRI and version: containerd
I’ve been running my blog from a small bare metal microk8s cluster using wordpress and mysql to learn k8s at home. I had this all working find until recently. The problem I am running into is that am unable to write media out to certain folders on my nfs mount now. This was working just fine before.
Changes:
I updated from ubuntu 20.04LTS to Ubuntu 20.10
I updated from k8s 1.19 to 1.20
I can share my deployment, but I did not change anything in the yamls between upgrades. Reads to appear to work and the site is up. Even more interesting, I am able to create new blocg posts, but not able to upload media.
The volume
apiVersion: v1
kind: PersistentVolume
metadata:
name: wordpress-persistent-storage
labels:
app: wordpress
tier: frontend
spec:
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
nfs:
server: 192.168.1.99
# Exported path of your NFS server
path: “/media/usb/shared/html”
The claims
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wordpress-persistent-storage
labels:
app: wordpress
tier: frontend
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 6Gi
—
The deployment
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: wordpress
labels:
app: wordpress
spec:
selector:
matchLabels:
app: wordpress
tier: frontend
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
tier: frontend
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: NotIn
values:
- master
containers:
# - image: wordpress:4.8-apache
- image: wordpress:php7.3-apache
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: wordpress-mysql
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass # generated before in secret.yml
key: password
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: “/var/www/html” # which data will be stored
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wordpress-persistent-storage
some additional info
checking the fs on the pod
$ k exec -it wordpress-7d5cdcc77b-lvlnk – ls -ltra /var/www/html
total 224
-rw-r–r-- 1 27 sudo 4755 Feb 6 2020 wp-trackback.php
-rw-r–r-- 1 27 sudo 3300 Feb 6 2020 wp-load.php
-rw-r–r-- 1 27 sudo 2496 Feb 6 2020 wp-links-opml.php
-rw-r–r-- 1 27 sudo 3940 Feb 6 2020 wp-cron.php
-rw-r–r-- 1 27 sudo 351 Feb 6 2020 wp-blog-header.php
-rw-r–r-- 1 27 sudo 405 Feb 6 2020 index.php
-rw-r–r-- 1 27 sudo 19915 Feb 12 2020 license.txt
-rw-r–r-- 1 27 sudo 8509 Apr 14 2020 wp-mail.php
-rw-r–r-- 1 27 sudo 3236 Jun 8 2020 xmlrpc.php
-rw-r–r-- 1 27 sudo 7278 Jun 26 2020 readme.html
-rw-r–r-- 1 27 sudo 20181 Jul 6 2020 wp-settings.php
-rw-r–r-- 1 27 sudo 48761 Jul 7 2020 wp-login.php
-rw-r–r-- 1 27 sudo 2332 Jul 23 2020 wp-comments-post.php
-rw-r–r-- 1 27 sudo 31159 Jul 23 2020 wp-signup.php
-rw-r–r-- 1 27 sudo 7101 Jul 28 2020 wp-activate.php
drwxr-xr-x 24 27 sudo 12288 Oct 30 20:41 wp-includes
drwxr-xr-x 9 27 sudo 4096 Oct 30 20:41 wp-admin
-rw-r–r-- 1 27 sudo 461 Dec 12 23:15 .htaccess
drwxr-xr-x 7 27 sudo 4096 Dec 18 02:42 wp-content
drwxr-xr-x 1 root root 4096 Jan 12 09:57 …
-rw-r–r-- 1 27 sudo 2823 Jan 31 16:44 wp-config-sample.php
-rw-r–r-- 1 27 sudo 3198 Jan 31 16:44 wp-config.php
drwxrwxr-x 5 27 sudo 4096 Jan 31 16:44 .
the exported fs on my nfs server(I have a 512 GB nvme ssd mounted here and shared out properly, tested and confirmed)
ubuntu@icetower:~$ ll /media/usb/shared/
total 16
drwxrwxr-x 4 ubuntu ubuntu 4096 Nov 30 03:30 ./
drwxrwxr-x 4 ubuntu ubuntu 4096 Jan 30 00:15 …/
drwxrwxr-x 5 27 sudo 4096 Jan 31 16:44 html/
drwxrwxr-x 10 27 sudo 4096 Jan 30 00:30 mysql/
now I may be mis-remembering but I recall this filesystem being owned by ubuntu:ubuntu.
more info from my nfs server
ubuntu@icetower:~$ cat /etc/group | grep 27
sudo:x:27:ubuntu
and from my pod
$ k exec -it wordpress-7d5cdcc77b-lvlnk – id
uid=0(root) gid=0(root) groups=0(root)
the process is actually running on the pod as www-data
$ k exec -it wordpress-7d5cdcc77b-lvlnk – ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 16:44 ? 00:00:00 apache2 -DFOREGROUND
www-data 82 1 0 16:44 ? 00:00:02 apache2 -DFOREGROUND
www-data 83 1 0 16:44 ? 00:00:00 apache2 -DFOREGROUND
www-data 84 1 0 16:44 ? 00:00:00 apache2 -DFOREGROUND
www-data 85 1 0 16:44 ? 00:00:00 apache2 -DFOREGROUND
www-data 86 1 0 16:44 ? 00:00:00 apache2 -DFOREGROUND
www-data 87 1 0 16:45 ? 00:00:00 apache2 -DFOREGROUND
root 103 0 0 17:11 pts/0 00:00:00 ps -ef
www-data is actuall uid 33
$ k exec -it wordpress-7d5cdcc77b-lvlnk – cat /etc/passwd | grep www-data
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
not in sudoers
$ k exec -it wordpress-7d5cdcc77b-lvlnk – cat /etc/group | grep 27
sudo:x:27:
Now when I try to upload media, I get the error
The uploaded file could not be moved to wp-content/uploads/2021/01.
I dont think this was necessary, but I even created the directories
ubuntu@icetower:~$ ll /media/usb/shared/html/wp-content/uploads/2021/01
total 8
drwxr-xr-x 2 27 sudo 4096 Jan 30 05:00 ./
drwxr-xr-x 3 27 sudo 4096 Jan 30 05:00 …/
kind of stumped on my next move here.
Should I set the perms to ubuntu:ubuntu and then
spec:
securityContext:
fsGroup: 1000
??