Running tutorial mysql example gives me "read-only file system" error

General Discussions
1

Pretty surprising that running a mysql example tutorial gave me:

Error: failed to start container "mysql": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system

From the main kubernetes webpage:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/

More info

dan@dan-ThinkPad-X1-Carbon:~/cromwell/ops/kubernetes$ kubectl get pods -l app=mysql
NAME                     READY   STATUS              RESTARTS   AGE
mysql-648d67bf7c-llkjk   0/1     RunContainerError   2          38s
dan@dan-ThinkPad-X1-Carbon:~/cromwell/ops/kubernetes$ kubectl describe pod mysql
Name:           mysql-648d67bf7c-llkjk
Namespace:      default
Node:           gke-db-test-default-pool-ad7d8d69-5rhb/10.240.0.30
Start Time:     Sun, 21 Oct 2018 18:31:01 -0400
Labels:         app=mysql
                pod-template-hash=2048236937
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container mysql
Status:         Running
IP:             10.8.0.9
Controlled By:  ReplicaSet/mysql-648d67bf7c
Containers:
  mysql:
    Container ID:   docker://887c0975b206df73d44c03f00f0d89043cd1e1d066af235dd19dde0bbe7e746c
    Image:          mysql:5.6
    Image ID:       docker-pullable://mysql@sha256:0ae5d57a7b349384bb46bc7f6b73c21d2806b02bbb99554684e78b700094ab62
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       RunContainerError
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
      Exit Code:    128
      Started:      Sun, 21 Oct 2018 18:31:47 -0400
      Finished:     Sun, 21 Oct 2018 18:31:47 -0400
    Ready:          False
    Restart Count:  3
    Requests:
      cpu:  100m
    Environment:
      MYSQL_ROOT_PASSWORD:  password
    Mounts:
      /var/lib/mysql from mysql-persistent-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-8828d (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  mysql-persistent-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-pv-claim
    ReadOnly:   false
  default-token-8828d:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-8828d
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                From                                             Message
  ----     ------                 ----               ----                                             -------
  Normal   Scheduled              61s                default-scheduler                                Successfully assigned mysql-648d67bf7c-llkjk to gke-db-test-default-pool-ad7d8d69-5rhb
  Normal   SuccessfulMountVolume  61s                kubelet, gke-db-test-default-pool-ad7d8d69-5rhb  MountVolume.SetUp succeeded for volume "mysql-pv-volume"
  Normal   SuccessfulMountVolume  61s                kubelet, gke-db-test-default-pool-ad7d8d69-5rhb  MountVolume.SetUp succeeded for volume "default-token-8828d"
  Normal   Pulled                 15s (x4 over 60s)  kubelet, gke-db-test-default-pool-ad7d8d69-5rhb  Container image "mysql:5.6" already present on machine
  Normal   Created                15s (x4 over 60s)  kubelet, gke-db-test-default-pool-ad7d8d69-5rhb  Created container
  Warning  Failed                 15s (x4 over 60s)  kubelet, gke-db-test-default-pool-ad7d8d69-5rhb  Error: failed to start container "mysql": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
  Warning  BackOff                4s (x3 over 57s)   kubelet, gke-db-test-default-pool-ad7d8d69-5rhb  Back-off restarting failed container

dan@dan-ThinkPad-X1-Carbon:~/cromwell/ops/kubernetes$ kubectl describe pv 
Name:            mysql-pv-volume
Labels:          type=local
Annotations:     pv.kubernetes.io/bound-by-controller: yes
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    manual
Status:          Bound
Claim:           default/mysql-pv-claim
Reclaim Policy:  Retain
Access Modes:    RWO
Capacity:        20Gi
Node Affinity:   <none>
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /mnt/data
    HostPathType:  
Events:            <none>
dan@dan-ThinkPad-X1-Carbon:~/cromwell/ops/kubernetes$ kubectl describe pvc
Name:          mysql-pv-claim
Namespace:     default
StorageClass:  manual
Status:        Bound
Volume:        mysql-pv-volume
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      20Gi
Access Modes:  RWO
Events:        <none>
Mounted By:    mysql-648d67bf7c-llkjk
1 Like
2

It turns out that when running on GKE there is no need to explicitly create a volume. Editing the volume claim to remove the storageclassName and deleting the persistent volume section did the trick.

Note well, that using hostPath is “for testing purposes only” on a single node cluster per the docs.

Perhaps this tutorial should include an intro on what kind of cluster is necessary in the “before you begin.”

3

@Dan_Billings - It might be worth passing that on as an issue or PR’ing the caveat into the kubernetes website.

If you do post over there, can reference this thread within the comments :slight_smile:

5

Hi Dan,
I can not make it work with your instructions. Can you be more explicit, please? Like pasting the YAML files to see the modifications that work.
Thanks a lot.

6

With this mysql-pv YAML file, it works:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: manual
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
replication-type: none

7

Below is @Tonterias yaml with the correct indentation to make it easier to copy/paste:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---      
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: manual
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard
  replication-type: none