How to enable binlog for 1 pod in galera mariadb statefulset


#1

Hi all,
I followed this article to create galera mariadb cluster with kubernetes , 3 pods with 3 PersistentVolume .
mariadb-pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-galera-0
  labels:
    app: galera-ss
    podindex: "0"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  hostPath:
    path: /data/pods/galera-0/datadir

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-galera-1
  labels:
    app: galera-ss
    podindex: "1"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  hostPath:
    path: /data/pods/galera-1/datadir

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-galera-2
  labels:
    app: galera-ss
    podindex: "2"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  hostPath:
    path: /data/pods/galera-2/datadir

mariadb-pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-datadir-galera-ss-0
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "0"

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-datadir-galera-ss-1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "1"

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-datadir-galera-ss-2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "2"

mariadb-ss.yml

apiVersion: v1
kind: Service
metadata:
  name: galera-ss
  labels:
    app: galera-ss
spec:
  ports:
  - port: 3306
    name: mysql
  clusterIP: None
  selector:
    app: galera-ss

---

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: galera-ss
spec:
  serviceName: "galera-ss"
  replicas: 3
  template:
    metadata:
      labels:
        app: galera-ss
    spec:
      containers:
      - name: galera
        image: severalnines/mariadb:10.1
        ports:
        - name: mysql
          containerPort: 3306
        env:
        # kubectl create secret generic mysql-pass --from-file=password.txt
        - name: MYSQL_ROOT_PASSWORD
          value: myrootpassword
        - name: DISCOVERY_SERVICE
          value: 10.240.0.10:2379,10.240.0.11:2379,10.240.0.12:2379
        - name: XTRABACKUP_PASSWORD
          value: password
        - name: CLUSTER_NAME
          value: mariadb_galera_ss
        - name: MYSQL_DATABASE
          value: mydatabase
        - name: MYSQL_USER
          value: myuser
        - name: MYSQL_PASSWORD
          value: myuserpassword
        readinessProbe:
          exec:
            command:
            - /healthcheck.sh
            - --readiness
          initialDelaySeconds: 120
          periodSeconds: 1
        livenessProbe:
          exec:
            command:
            - /healthcheck.sh
            - --liveness
          initialDelaySeconds: 120
          periodSeconds: 1
        volumeMounts:
        - name: mysql-datadir
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: mysql-datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      # uncomment if using slow storageClass on AWS
      # then no need for running pv or pvc manifests
      storageClassName: slow
      resources:
        requests:
          storage: 50Gi

Now I want to enable binlog for pod galera-ss-0 only , my intention is create a persistentvolume and persistentvolumeclaim and mount this volume into pod galera-ss-0 for binlog storage and create a config map that will map mysql config enable binlog

[mysqld]
log_bin=/opt/binlog/mariadb-binlog
expire_logs_days = 30
max_binlog_size = 32M

binlog-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: binlog-galera-0
  labels:
    app: galera-ss
    podindex: "0"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /data/pods/galera-0/binlog

binlog-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-binlog-galera-ss-0
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "0"

kubectl get statefulsets,po,pv,pvc,deploy -o wide

NAME                         DESIRED   CURRENT   AGE       CONTAINERS   IMAGES
statefulset.apps/galera-ss   3         3         1d        galera       severalnines/mariadb:10.1

NAME              READY     STATUS    RESTARTS   AGE       IP             NODE
pod/galera-ss-0   1/1       Running   1          5m        10.200.0.4     worker-1
pod/galera-ss-1   1/1       Running   2          1d        10.200.128.5   worker-2
pod/galera-ss-2   1/1       Running   1          1d        10.200.96.5    worker-0

NAME                                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                               STORAGECLASS   REASON    AGE
persistentvolume/binlog-galera-0    10Gi       RWO            Retain           Bound     default/mysql-binlog-galera-ss-0                             2h
persistentvolume/datadir-galera-0   50Gi       RWO            Retain           Bound     default/mysql-datadir-galera-ss-0                            1d
persistentvolume/datadir-galera-1   50Gi       RWO            Retain           Bound     default/mysql-datadir-galera-ss-1                            1d
persistentvolume/datadir-galera-2   50Gi       RWO            Retain           Bound     default/mysql-datadir-galera-ss-2                            1d

NAME                                              STATUS    VOLUME             CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-binlog-galera-ss-0    Bound     binlog-galera-0    10Gi       RWO                           2h
persistentvolumeclaim/mysql-datadir-galera-ss-0   Bound     datadir-galera-0   50Gi       RWO                           1d
persistentvolumeclaim/mysql-datadir-galera-ss-1   Bound     datadir-galera-1   50Gi       RWO                           1d
persistentvolumeclaim/mysql-datadir-galera-ss-2   Bound     datadir-galera-2   50Gi       RWO                           1d

How can I mount binlog-galera-0 pv into running pod galera-ss-0 ?
kubectl describe pod/galera-ss-0

Name:               galera-ss-0
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               worker-1/10.240.0.21
Start Time:         Tue, 23 Oct 2018 13:44:49 +0700
Labels:             app=galera-ss
                    controller-revision-hash=galera-ss-644bbdfcf7
                    statefulset.kubernetes.io/pod-name=galera-ss-0
Annotations:        <none>
Status:             Running
IP:                 10.200.0.4
Controlled By:      StatefulSet/galera-ss
Containers:
  galera:
    Container ID:   containerd://2a810cb8cec9c0539ea6e29c38b4165880d25fc67dd61a38be1623a9a5ae705e
    Image:          severalnines/mariadb:10.1
    Image ID:       docker.io/severalnines/mariadb@sha256:18643a5787e34d2ffc6d036510cfd58ae4c159a386150f03d5cb6cf0cecafd55
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 23 Oct 2018 13:47:22 +0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 23 Oct 2018 13:44:49 +0700
      Finished:     Tue, 23 Oct 2018 13:47:21 +0700
    Ready:          True
    Restart Count:  1
    Liveness:       exec [/healthcheck.sh --liveness] delay=120s timeout=1s period=1s #success=1 #failure=3
    Readiness:      exec [/healthcheck.sh --readiness] delay=120s timeout=1s period=1s #success=1 #failure=3
    Environment:
      MYSQL_ROOT_PASSWORD:  myrootpassword
      DISCOVERY_SERVICE:    10.240.0.10:2379,10.240.0.11:2379,10.240.0.12:2379
      XTRABACKUP_PASSWORD:  password
      CLUSTER_NAME:         mariadb_galera_ss
      MYSQL_DATABASE:       mydatabase
      MYSQL_USER:           myuser
      MYSQL_PASSWORD:       myuserpassword
    Mounts:
      /var/lib/mysql from mysql-datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jsvmt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  mysql-datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-datadir-galera-ss-0
    ReadOnly:   false
  default-token-jsvmt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jsvmt
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  47m                 default-scheduler  Successfully assigned default/galera-ss-0 to worker-1
  Normal   Pulled     47m                 kubelet, worker-1  Container image "severalnines/mariadb:10.1" already present on machine
  Normal   Created    47m                 kubelet, worker-1  Created container
  Normal   Started    47m                 kubelet, worker-1  Started container
  Warning  Unhealthy  45m (x3 over 45m)   kubelet, worker-1  Liveness probe failed: /healthcheck.sh: line 80: [: !=: unary operator expected
  Warning  Unhealthy  45m (x19 over 45m)  kubelet, worker-1  Readiness probe failed: /healthcheck.sh: line 66: [: -eq: unary operator expected

I tried “kubectl edit pod/galera-ss-0” and add more in “volumeMounts:” and “volumes:” but I get error " # pods “galera-ss-0” was not valid: # * spec: Forbidden: pod updates may not change fields other than spec.containers[*].image …"


#2

My solution:

  • Create a configmap like this:
    kind: ConfigMap
    apiVersion: v1
    metadata:
    name: mysql-config
    data:
    confluence.cnf: |
    [mysqld]
    log_bin=/opt/binlog/mariadb-binlog
    expire_logs_days = 30
    max_binlog_size = 32M

Create a Deployment (1 replica) name galera-ss-2-something like this
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: galera-ss-2
spec:
replicas: 1
template:
metadata:
labels:
app: galera-ss
spec:
terminationGracePeriodSeconds: 10
hostname: galera-ss-2
nodeName: worker-2
containers:

  • name: galera
    image: severalnines/mariadb:10.1
    ports:
    • containerPort: 3306
      name: mysql
      env:

    kubectl create secret generic mysql-pass --from-file=password.txt

    • name: MYSQL_ROOT_PASSWORD
      value: myrootpassword
    • name: DISCOVERY_SERVICE
      value: 10.240.0.10:2379,10.240.0.11:2379,10.240.0.12:2379
    • name: XTRABACKUP_PASSWORD
      value: password
    • name: CLUSTER_NAME
      value: mariadb_galera_ss
    • name: MYSQL_DATABASE
      value: mydatabase
    • name: MYSQL_USER
      value: myuser
    • name: MYSQL_PASSWORD
      value: myuserpassword
      readinessProbe:
      exec:
      command:
      • /healthcheck.sh
      • –readiness
        initialDelaySeconds: 120
        periodSeconds: 1
        livenessProbe:
        exec:
        command:
      • /healthcheck.sh
      • –liveness
        initialDelaySeconds: 120
        periodSeconds: 1
        volumeMounts:
    • name: mysql-datadir
      mountPath: /var/lib/mysql
    • name: binlog
      mountPath: /opt/binlog
    • name: mysql-config
      mountPath: /etc/my.cnf.d
      volumes:
  • name: mysql-config
    configMap:
    name: mysql-config
  • name: binlog
    persistentVolumeClaim:
    claimName: binlog
  • name: mysql-datadir
    persistentVolumeClaim:
    claimName: mysql-datadir-galera-ss-2

Change number of replicas from 3 to 2 in mariadb-ss.yml , so after all I will have 3 pod : galera-ss-0 , galera-ss-1 , galera-ss-2-something (enable binlog)