How to enable binlog for 1 pod in galera mariadb statefulset

General Discussions
1

Hi all,
I followed this article to create galera mariadb cluster with kubernetes , 3 pods with 3 PersistentVolume .
mariadb-pv.yml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-galera-0
  labels:
    app: galera-ss
    podindex: "0"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  hostPath:
    path: /data/pods/galera-0/datadir

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-galera-1
  labels:
    app: galera-ss
    podindex: "1"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  hostPath:
    path: /data/pods/galera-1/datadir

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: datadir-galera-2
  labels:
    app: galera-ss
    podindex: "2"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 50Gi
  hostPath:
    path: /data/pods/galera-2/datadir

mariadb-pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-datadir-galera-ss-0
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "0"

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-datadir-galera-ss-1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "1"

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-datadir-galera-ss-2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "2"

mariadb-ss.yml

apiVersion: v1
kind: Service
metadata:
  name: galera-ss
  labels:
    app: galera-ss
spec:
  ports:
  - port: 3306
    name: mysql
  clusterIP: None
  selector:
    app: galera-ss

---

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: galera-ss
spec:
  serviceName: "galera-ss"
  replicas: 3
  template:
    metadata:
      labels:
        app: galera-ss
    spec:
      containers:
      - name: galera
        image: severalnines/mariadb:10.1
        ports:
        - name: mysql
          containerPort: 3306
        env:
        # kubectl create secret generic mysql-pass --from-file=password.txt
        - name: MYSQL_ROOT_PASSWORD
          value: myrootpassword
        - name: DISCOVERY_SERVICE
          value: 10.240.0.10:2379,10.240.0.11:2379,10.240.0.12:2379
        - name: XTRABACKUP_PASSWORD
          value: password
        - name: CLUSTER_NAME
          value: mariadb_galera_ss
        - name: MYSQL_DATABASE
          value: mydatabase
        - name: MYSQL_USER
          value: myuser
        - name: MYSQL_PASSWORD
          value: myuserpassword
        readinessProbe:
          exec:
            command:
            - /healthcheck.sh
            - --readiness
          initialDelaySeconds: 120
          periodSeconds: 1
        livenessProbe:
          exec:
            command:
            - /healthcheck.sh
            - --liveness
          initialDelaySeconds: 120
          periodSeconds: 1
        volumeMounts:
        - name: mysql-datadir
          mountPath: /var/lib/mysql
  volumeClaimTemplates:
  - metadata:
      name: mysql-datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      # uncomment if using slow storageClass on AWS
      # then no need for running pv or pvc manifests
      storageClassName: slow
      resources:
        requests:
          storage: 50Gi

Now I want to enable binlog for pod galera-ss-0 only , my intention is create a persistentvolume and persistentvolumeclaim and mount this volume into pod galera-ss-0 for binlog storage and create a config map that will map mysql config enable binlog

[mysqld]
log_bin=/opt/binlog/mariadb-binlog
expire_logs_days = 30
max_binlog_size = 32M

binlog-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: binlog-galera-0
  labels:
    app: galera-ss
    podindex: "0"
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /data/pods/galera-0/binlog

binlog-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-binlog-galera-ss-0
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  selector:
    matchLabels:
      app: galera-ss
      podindex: "0"

kubectl get statefulsets,po,pv,pvc,deploy -o wide

NAME                         DESIRED   CURRENT   AGE       CONTAINERS   IMAGES
statefulset.apps/galera-ss   3         3         1d        galera       severalnines/mariadb:10.1

NAME              READY     STATUS    RESTARTS   AGE       IP             NODE
pod/galera-ss-0   1/1       Running   1          5m        10.200.0.4     worker-1
pod/galera-ss-1   1/1       Running   2          1d        10.200.128.5   worker-2
pod/galera-ss-2   1/1       Running   1          1d        10.200.96.5    worker-0

NAME                                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                               STORAGECLASS   REASON    AGE
persistentvolume/binlog-galera-0    10Gi       RWO            Retain           Bound     default/mysql-binlog-galera-ss-0                             2h
persistentvolume/datadir-galera-0   50Gi       RWO            Retain           Bound     default/mysql-datadir-galera-ss-0                            1d
persistentvolume/datadir-galera-1   50Gi       RWO            Retain           Bound     default/mysql-datadir-galera-ss-1                            1d
persistentvolume/datadir-galera-2   50Gi       RWO            Retain           Bound     default/mysql-datadir-galera-ss-2                            1d

NAME                                              STATUS    VOLUME             CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-binlog-galera-ss-0    Bound     binlog-galera-0    10Gi       RWO                           2h
persistentvolumeclaim/mysql-datadir-galera-ss-0   Bound     datadir-galera-0   50Gi       RWO                           1d
persistentvolumeclaim/mysql-datadir-galera-ss-1   Bound     datadir-galera-1   50Gi       RWO                           1d
persistentvolumeclaim/mysql-datadir-galera-ss-2   Bound     datadir-galera-2   50Gi       RWO                           1d

How can I mount binlog-galera-0 pv into running pod galera-ss-0 ?
kubectl describe pod/galera-ss-0

Name:               galera-ss-0
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               worker-1/10.240.0.21
Start Time:         Tue, 23 Oct 2018 13:44:49 +0700
Labels:             app=galera-ss
                    controller-revision-hash=galera-ss-644bbdfcf7
                    statefulset.kubernetes.io/pod-name=galera-ss-0
Annotations:        <none>
Status:             Running
IP:                 10.200.0.4
Controlled By:      StatefulSet/galera-ss
Containers:
  galera:
    Container ID:   containerd://2a810cb8cec9c0539ea6e29c38b4165880d25fc67dd61a38be1623a9a5ae705e
    Image:          severalnines/mariadb:10.1
    Image ID:       docker.io/severalnines/mariadb@sha256:18643a5787e34d2ffc6d036510cfd58ae4c159a386150f03d5cb6cf0cecafd55
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 23 Oct 2018 13:47:22 +0700
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 23 Oct 2018 13:44:49 +0700
      Finished:     Tue, 23 Oct 2018 13:47:21 +0700
    Ready:          True
    Restart Count:  1
    Liveness:       exec [/healthcheck.sh --liveness] delay=120s timeout=1s period=1s #success=1 #failure=3
    Readiness:      exec [/healthcheck.sh --readiness] delay=120s timeout=1s period=1s #success=1 #failure=3
    Environment:
      MYSQL_ROOT_PASSWORD:  myrootpassword
      DISCOVERY_SERVICE:    10.240.0.10:2379,10.240.0.11:2379,10.240.0.12:2379
      XTRABACKUP_PASSWORD:  password
      CLUSTER_NAME:         mariadb_galera_ss
      MYSQL_DATABASE:       mydatabase
      MYSQL_USER:           myuser
      MYSQL_PASSWORD:       myuserpassword
    Mounts:
      /var/lib/mysql from mysql-datadir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jsvmt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  mysql-datadir:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-datadir-galera-ss-0
    ReadOnly:   false
  default-token-jsvmt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jsvmt
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  47m                 default-scheduler  Successfully assigned default/galera-ss-0 to worker-1
  Normal   Pulled     47m                 kubelet, worker-1  Container image "severalnines/mariadb:10.1" already present on machine
  Normal   Created    47m                 kubelet, worker-1  Created container
  Normal   Started    47m                 kubelet, worker-1  Started container
  Warning  Unhealthy  45m (x3 over 45m)   kubelet, worker-1  Liveness probe failed: /healthcheck.sh: line 80: [: !=: unary operator expected
  Warning  Unhealthy  45m (x19 over 45m)  kubelet, worker-1  Readiness probe failed: /healthcheck.sh: line 66: [: -eq: unary operator expected

I tried “kubectl edit pod/galera-ss-0” and add more in “volumeMounts:” and “volumes:” but I get error " # pods “galera-ss-0” was not valid: # * spec: Forbidden: pod updates may not change fields other than spec.containers[*].image …"

2

My solution:

  • Create a configmap like this:
    kind: ConfigMap
    apiVersion: v1
    metadata:
    name: mysql-config
    data:
    confluence.cnf: |
    [mysqld]
    log_bin=/opt/binlog/mariadb-binlog
    expire_logs_days = 30
    max_binlog_size = 32M

Create a Deployment (1 replica) name galera-ss-2-something like this
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: galera-ss-2
spec:
replicas: 1
template:
metadata:
labels:
app: galera-ss
spec:
terminationGracePeriodSeconds: 10
hostname: galera-ss-2
nodeName: worker-2
containers:

  • name: galera
    image: severalnines/mariadb:10.1
    ports:
    • containerPort: 3306
      name: mysql
      env:

    kubectl create secret generic mysql-pass --from-file=password.txt

    • name: MYSQL_ROOT_PASSWORD
      value: myrootpassword
    • name: DISCOVERY_SERVICE
      value: 10.240.0.10:2379,10.240.0.11:2379,10.240.0.12:2379
    • name: XTRABACKUP_PASSWORD
      value: password
    • name: CLUSTER_NAME
      value: mariadb_galera_ss
    • name: MYSQL_DATABASE
      value: mydatabase
    • name: MYSQL_USER
      value: myuser
    • name: MYSQL_PASSWORD
      value: myuserpassword
      readinessProbe:
      exec:
      command:
      • /healthcheck.sh
      • –readiness
        initialDelaySeconds: 120
        periodSeconds: 1
        livenessProbe:
        exec:
        command:
      • /healthcheck.sh
      • –liveness
        initialDelaySeconds: 120
        periodSeconds: 1
        volumeMounts:
    • name: mysql-datadir
      mountPath: /var/lib/mysql
    • name: binlog
      mountPath: /opt/binlog
    • name: mysql-config
      mountPath: /etc/my.cnf.d
      volumes:
  • name: mysql-config
    configMap:
    name: mysql-config
  • name: binlog
    persistentVolumeClaim:
    claimName: binlog
  • name: mysql-datadir
    persistentVolumeClaim:
    claimName: mysql-datadir-galera-ss-2

Change number of replicas from 3 to 2 in mariadb-ss.yml , so after all I will have 3 pod : galera-ss-0 , galera-ss-1 , galera-ss-2-something (enable binlog)

3

Hey. I am also following the same article to deploy galera cluster on kubernetes. But i am unable to connect to the stateful set cluster using the port number. Which command did you use to connect to the mysql server on the cluster? Also the command to connect to the cluster is “mysql -udb_user -ppassword -hgalera-ss -P3306” so does that mean that mysql should be preinstalled in our machine. Because the whole point was to run mysql on the cluster