Statefulset pod is getting killed time to time

General Discussions
1

Kubernetes version: 1.9.7-gke.11
Cloud being used: Google Cloud
Statefulset deployment method: Using https://github.com/cvallance/mongo-k8s-sidecar and https://hub.docker.com/r/cvallance/mongo-k8s-sidecar/

I have added the YAML file of the deployment to the end of the document.
I did a MongoDb stateful set deployment about a year ago and I notice that the Mongo DB container is getting killed and respawned periodically (around once a week).

First I thought this is because of the memory consumption of the container. So I increased the memory to 11Gi (Running on a 16Gi node), but it only utilizes around 4Gi and crashes.
See the memory utilization graph here.

image
image.png755×282 17.4 KB

According to the mongo container log, the Mongo server received the Kill signal (signal 15). As far as I understand the mongo container got the kill signal from an external source.

I looked at the kubernetes audit logs as well. Nothing interesting found there (Could not upload the log file due to restriction).

Kubernetes is killing the pod but I cannot see a reason anywhere. Is there any other place that I can look for logs or get any other information? Any suggestion would be appreciated.

Yaml:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
      mongo; cpu request for container mongo-sidecar'
  creationTimestamp: 2019-06-30T03:27:39Z
  generateName: mongo-
  labels:
    controller-revision-hash: mongo-567f44d6d8
    environment: test
    role: mongo
    statefulset.kubernetes.io/pod-name: mongo-0
  name: mongo-0
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: mongo
    uid: 2dd5d279-6b9d-11e7-99db-42010af000ac
  resourceVersion: "150388949"
  selfLink: /api/v1/namespaces/default/pods/mongo-0
  uid: 03468c71-9ae7-11e9-b2e0-42010af00190
spec:
  containers:
  - command:
    - mongod
    - --replSet
    - rs0
    - --bind_ip
    - 0.0.0.0
    - --smallfiles
    - --noprealloc
    image: mongo:3.4.20
    imagePullPolicy: Always
    name: mongo
    ports:
    - containerPort: 27017
      protocol: TCP
    resources:
      limits:
        memory: 11Gi
      requests:
        cpu: 100m
        memory: 11Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /data/db
      name: mongo-persistent-storage
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-jlldw
      readOnly: true
  - env:
    - name: MONGO_SIDECAR_POD_LABELS
      value: role=mongo,environment=test
    image: cvallance/mongo-k8s-sidecar
    imagePullPolicy: Always
    name: mongo-sidecar
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-jlldw
      readOnly: true
  dnsPolicy: ClusterFirst
  hostname: mongo-0
  nodeName: gke-cluste-high-performance-91c7309f-29t8
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: mongo
  terminationGracePeriodSeconds: 10
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: mongo-persistent-storage
    persistentVolumeClaim:
      claimName: mongo-persistent-storage-mongo-0
  - name: default-token-jlldw
    secret:
      defaultMode: 420
      secretName: default-token-jlldw
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2019-06-30T03:29:15Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2019-06-30T03:29:53Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: null
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: 2019-06-30T03:29:15Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://30f8dfeba7301f55e37e7202e9627a05e0fdb73fd7dbb47d1e4ca202d7d99c67
    image: mongo:3.4.20
    imageID: docker-pullable://mongo@sha256:3e825c55a4e6c6c4ea74f6e53a372519da0d7a7c98825a441c0f199f8ab5110a
    lastState: {}
    name: mongo
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2019-06-30T03:29:47Z
  - containerID: docker://c0d413fd84c3235d96f458732fcbf47865de4c0c0c20fb186bfe826abef9b006
    image: cvallance/mongo-k8s-sidecar:latest
    imageID: docker-pullable://cvallance/mongo-k8s-sidecar@sha256:cd62d32db488fbf78dfbaef020edd7fc09ee4d3fe5d50cc0579e747e8232c77f
    lastState: {}
    name: mongo-sidecar
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2019-06-30T03:29:53Z
  hostIP: 10.128.15.193
  phase: Running
  podIP: 10.60.4.3
  qosClass: Burstable
  startTime: 2019-06-30T03:29:15Z