Alpine Git Clone only downloads index

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 19.4.
Cloud being used: Minikube Hyperkit
Installation method: Homebrew
Host OS: macOS Mojave

We are using initContainer:

initContainers:
  - name: git-cloner
    image: alpine/git
    imagePullPolicy: IfNotPresent
    args:
        - clone
        - --single-branch
        - --
        - https://github.com/smart48/smt-demo
        # - /code
    volumeMounts:
    - mountPath: /code
      name: code-storage

But some how it only added the index to directory and only once yesterday:

minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ cd /tmp/hostpath-provisioner/smt-local/code-pv-claim/
$ ls -la
total 12
drwxrwxrwx 2 root root 4096 Dec  7 06:57 .
drwxr-xr-x 5 root root 4096 Dec  7 05:02 ..
-rw-r--r-- 1 root root 1823 Dec  7 07:07 index.php

I need to have the whole repository downloaded minus .git dirs. And this is what I see when I check the container logs for the pod

kubectl logs -f app-74b5bdd58d-89hsx git-cloner 
Cloning into 'smt-demo'...

And that is not telling me much. How can I achieve a full repository download?

Testing again and now see the following inside minikube post minikube ssh:

$ cd /tmp/hostpath-provisioner/smt-local/code-pv-claim/
$ ls -la
total 12
drwxrwxrwx 2 root root 4096 Dec  7 06:57 .
drwxr-xr-x 5 root root 4096 Dec  7 05:02 ..
-rw-r--r-- 1 root root 1823 Dec  7 07:07 index.php
$ date
Sun Dec 13 03:30:03 UTC 2020

so that index was only added once and I can check in in the Laravel container:

kubectl exec -it app-74b5bdd58d-42rt6 -c laravel -- /bin/bash
root@app-74b5bdd58d-42rt6:/code# ls -la
total 12
drwxrwxrwx 2 root root 4096 Dec  7 06:57 .
drwxr-xr-x 1 root root 4096 Dec 13 03:19 ..
-rw-r--r-- 1 root root 1823 Dec  7 07:07 index.php
root@app-74b5bdd58d-42rt6:/code

# head index.php 
<?php

/**
 * Laravel - A PHP Framework For Web Artisans
 *
 * @package  Laravel
 * @author   Taylor Otwell <taylor@laravel.com>
 */

define('LARAVEL_START', microtime(true));

and though the initContainer does seem to run and terminate successfully it does not seem to add the repository code at all as of yet.

Getting more data / information using Git Sync

initContainers:
      - image: k8s.gcr.io/git-sync:v3.0.1
        name: git-data
        volumeMounts:
        - name: git-source
          mountPath: /tmp/git
        env:
        - name: GIT_SYNC_REPO
          value: https://github.com/smart48/smt-demo.git
        - name: GIT_SYNC_BRANCH
          value: master
        - name: GIT_SYNC_DEPTH
          value: "1"
        - name: GIT_SYNC_DEST
          value: git-sync
        - name: GIT_SYNC_ONE_TIME
          value: "1"
        # - name: GIT_SYNC_USERNAME
        #   valueFrom:
        #     secretKeyRef:
        #       name: git-creds
        #       key: username
        # - name: GIT_SYNC_PASSWORD
        #   valueFrom:
        #     secretKeyRef:
        #       name: git-creds
        #       key: password
      volumes:
        - name: git-source
          emptyDir: {}

as you can see doing this check

1     Running   0          3h26m
➜  smt-deploy git:(main) ✗ kubectl apply -f local/deployments/app.yml                     
deployment.apps/app created
➜  smt-deploy git:(main) ✗ kubectl logs -f app-55cf5f98d7-mkm6w git-data 
I1213 06:52:36.994829       1 main.go:179] starting up: ["/git-sync"]
I1213 06:52:44.510969       1 main.go:354] cloned https://github.com/smart48/smt-demo.git
I1213 06:52:44.512896       1 main.go:285] syncing to HEAD (debce33c0a2d34670262dd6ca241b844fd3dbc66)
I1213 06:52:46.182892       1 main.go:298] added worktree /tmp/git/rev-debce33c0a2d34670262dd6ca241b844fd3dbc66 for origin/master
I1213 06:52:46.766863       1 main.go:318] reset worktree /tmp/git/rev-debce33c0a2d34670262dd6ca241b844fd3dbc66 to debce33c0a2d34670262dd6ca241b844fd3dbc66
➜  smt-deploy git:(main) ✗ kubectl logs -f app-55cf5f98d7-mkm6w git-data
I1213 06:52:36.994829       1 main.go:179] starting up: ["/git-sync"]
I1213 06:52:44.510969       1 main.go:354] cloned https://github.com/smart48/smt-demo.git
I1213 06:52:44.512896       1 main.go:285] syncing to HEAD (debce33c0a2d34670262dd6ca241b844fd3dbc66)
I1213 06:52:46.182892       1 main.go:298] added worktree /tmp/git/rev-debce33c0a2d34670262dd6ca241b844fd3dbc66 for origin/master
I1213 06:52:46.766863       1 main.go:318] reset worktree /tmp/git/rev-debce33c0a2d34670262dd6ca241b844fd3dbc66 to debce33c0a2d34670262dd6ca241b844fd3dbc66

but still the actual storage remains an issue as `/tmp/git never seems to be created in the Minikube VM:

$ pwd
/tmp
$ ls -la
total 36
drwxrwxrwt 13 root   root    480 Dec 13 03:31 .
drwxr-xr-x 18 root   root    480 Dec 13 03:13 ..
drwxrwxrwt  2 root   root     40 Dec 13 03:13 .ICE-unix
drwxrwxrwt  2 root   root     40 Dec 13 03:13 .Test-unix
drwxrwxrwt  2 root   root     40 Dec 13 03:13 .X11-unix
drwxrwxrwt  2 root   root     40 Dec 13 03:13 .XIM-unix
drwxrwxrwt  2 root   root     40 Dec 13 03:13 .font-unix
drwxr-xr-x  2 root   root     40 Dec 13 03:13 gvisor
-rw-r--r--  1 docker docker   75 Dec 13 03:13 h.2198
-rw-r--r--  1 docker docker  121 Dec 13 03:13 h.2238
drwxr-xr-x  3 root   root   4096 Dec  7 05:01 hostpath-provisioner
drwxr-xr-x  2 root   root   4096 Dec  7 04:43 hostpath_pv
lrwxrwxrwx  1 root   root     68 Dec 13 03:16 storage-provisioner.ERROR -> storage-provisioner.minikube.unknownuser.log.ERROR.20201213-031625.1
lrwxrwxrwx  1 root   root     68 Dec 13 03:16 storage-provisioner.FATAL -> storage-provisioner.minikube.unknownuser.log.FATAL.20201213-031625.1
lrwxrwxrwx  1 root   root     67 Dec 13 03:17 storage-provisioner.INFO -> storage-provisioner.minikube.unknownuser.log.INFO.20201213-031747.1
lrwxrwxrwx  1 root   root     70 Dec 13 03:16 storage-provisioner.WARNING -> storage-provisioner.minikube.unknownuser.log.WARNING.20201213-031625.1
-rw-r--r--  1 root   root    340 Dec 13 03:16 storage-provisioner.minikube.unknownuser.log.ERROR.20201213-031625.1
-rw-r--r--  1 root   root    340 Dec 13 03:16 storage-provisioner.minikube.unknownuser.log.FATAL.20201213-031625.1
-rw-r--r--  1 root   root    447 Dec 13 03:16 storage-provisioner.minikube.unknownuser.log.INFO.20201213-031555.1
-rw-r--r--  1 root   root    403 Dec 13 03:18 storage-provisioner.minikube.unknownuser.log.INFO.20201213-031747.1
-rw-r--r--  1 root   root    340 Dec 13 03:16 storage-provisioner.minikube.unknownuser.log.WARNING.20201213-031625.1
drwx------  3 root   root     60 Dec 13 03:13 systemd-private-cf44a1507f1943c4a5445567b1ca5d73-systemd-logind.service-WZY7if
drwx------  3 root   root     60 Dec 13 03:13 systemd-private-cf44a1507f1943c4a5445567b1ca5d73-systemd-resolved.service-6E7a0h
drwx------  3 root   root     60 Dec 13 03:13 systemd-private-cf44a1507f1943c4a5445567b1ca5d73-systemd-timesyncd.service-zWXMCh

Found a better version of Git Sync Sidecart used once of in initContainer that sort of works:

# Laravel Code Download so we can run Horizon without issues 
# and have a codebase to start with For private repo better to add 
# code to laravel image https://codepre.com/how-to-perform-git-clone-in-kubernetes-pod-deployment.html
# https://github.com/kubernetes/git-sync/issues/127
# https://github.com/kubernetes/git-sync/issues/286
initContainers:
- image: k8s.gcr.io/git-sync:v3.0.1
  name: git-data
  # command: [ "sh", "-c", "ls /code/project" ]
  volumeMounts:
  - name: code-storage
    mountPath: /smt-local
  env:
  - name: GIT_SYNC_REPO
    value: https://github.com/smart48/smt-demo.git
  # - name: GIT_SYNC_ROOT
  #   value: /code
  - name: GIT_SYNC_BRANCH
    value: master
  - name: GIT_SYNC_DEPTH
    value: "1"
  - name: GIT_SYNC_DEST
    value: code
  - name: GIT_SYNC_ONE_TIME
    value: "1"
  # - name: GIT_SYNC_USERNAME
  #   valueFrom:
  #     secretKeyRef:
  #       name: git-creds
  #       key: username
  # - name: GIT_SYNC_PASSWORD
  #   valueFrom:
  #     secretKeyRef:
  #       name: git-creds
  #       key: password

works as in it does at all to symlink code leading to rev-debce33c0a2d34670262dd6ca241b844fd3dbc66/ where the code is stored:

tmp/hostpath-provisioner/smt-local/code-pv-claim
$ ls -la
total 16
drwxrwxrwx  4 root   root    4096 Dec 13 07:25 .
drwxr-xr-x  5 root   root    4096 Dec  7 05:02 ..
drwxr-xr-x  9 nobody nogroup 4096 Dec 13 07:25 .git
lrwxrwxrwx  1 nobody nogroup   44 Dec 13 07:25 code -> rev-debce33c0a2d34670262dd6ca241b844fd3dbc66
drwxr-xr-x 13 nobody nogroup 4096 Dec 13 07:25 rev-debce33c0a2d34670262dd6ca241b844fd3dbc66

Only no group and user added and… I would prefer a clone with out .git directory and perhaps a different name than rev-.. But perhaps this is a deployment solution like PHP Deployer I could use as well. And then with GIT_SYNC_ONE_TIME false

Will also run only once. That seems to be because after first run there is a conflict that is not shown in the logs because file(s) already exist

Because init containers can be restarted, retried, or re-executed, init container code should be idempotent. In particular, code that writes to files on EmptyDirs should be prepared for the possibility that an output file already exists.

Well, went back to git clone and reread https://codepre.com/how-to-perform-git-clone-in-kubernetes-pod-deployment.html . Needed to mount to data, connect it to same volume as Laravel Container volume:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: smt-local
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: laravel
          image: smart48/smt-laravel:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9000
          resources:
            requests:
              cpu: 250m
            limits:
              cpu: 500m
          volumeMounts:
          - name: code-storage
            mountPath: /code
        - name: nginx
          image: smart48/smt-nginx:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9376
          volumeMounts:
          - name: code-storage
            mountPath: /code
          - name: nginx-config
            mountPath: /etc/nginx/conf.d
        # - name: laravel-horizon
        #   image: smart48/smt-horizon:latest
        #   # not sure if we need the port and volume here
        #   imagePullPolicy: IfNotPresent
        #   ports:
        #     - containerPort: 9377
        #   volumeMounts:
        #     - name: code-storage
        #       mountPath: /code
        #   command: ["/usr/local/bin/php", "artisan", "horizon"]
        #   lifecycle:
        #     preStop:
        #       exec:
        #         command: ["/usr/local/bin/php", "artisan", "horizon:terminate"]
      # Laravel Code Download so we can run Horizon without issues 
      # and have a codebase to start with For private repo better to add 
      # code to laravel image https://codepre.com/how-to-perform-git-clone-in-kubernetes-pod-deployment.html
      initContainers:
        - name: git-cloner
          image: alpine/git
          args:
              - clone
              - --single-branch
              - --
              - https://github.com/smart48/smt-demo
              - /data
          volumeMounts:
          - mountPath: /data
            name: code-storage
      volumes:
        - name: code-storage
          persistentVolumeClaim:
            claimName: code-pv-claim
        - name: nginx-config
          persistentVolumeClaim:
            claimName: nginx-pv-claim

Did get all the files this time :slight_smile:
This worked, but for one of the pods replicas: 2 the init part crashed. Probably because the first pod beat the second one to it:

kubectl get all
NAME                         READY   STATUS                  RESTARTS   AGE
pod/app-7fd554668-hr4cj      2/2     Running                 0          2m35s
pod/app-7fd554668-xxxfp      0/2     Init:CrashLoopBackOff   2          2m35s
pod/mysql-686f78b8dd-9f7nx   1/1     Running                 0          2m42s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/app     1/2     2            1           2m35s
deployment.apps/mysql   1/1     1            1           2m42s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/app-7fd554668      2         2         1       2m35s
replicaset.apps/mysql-686f78b8dd   1         1         1       2m42s
➜  smt-deploy git:(main) ✗ kubectl get pods
NAME                     READY   STATUS                  RESTARTS   AGE
app-7fd554668-hr4cj      2/2     Running                 0          3m
app-7fd554668-xxxfp      0/2     Init:CrashLoopBackOff   3          3m
mysql-686f78b8dd-9f7nx   1/1     Running                 0          3m7s

And on inspection of the pod I see

kubectl describe pod app-7fd554668-xxxfp
Name:         app-7fd554668-xxxfp
Namespace:    smt-local
Priority:     0
Node:         minikube/192.168.64.15
Start Time:   Mon, 14 Dec 2020 08:44:47 +0700
Labels:       app=web
              pod-template-hash=7fd554668
Annotations:  <none>
Status:       Pending
IP:           172.17.0.6
IPs:
  IP:           172.17.0.6
Controlled By:  ReplicaSet/app-7fd554668
Init Containers:
  git-cloner:
    Container ID:  docker://f77942ea372792ec531cf699381abbd57ac334087d2fcdbfec59b4fc36152919
    Image:         alpine/git
    Image ID:      docker-pullable://alpine/git@sha256:24ad947c69479c22b1b394a2a1c36db3e416aa9b393dbe3bfc7515597140e411
    Port:          <none>
    Host Port:     <none>
    Args:
      clone
      --single-branch
      --
      https://github.com/smart48/smt-demo
      /data
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    128
      Started:      Mon, 14 Dec 2020 08:52:37 +0700
      Finished:     Mon, 14 Dec 2020 08:52:37 +0700
    Ready:          False
    Restart Count:  6
    Environment:    <none>
    Mounts:
      /data from code-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-zpncs (ro)

and the logs confirm the issue:

kubectl logs -f app-7fd554668-xxxfp git-cloner 
fatal: destination path '/data' already exists and is not an empty directory.

Not sure how I am supposed to deal with this yet.

Decided to not set replicas and perhaps later on fire a autoscale command instead.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: smt-local
spec:
  # autoscale using `kubectl autoscale deployment x --cpu-percent=50 --min=1 --max=10` instead of setting replicas
  # https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
  # replicas: 2 
  selector:
    matchLabels:
      app: web
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: laravel
          image: smart48/smt-laravel:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9000
          resources:
            requests:
              cpu: 250m
            limits:
              cpu: 500m
          volumeMounts:
          - name: code-storage
            mountPath: /code
        - name: nginx
          image: smart48/smt-nginx:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9376
          volumeMounts:
          - name: code-storage
            mountPath: /code
          - name: nginx-config
            mountPath: /etc/nginx/conf.d
        # - name: laravel-horizon
        #   image: smart48/smt-horizon:latest
        #   # not sure if we need the port and volume here
        #   imagePullPolicy: IfNotPresent
        #   ports:
        #     - containerPort: 9377
        #   volumeMounts:
        #     - name: code-storage
        #       mountPath: /code
        #   command: ["/usr/local/bin/php", "artisan", "horizon"]
        #   lifecycle:
        #     preStop:
        #       exec:
        #         command: ["/usr/local/bin/php", "artisan", "horizon:terminate"]
      # Laravel Code Download so we can run Horizon without issues 
      # and have a codebase to start with For private repo better to add 
      # code to laravel image https://codepre.com/how-to-perform-git-clone-in-kubernetes-pod-deployment.html
      initContainers:
        - name: git-cloner
          image: alpine/git
          args:
              - clone
              - --single-branch
              - --
              - https://github.com/smart48/smt-demo
              - /data
          volumeMounts:
          - mountPath: /data
            name: code-storage
      volumes:
        - name: code-storage
          persistentVolumeClaim:
            claimName: code-pv-claim
        - name: nginx-config
          persistentVolumeClaim:
            claimName: nginx-pv-claim

And at least for local testing this works now we have Laravel container, Nginx Container and Git Cloner working now though the initContainer only works once now.