Hi Team ,
I am testing out the Shared Persistent Volumes on my Cluster . I have created an EKS cluster on AWS , I have installed nfs server on one of the nodes & created PV& PVC for the same . I am able to mount the volumes with multiple applications (i am able to see the data in nfs-server, application pod & nginx pod ) .But i am unable to access the html files from nginx server as it throws 403 Forbidden Error .
Below are the Screenshot of my NFS -Sever ,nginx where my data mounted on it with permissions
Kindly help me with where i am doing wrong
Thank you in advance
Please find the nginx screenshot which shows the data is actually being mounted to Nginx pod
What’s in the nginx error logs?
When I checked out the container, the nginx image is built to direct all error output to stderr:
$ kubectl run nginx --rm -it --image=nginx:latest -- /bin/bash
If you don't see a command prompt, try pressing enter.
root@nginx:/# ls -lah /var/log/nginx/error.log
lrwxrwxrwx 1 root root 11 May 25 15:43 /var/log/nginx/error.log -> /dev/stderr
So this output should land in the container logs. You can check with this command:
kubectl logs pod/nginx-pod-name -c container-name-here
The -c container-name-here is not necessary if this is a single container pod.
Hi @protosam Thank you for the reply , below are the logs for my pod
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/06/07 13:42:14 [notice] 1#1: using the “epoll” event method
2021/06/07 13:42:14 [notice] 1#1: nginx/1.21.0
2021/06/07 13:42:14 [notice] 1#1: built by gcc 8.3.0 (Debian 8.3.0-6)
2021/06/07 13:42:14 [notice] 1#1: OS: Linux 5.4.117-58.216.amzn2.x86_64
2021/06/07 13:42:14 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/06/07 13:42:14 [notice] 1#1: start worker processes
2021/06/07 13:42:14 [notice] 1#1: start worker process 30
2021/06/07 13:42:14 [notice] 1#1: start worker process 31
2021/06/07 13:42:14 [notice] 1#1: start worker process 32
2021/06/07 13:42:14 [notice] 1#1: start worker process 33
2021/06/07 13:42:14 [notice] 1#1: start worker process 34
2021/06/07 13:42:14 [notice] 1#1: start worker process 35
2021/06/07 13:42:14 [notice] 1#1: start worker process 36
2021/06/07 13:42:14 [notice] 1#1: start worker process 37
2021/06/07 13:44:15 [error] 30#30: *1 directory index of “/usr/share/nginx/html/” is forbidden, client: 127.0.0.1, server: localhost, request: “GET / HTTP/1.1”, host: “localhost”
127.0.0.1 - - [07/Jun/2021:13:44:15 +0000] “GET / HTTP/1.1” 403 153 “-” “curl/7.64.0” “-”
What’s the output from ls -lah /usr/share/nginx/html/ inside that container?
Same as you got when you ran the command
Oh, I missed your 2nd post. Looks like you don’t have an index file.
i do have .html files inside the folders test & test1
Your screenshot shows mysql, test, and test1 inside of /usr/share/nginx/html.
I don’t see an index file in /usr/share/nginx/html.
Hi @protosam , Thank you for the reply . FYI : I am using NFS Shared volume between Minio & Nginx . Nginx (/usr/share/nginx/html) will have minio buckets & respective files(files are inside the test folder & test 1folder )
Nginx should show those files on it servers but it throws 403 forbidden .
I hope this give u clarification on my scenario
Here’s a demo of how the nginx pod works. I started off with a plain nginx pod, replicated the 403 Forbidden error, and fixed it. Hope this helps clarify. 
protosam@github-storage $ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: nginx-demo
spec:
containers:
- name: nginx
image: nginx:latest
EOF
pod/nginx-demo created
protosam@github-storage $ kubectl exec -it nginx-demo -- bash
root@nginx-demo:/# curl localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@nginx-demo:/# cd /usr/share/nginx/html
root@nginx-demo:/usr/share/nginx/html# ls -lah
total 16K
drwxr-xr-x 2 root root 4.0K May 25 15:43 .
drwxr-xr-x 3 root root 4.0K May 25 15:43 ..
-rw-r--r-- 1 root root 494 May 25 12:28 50x.html
-rw-r--r-- 1 root root 612 May 25 12:28 index.html
root@nginx-demo:/usr/share/nginx/html# mv index.html removed.html
root@nginx-demo:/usr/share/nginx/html# curl localhost
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.21.0</center>
</body>
</html>
root@nginx-demo:/usr/share/nginx/html# mv removed.html index.html
root@nginx-demo:/usr/share/nginx/html# ls -lah
total 24K
drwxr-xr-x 1 root root 4.0K Jun 10 19:29 .
drwxr-xr-x 1 root root 4.0K May 25 15:43 ..
-rw-r--r-- 1 root root 494 May 25 12:28 50x.html
-rw-r--r-- 1 root root 612 May 25 12:28 removed.html
root@nginx-demo:/usr/share/nginx/html# curl localhost
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@nginx-demo:/usr/share/nginx/html# exit
command terminated with exit code 130
protosam@github-storage $ kubectl delete pod nginx-demo
pod "nginx-demo" deleted
@Nisha_Shaik any luck with this? I have this same issue and I do not have it on other clusters with the same configuration.
Hi @Mark_Meadows , I have figured out solution. In my case as I am using NFS on my EC2 instances we need to mount the NFS on both instances (for ex : if 2 instances then 1 instance work as Server & other as Client). I missed mounting the NFS Mount on Client machine.so i was facing the Forbidden error
In case anyone else stumbles on this, in my case it was an ingress annotation from an old ingress that did a rewrite. I had to turn on autoindex to find the rewrite caused an infinite directory loop. (/static/static/static/static)
Nginx 403 Forbidden error is a status code generated and displayed to the user when a client tries to access a part of the webserver with insufficient permissions. When nginx access a directory, it tries to index it and return the list of files inside it to the browser/client, however by default directory indexing is disabled, and so it returns the Nginx 403 forbidden error.
Incorrect Index File
The try_files tries the literal path you specify in relation to the defined root directive and sets the internal file pointer. If you have directory indexing off, and is having this problem, it’s probably because the try_files you are using has a directory option:
location / {
try_files $uri $uri/ /index.html index.php;
}
to
location / {
try_files $uri /index.html index.php;
}
Incorrectly set permissions
This error can also result from files and directories having incorrectly set permissions. In order to resolve this , change the directories permission to 755 and the file permissions to 644 . Make sure that the user running the Nginx process owns the files. For example, set user to www-data:
sudo chown -R www-data:www-data *
Finally, set the directory and file permissions as:
sudo chmod 755 {dir}
sudo chmod 644 {files}
