Pod can't connect to external services

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.16.6-do.0
Cloud being used: DigitalOcean
Installation method: DigitalOcean UI
Host OS: Debian GNU/Linux 9 (stretch) 4.19.0-0.bpo.6-amd64
CNI and version: Not sure how to find this
CRI and version: Not sure how to find this

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

I’ve been trying to create a simple deployment with a single container that runs a Node.js application. The application needs to be able to call out to MongoDB Atlas and RabbitMQ on CloudAMQP. I’ve gotten it to the point where the application runs and I can connect to it externally (using LB and Ingress Controller). However, when I run the application, it fails to connect to either service and shows the following errors that I have outputted to a status page:

DB connection: Error: MongoServerSelectionError: Authentication failed.
Queue channel connection: Error: getaddrinfo ENOTFOUND albatross.rmq.cloudamqp.com

Running the app locally and/or running the exact same container on my DO droplet works fine and successfully connects to both, so I know it is not the application configuration that is the issue. Additionally, I created 2 node test scripts that simply try connecting to those services and report back whether they succeed or not. And when connecting to the running pod using kubectl exec -it node-template-uid -- /bin/bash, I can run the scripts manually using node and they report back success. Therefore, there’s not an issue with the pod itself connecting to the services, just an issue from the foreground running node service.

Here is my overall deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: node-template
spec:
  selector:
    matchLabels:
      app: node-template
  replicas: 1
  template:
    metadata:
      labels:
        app: node-template
    spec:
      containers:
      - name: node-template
        image: thetablestop/node-template
        imagePullPolicy: Always
        env:
        - name: NODE_ENV
          value: Production
        - name: NODE_PORT
          value: "8080"
        - name: MONGODB_NAME
          value: "processor"
        envFrom:
        - secretRef:
            name: pod-env-vars
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /
            port: 8080
            scheme: HTTP
        readinessProbe:
          httpGet:
            path: /
            port: 8080
            scheme: HTTP
      dnsPolicy: Default
---
apiVersion: v1
kind: Service
metadata:
  name: node-template-svc
  annotations:
    external-dns.alpha.kubernetes.io/hostname: node-template.thetablestop.com
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: node-template
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: node-template-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - node-template.thetablestop.com
    secretName: node-template-tls
  rules:
  - host: node-template.thetablestop.com
    http:
      paths:
      - backend:
          serviceName: node-template-svc
          servicePort: 80
---
apiVersion: v1
kind: Secret
metadata:
  name: pod-env-vars
type: Opaque
data:
  MONGODB_HOST: <redacted>
  MONGODB_LOGINDB: <redacted>
  MONGODB_USER: <redacted>
  MONGODB_PASS: <redacted>
  RABBITMQ_HOST: <redacted>
  RABBITMQ_VHOST: <redacted>
  RABBITMQ_USER: <redacted>
  RABBITMQ_PASS: <redacted>