How to solve 502 bad gateway in GKE cluster

Hi All,

I’m getting below error in GKE logging cluster on PROD ENV

I have 2 GKE cluster running in GCP

  1. Dev Env Cluster
  2. Prod Env Cluster

1 service is running in dev env same is security-test that has domain name and endpoint
security-test (this service name as running as container/pod in dev env)
Domain name :- https://privacy.ai.dev.test.ai (register in godaddy as A record can say subdomain)
endpoint :- https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec (this is 100% working as told by developer)

Below error is showing in prod env of another service name called TEST
test (this service name as running as container/pod in prod env)
endpoint :- https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec (this endpoint added in github repo in .ENV from there is fetching to dev env of security-test (anyway its given by developer in code they know it)

Here is the log showing in PROD env

ERROR 2024-07-09T17:45:38.167685925Z [resource.labels.containerName: test] }
ERROR 2024-07-09T17:45:38.167691406Z [resource.labels.containerName: test] }
ERROR 2024-07-09T17:45:38.176754883Z [resource.labels.containerName: test] Error updating field details with AI service: https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec AxiosError: Request failed with status code 502 at settle (/app/node_modules/axios/dist/node/axios.cjs:1967:12) at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3066:11) at IncomingMessage.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1358:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:3877:41) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) {
ERROR 2024-07-09T17:45:38.176931732Z [resource.labels.containerName: test] maxBodyLength: -1,
ERROR 2024-07-09T17:45:38.176936968Z [resource.labels.containerName: test] env: { FormData: [Function], Blob: null },
ERROR 2024-07-09T17:45:38.176942716Z [resource.labels.containerName: test] validateStatus: [Function: validateStatus],
ERROR 2024-07-09T17:45:38.176967489Z [resource.labels.containerName: test] headers: Object [AxiosHeaders] {
ERROR 2024-07-09T17:45:38.176972838Z [resource.labels.containerName: test] Accept: 'application/json, text/plain, */*',
ERROR 2024-07-09T17:45:38.176978217Z [resource.labels.containerName: test] 'Content-Type': 'application/json',
ERROR 2024-07-09T17:45:38.176983247Z [resource.labels.containerName: test] 'User-Agent': 'axios/1.6.7',
ERROR 2024-07-09T17:45:38.176988581Z [resource.labels.containerName: test] 'Content-Length': '1140',
ERROR 2024-07-09T17:45:38.176997428Z [resource.labels.containerName: test] 'Accept-Encoding': 'gzip, compress, deflate, br'
ERROR 2024-07-09T17:45:38.177003006Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.177008558Z [resource.labels.containerName: test] method: 'post',
ERROR 2024-07-09T17:45:38.177045615Z [resource.labels.containerName: test] request: <ref *1> ClientRequest {
ERROR 2024-07-09T17:45:38.176853594Z [resource.labels.containerName: test] code: 'ERR_BAD_RESPONSE',
ERROR 2024-07-09T17:45:38.176859304Z [resource.labels.containerName: test] config: {
ERROR 2024-07-09T17:45:38.176865215Z [resource.labels.containerName: test] transitional: {
ERROR 2024-07-09T17:45:38.176869804Z [resource.labels.containerName: test] silentJSONParsing: true,
ERROR 2024-07-09T17:45:38.176876185Z [resource.labels.containerName: test] forcedJSONParsing: true,
ERROR 2024-07-09T17:45:38.176881243Z [resource.labels.containerName: test] clarifyTimeoutError: false
ERROR 2024-07-09T17:45:38.178548164Z [resource.labels.containerName: test] transitional: [Object],
ERROR 2024-07-09T17:45:38.178554160Z [resource.labels.containerName: test] adapter: [Array],
ERROR 2024-07-09T17:45:38.178559991Z [resource.labels.containerName: test] transformRequest: [Array],
ERROR 2024-07-09T17:45:38.178566239Z [resource.labels.containerName: test] transformResponse: [Array],
ERROR 2024-07-09T17:45:38.178572641Z [resource.labels.containerName: test] timeout: 0,
ERROR 2024-07-09T17:45:38.178578788Z [resource.labels.containerName: test] xsrfCookieName: 'XSRF-TOKEN',
ERROR 2024-07-09T17:45:38.178588035Z [resource.labels.containerName: test] xsrfHeaderName: 'X-XSRF-TOKEN',
ERROR 2024-07-09T17:45:38.178594880Z [resource.labels.containerName: test] maxContentLength: -1,
ERROR 2024-07-09T17:45:38.178601129Z [resource.labels.containerName: test] maxBodyLength: -1,
ERROR 2024-07-09T17:45:38.178606872Z [resource.labels.containerName: test] env: [Object],
ERROR 2024-07-09T17:45:38.178612398Z [resource.labels.containerName: test] validateStatus: [Function: validateStatus],
ERROR 2024-07-09T17:45:38.178618445Z [resource.labels.containerName: test] headers: [Object [AxiosHeaders]],
ERROR 2024-07-09T17:45:38.178624425Z [resource.labels.containerName: test] method: 'post',
ERROR 2024-07-09T17:45:38.178630320Z [resource.labels.containerName: test] url: 'https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec',
ERROR 2024-07-09T17:45:38.178639363Z [resource.labels.containerName: test] data: '{"initialEndpoints":[{"name":"business_id","location":"Request Field.business_id","label":"business_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"customer_id","location":"Request Field.customer_id","label":"customer_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"request_id","location":"Request Field.request_id","label":"request_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"vendor_access_token","location":"Request Field.vendor_access_token","label":"vendor_access_token","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"vendor_customer_id","location":"Request Field.vendor_customer_id","label":"vendor_customer_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"vendor_institution_id","location":"Request Field.vendor_institution_id","label":"vendor_institution_id","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"},{"name":"vendor_public_token","location":"Request Field.vendor_public_token","label":"vendor_public_token","isPII":false,"isPCI":false,"isHIPAA":false,"impact":"Secure"}]}'
ERROR 2024-07-09T17:45:38.178646406Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.178652737Z [resource.labels.containerName: test] request: <ref *1> ClientRequest {
ERROR 2024-07-09T17:45:38.178658793Z [resource.labels.containerName: test] _events: [Object: null prototype],
ERROR 2024-07-09T17:45:38.178664457Z [resource.labels.containerName: test] _eventsCount: 7,
ERROR 2024-07-09T17:45:38.178670377Z [resource.labels.containerName: test] _maxListeners: undefined,
ERROR 2024-07-09T17:45:38.178676401Z [resource.labels.containerName: test] outputData: [],
ERROR 2024-07-09T17:45:38.178682602Z [resource.labels.containerName: test] outputSize: 0,
ERROR 2024-07-09T17:45:38.178688339Z [resource.labels.containerName: test] writable: true,
ERROR 2024-07-09T17:45:38.178694513Z [resource.labels.containerName: test] destroyed: false,
ERROR 2024-07-09T17:45:38.178700556Z [resource.labels.containerName: test] _last: true,
ERROR 2024-07-09T17:45:38.178706554Z [resource.labels.containerName: test] chunkedEncoding: false,
ERROR 2024-07-09T17:45:38.178712375Z [resource.labels.containerName: test] shouldKeepAlive: false,
ERROR 2024-07-09T17:45:38.178718126Z [resource.labels.containerName: test] maxRequestsOnConnectionReached: false,
ERROR 2024-07-09T17:45:38.178724127Z [resource.labels.containerName: test] _defaultKeepAlive: true,
ERROR 2024-07-09T17:45:38.178730216Z [resource.labels.containerName: test] useChunkedEncodingByDefault: true,
ERROR 2024-07-09T17:45:38.178736233Z [resource.labels.containerName: test] sendDate: false,
ERROR 2024-07-09T17:45:38.178742273Z [resource.labels.containerName: test] _removedConnection: false,
ERROR 2024-07-09T17:45:38.178748093Z [resource.labels.containerName: test] _removedContLen: false,
ERROR 2024-07-09T17:45:38.178753683Z [resource.labels.containerName: test] _removedTE: false,
ERROR 2024-07-09T17:45:38.178760256Z [resource.labels.containerName: test] strictContentLength: false,
ERROR 2024-07-09T17:45:38.178766740Z [resource.labels.containerName: test] _contentLength: '1140',
ERROR 2024-07-09T17:45:38.178772550Z [resource.labels.containerName: test] _hasBody: true,
ERROR 2024-07-09T17:45:38.178779103Z [resource.labels.containerName: test] _trailer: '',
ERROR 2024-07-09T17:45:38.178785017Z [resource.labels.containerName: test] finished: true,
ERROR 2024-07-09T17:45:38.178790888Z [resource.labels.containerName: test] _headerSent: true,
ERROR 2024-07-09T17:45:38.178804555Z [resource.labels.containerName: test] _closed: false,
ERROR 2024-07-09T17:45:38.177940275Z [resource.labels.containerName: test] aborted: false,
ERROR 2024-07-09T17:45:38.177946381Z [resource.labels.containerName: test] upgrade: false,
ERROR 2024-07-09T17:45:38.177952468Z [resource.labels.containerName: test] url: '',
ERROR 2024-07-09T17:45:38.177958267Z [resource.labels.containerName: test] method: null,
ERROR 2024-07-09T17:45:38.177964449Z [resource.labels.containerName: test] statusCode: 502,
ERROR 2024-07-09T17:45:38.177970059Z [resource.labels.containerName: test] statusMessage: 'Bad Gateway',
ERROR 2024-07-09T17:45:38.177976151Z [resource.labels.containerName: test] client: [TLSSocket],
ERROR 2024-07-09T17:45:38.177981960Z [resource.labels.containerName: test] _consuming: false,
ERROR 2024-07-09T17:45:38.177987765Z [resource.labels.containerName: test] _dumped: false,
ERROR 2024-07-09T17:45:38.177993847Z [resource.labels.containerName: test] req: [Circular *1],
ERROR 2024-07-09T17:45:38.178000128Z [resource.labels.containerName: test] responseUrl: 'https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec',
ERROR 2024-07-09T17:45:38.178015296Z [resource.labels.containerName: test] redirects: [],
ERROR 2024-07-09T17:45:38.178021831Z [resource.labels.containerName: test] [Symbol(kCapture)]: false,
ERROR 2024-07-09T17:45:38.178027843Z [resource.labels.containerName: test] [Symbol(kHeaders)]: [Object],
ERROR 2024-07-09T17:45:38.178034523Z [resource.labels.containerName: test] [Symbol(kHeadersCount)]: 12,
ERROR 2024-07-09T17:45:38.178040646Z [resource.labels.containerName: test] [Symbol(kTrailers)]: null,
ERROR 2024-07-09T17:45:38.178050602Z [resource.labels.containerName: test] [Symbol(kTrailersCount)]: 0,
ERROR 2024-07-09T17:45:38.178056327Z [resource.labels.containerName: test] [Symbol(RequestTimeout)]: undefined
ERROR 2024-07-09T17:45:38.178062331Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.178068224Z [resource.labels.containerName: test] aborted: false,
ERROR 2024-07-09T17:45:38.178810323Z [resource.labels.containerName: test] socket: [TLSSocket],
ERROR 2024-07-09T17:45:38.178816800Z [resource.labels.containerName: test] _header: 'POST /api-analyzer/fetch-api-spec HTTP/1.1\r\n' +
ERROR 2024-07-09T17:45:38.178822068Z [resource.labels.containerName: test] 'Accept: application/json, text/plain, */*\r\n' +
ERROR 2024-07-09T17:45:38.178827145Z [resource.labels.containerName: test] 'Content-Type: application/json\r\n' +
ERROR 2024-07-09T17:45:38.178833020Z [resource.labels.containerName: test] 'User-Agent: axios/1.6.7\r\n' +
ERROR 2024-07-09T17:45:38.178838987Z [resource.labels.containerName: test] 'Content-Length: 1140\r\n' +
ERROR 2024-07-09T17:45:38.178844833Z [resource.labels.containerName: test] 'Accept-Encoding: gzip, compress, deflate, br\r\n' +
ERROR 2024-07-09T17:45:38.178850525Z [resource.labels.containerName: test] 'Host: privacy.ai.dev.test.ai\r\n' +
ERROR 2024-07-09T17:45:38.178856171Z [resource.labels.containerName: test] 'Connection: close\r\n' +
ERROR 2024-07-09T17:45:38.178861452Z [resource.labels.containerName: test] '\r\n',
ERROR 2024-07-09T17:45:38.178867493Z [resource.labels.containerName: test] _keepAliveTimeout: 0,
ERROR 2024-07-09T17:45:38.178873614Z [resource.labels.containerName: test] _onPendingData: [Function: nop],
ERROR 2024-07-09T17:45:38.178879514Z [resource.labels.containerName: test] agent: [Agent],
ERROR 2024-07-09T17:45:38.178885497Z [resource.labels.containerName: test] socketPath: undefined,
ERROR 2024-07-09T17:45:38.178891154Z [resource.labels.containerName: test] method: 'POST',
ERROR 2024-07-09T17:45:38.178897283Z [resource.labels.containerName: test] maxHeaderSize: undefined,
ERROR 2024-07-09T17:45:38.178903044Z [resource.labels.containerName: test] insecureHTTPParser: undefined,
ERROR 2024-07-09T17:45:38.178909214Z [resource.labels.containerName: test] path: '/api-analyzer/fetch-api-spec',
ERROR 2024-07-09T17:45:38.178915164Z [resource.labels.containerName: test] _ended: true,
ERROR 2024-07-09T17:45:38.178920714Z [resource.labels.containerName: test] res: [IncomingMessage],
ERROR 2024-07-09T17:45:38.178926597Z [resource.labels.containerName: test] aborted: false,
ERROR 2024-07-09T17:45:38.178932395Z [resource.labels.containerName: test] timeoutCb: null,
ERROR 2024-07-09T17:45:38.178938568Z [resource.labels.containerName: test] upgradeOrConnect: false,
ERROR 2024-07-09T17:45:38.178971909Z [resource.labels.containerName: test] parser: null,
ERROR 2024-07-09T17:45:38.178979560Z [resource.labels.containerName: test] maxHeadersCount: null,
ERROR 2024-07-09T17:45:38.178985241Z [resource.labels.containerName: test] reusedSocket: false,
ERROR 2024-07-09T17:45:38.178991521Z [resource.labels.containerName: test] host: 'privacy.ai.dev.test.ai',
ERROR 2024-07-09T17:45:38.178449005Z [resource.labels.containerName: test] [Symbol(kUniqueHeaders)]: null
ERROR 2024-07-09T17:45:38.178455189Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.178461168Z [resource.labels.containerName: test] response: {
ERROR 2024-07-09T17:45:38.178467312Z [resource.labels.containerName: test] status: 502,
ERROR 2024-07-09T17:45:38.178473740Z [resource.labels.containerName: test] statusText: 'Bad Gateway',
ERROR 2024-07-09T17:45:38.178479670Z [resource.labels.containerName: test] headers: Object [AxiosHeaders] {
ERROR 2024-07-09T17:45:38.178486053Z [resource.labels.containerName: test] 'content-type': 'text/html; charset=UTF-8',
ERROR 2024-07-09T17:45:38.178492197Z [resource.labels.containerName: test] 'referrer-policy': 'no-referrer',
ERROR 2024-07-09T17:45:38.178498332Z [resource.labels.containerName: test] 'content-length': '332',
ERROR 2024-07-09T17:45:38.178504108Z [resource.labels.containerName: test] date: 'Tue, 09 Jul 2024 17:45:38 GMT',
ERROR 2024-07-09T17:45:38.178511Z [resource.labels.containerName: test] 'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000',
ERROR 2024-07-09T17:45:38.178529028Z [resource.labels.containerName: test] connection: 'close'
ERROR 2024-07-09T17:45:38.178996766Z [resource.labels.containerName: test] protocol: 'https:',
ERROR 2024-07-09T17:45:38.179001897Z [resource.labels.containerName: test] _redirectable: [Writable],
ERROR 2024-07-09T17:45:38.179006948Z [resource.labels.containerName: test] [Symbol(kCapture)]: false,
ERROR 2024-07-09T17:45:38.179012617Z [resource.labels.containerName: test] [Symbol(kBytesWritten)]: 0,
ERROR 2024-07-09T17:45:38.179018157Z [resource.labels.containerName: test] [Symbol(kEndCalled)]: true,
ERROR 2024-07-09T17:45:38.179023423Z [resource.labels.containerName: test] [Symbol(kNeedDrain)]: false,
ERROR 2024-07-09T17:45:38.179028620Z [resource.labels.containerName: test] [Symbol(corked)]: 0,
ERROR 2024-07-09T17:45:38.179034587Z [resource.labels.containerName: test] [Symbol(kOutHeaders)]: [Object: null prototype],
ERROR 2024-07-09T17:45:38.179040173Z [resource.labels.containerName: test] [Symbol(kUniqueHeaders)]: null
ERROR 2024-07-09T17:45:38.179046285Z [resource.labels.containerName: test] },
ERROR 2024-07-09T17:45:38.179052728Z [resource.labels.containerName: test] data: '\n' +
ERROR 2024-07-09T17:45:38.179058145Z [resource.labels.containerName: test] '<html><head>\n' +
ERROR 2024-07-09T17:45:38.179063778Z [resource.labels.containerName: test] '<meta http-equiv="content-type" content="text/html;charset=utf-8">\n' +
ERROR 2024-07-09T17:45:38.179073355Z [resource.labels.containerName: test] '<title>502 Server Error</title>\n' +
ERROR 2024-07-09T17:45:38.179079448Z [resource.labels.containerName: test] '</head>\n' +
ERROR 2024-07-09T17:45:38.179085177Z [resource.labels.containerName: test] '<body text=#000000 bgcolor=#ffffff>\n' +
ERROR 2024-07-09T17:45:38.179090962Z [resource.labels.containerName: test] '<h1>Error: Server Error</h1>\n' +
ERROR 2024-07-09T17:45:38.179107509Z [resource.labels.containerName: test] '<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>\n' +
ERROR 2024-07-09T17:45:38.179113834Z [resource.labels.containerName: test] '<h2></h2>\n' +
ERROR 2024-07-09T17:45:38.179119609Z [resource.labels.containerName: test] '</body></html>\n'
ERROR 2024-07-09T17:45:38.179125141Z [resource.labels.containerName: test] }
ERROR 2024-07-09T17:45:38.179130953Z [resource.labels.containerName: test] }
ERROR 2024-07-09T17:45:38.194095673Z [resource.labels.containerName: test] Error updating field details with AI service: https://privacy.ai.dev.test.ai/api-analyzer/fetch-api-spec AxiosError: Request failed with status code 502 at settle (/app/node_modules/axios/dist/node/axios.cjs:1967:12) at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3066:11) at IncomingMessage.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1358:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:3877:41) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) {

Here is the YAML of security-test

apiVersion: apps/v1
kind: Deployment
metadata:
 name: security-ai
 namespace: test
spec:
 replicas: 2
 selector:
   matchLabels:
     app: security-test
 template:
   metadata:
     labels:
       app: security-test
   spec:
     containers:
       - name: security-test
         image: australia-southeast1-docker.pkg.dev/test-dev/security-test/security-test:2024-07-04-0856
         imagePullPolicy: Always
         ports:
         - containerPort: 3000
         readinessProbe:
          httpGet:
            path: /
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 1
         livenessProbe:
          httpGet:
            path: /
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
          timeoutSeconds: 1
         startupProbe:
          httpGet:
            path: /
            port: 3000
          initialDelaySeconds: 60
          failureThreshold: 0
          periodSeconds: 10
          timeoutSeconds: 1
         resources:
          requests: # Minimum amount of resources requested
             cpu: 500m
             memory: 1Gi
          limits: # Maximum amount of resources requested
             cpu: 1000m
             memory: 2Gi 
---
apiVersion: v1
kind: Service
metadata:
 name: security-test
 namespace: test
 labels:
   app: security-test
spec:
 type: ClusterIP
 selector:
  app: security-test
 ports:
  - port: 3000
    targetPort: 3000
    protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-security-test
  namespace: test
  annotations:
    kubernetes.io/ingress.global-static-ip-name: ingress-security-test
    networking.gke.io/managed-certificates: managed-cert-security-test
    ingressClassName: "gce"
spec:
  rules: 
  - host: "privacy.ai.dev.test.ai"
    http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
            service:
               name: security-test
               port:
                  number: 3000    
---
# apiVersion: networking.k8s.io/v1
# kind: Ingress
# metadata:
#   name: ingress-security-test
#   namespace: test
#   annotations:
#     kubernetes.io/ingress.global-static-ip-name: ingress-security-test
#     networking.gke.io/managed-certificates: managed-cert-security-test
#     ingressClassName: "gce"
# spec:
#   defaultBackend:
#     service:
#       name: security-test
#       port:
#         number: 3000
---
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
 name: managed-cert-security-test
 namespace: test
spec:
 domains:
   - privacy.ai.dev.test.ai

What im missing in ingress or deployment file please do let me know its high priority issue im facing since 4days

Even sometimes i get liveliness and readlines fail due to client.timeout issue
just now i describe pods i got readiness and liveness probe