Deployment failing on kubernetes with terraform (the server could not find the requested resource)

urmila.biradar · January 22, 2021, 12:26pm

I am trying to deploy our created docker image using terrafrom on local kubernetes (k3s). Also creating a service using nginx_ingress. While doing terraform apply getting error. Whatever information I have searched on net it looks like versioning issue but my kuberenetes version is up to date. Terrafrom version is also fine. Please help me to resolve this issue.
Error details are as below -

$ terraform apply --auto-approve
kubernetes_deployment.eve-statedb-deploy: Creating...
kubernetes_horizontal_pod_autoscaler.eve-statedb-hpa: Creating...
kubernetes_secret.eve-statedb-config: Creating...
kubernetes_secret.ghcr-cred: Creating...
Error: the server could not find the requested resource (post secrets)
Error: Failed to create deployment: the server could not find the requested resource (post deployments.apps)
Error: the server could not find the requested resource (post secrets)
Error: the server could not find the requested resource (post horizontalpodautoscalers.autoscaling)

Cluster information:

Kubernetes(using k3s for k8s set up) and terraform version:

$ sudo kubectl version  --short

Client Version: v1.20.0+k3s2
Server Version: v1.20.0+k3s2

$ terraform --version
Terraform v0.13.6
+ provider registry.terraform.io/hashicorp/kubernetes v2.0.1

devkulkarni · January 22, 2021, 12:49pm

One reason this type of error happens is if the API group of the Kubernetes objects is that is not recognizable in your cluster. For instance, if your Deployment YAML is using extensions/v1beta1 and if the cluster does not recognize Deployment to be part of extensions API group then this issue will happen.

Run ‘kubectl api-resources’ to find out what API groups exist for various resources in your cluster. Then update your YAML manifests to use those api groups (e.g.: use apps/v1 for Deployment instead of extensions/v1beta1, etc.)

urmila.biradar · January 22, 2021, 1:52pm

Thanks for the swift reply. In my deployment yaml, already included right api version “apiVersion: apps/v1”. I am new to terraform but as per my knowledge terraform will not use the existing deployment yaml. We need to configure it in “.tf” file like below

resource "kubernetes_deployment" "sample-statedb-deploy" {
  metadata {
    name = "sample-statedb-deploy"
    labels = {
      app = "sample-statedb"
    }
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        app = "sample-statedb"
      }
    }

    template {
      metadata {
        labels = {
          app = "sample-statedb"
        }
      }

      spec {
        container {
          image             = "statedb-sample-docker:dev-0.0.4"
          name              = "sample-statedb"
          image_pull_policy = "Always"

          port {
            container_port = 5000
          }

          lifecycle {
            pre_stop {
              exec {
                command = [ "/bin/bash",  "-c", "./config.sh remove --token $(curl -sS --request POST --url \"https://api.github.com/orgs/$${ORG_CODE}/actions/sample-statedbs/remove-token\" --header \"authorization: Bearer $${SVC_TOKEN}\"  --header \"content-type: application/json\" | jq -r .token)" ]
              }
            }
          }          

          resources {
            requests = {
              cpu    = "500m"
              memory = "1024Mi"
            }
            limits = {
              cpu    = "1"
              memory = "2048Mi"
            }
          }
          env {
            name = "SVC_TOKEN" 
              value_from {
                secret_key_ref {
                  name = "sample-statedb-config"
                  key  = "svc-token"  
                }
              }
          }
          env {
            name = "ORG_CODE" 
              value_from {
                secret_key_ref {
                  name = "sample-statedb-config"
                  key  = "org-code"
                }
              }
          } 
        }
        image_pull_secrets {
          name = "img-cred"
        } 
      } 
    }
  }
}
resource "kubernetes_horizontal_pod_autoscaler" "sample-statedb-hpa" {
  metadata {
    name = "sample-statedb-hpa"
  }

  spec {
    max_replicas = 4
    min_replicas = 1

    scale_target_ref {
      kind = "Deployment"
      name = "sample-statedb-deploy"
    }
    target_cpu_utilization_percentage = 75
  }
}

resource "kubernetes_service" "sample-statedb" {
  metadata {
    name = "sample-api-service"
  }
  spec {
    selector = {
      App = kubernetes_deployment.sample-statedb-deploy.metadata[0].labels.app
    }
    port {
      port        = 5000
      target_port = 5000
    }

    type = "LoadBalancer"
  }
}

output "lb_ip" {
  value = kubernetes_service.sample-statedb.status[0].load_balancer[0].ingress[0].ip
}

Output of kubectl api-resource :

$ sudo kubectl api-resources
NAME                              SHORTNAMES   APIVERSION                             NAMESPACED   KIND
bindings                                       v1                                     true         Binding
componentstatuses                 cs           v1                                     false        ComponentStatus
configmaps                        cm           v1                                     true         ConfigMap
endpoints                         ep           v1                                     true         Endpoints
events                            ev           v1                                     true         Event
limitranges                       limits       v1                                     true         LimitRange
namespaces                        ns           v1                                     false        Namespace
nodes                             no           v1                                     false        Node
persistentvolumeclaims            pvc          v1                                     true         PersistentVolumeClaim
persistentvolumes                 pv           v1                                     false        PersistentVolume
pods                              po           v1                                     true         Pod
podtemplates                                   v1                                     true         PodTemplate
replicationcontrollers            rc           v1                                     true         ReplicationController
resourcequotas                    quota        v1                                     true         ResourceQuota
secrets                                        v1                                     true         Secret
serviceaccounts                   sa           v1                                     true         ServiceAccount
services                          svc          v1                                     true         Service
mutatingwebhookconfigurations                  admissionregistration.k8s.io/v1        false        MutatingWebhookConfiguration
validatingwebhookconfigurations                admissionregistration.k8s.io/v1        false        ValidatingWebhookConfiguration
customresourcedefinitions         crd,crds     apiextensions.k8s.io/v1                false        CustomResourceDefinition
apiservices                                    apiregistration.k8s.io/v1              false        APIService
controllerrevisions                            apps/v1                                true         ControllerRevision
daemonsets                        ds           apps/v1                                true         DaemonSet
deployments                       deploy       apps/v1                                true         Deployment
replicasets                       rs           apps/v1                                true         ReplicaSet
statefulsets                      sts          apps/v1                                true         StatefulSet
tokenreviews                                   authentication.k8s.io/v1               false        TokenReview
localsubjectaccessreviews                      authorization.k8s.io/v1                true         LocalSubjectAccessReview
selfsubjectaccessreviews                       authorization.k8s.io/v1                false        SelfSubjectAccessReview
selfsubjectrulesreviews                        authorization.k8s.io/v1                false        SelfSubjectRulesReview
subjectaccessreviews                           authorization.k8s.io/v1                false        SubjectAccessReview
horizontalpodautoscalers          hpa          autoscaling/v1                         true         HorizontalPodAutoscaler
cronjobs                          cj           batch/v1beta1                          true         CronJob
jobs                                           batch/v1                               true         Job
certificatesigningrequests        csr          certificates.k8s.io/v1                 false        CertificateSigningRequest
leases                                         coordination.k8s.io/v1                 true         Lease
endpointslices                                 discovery.k8s.io/v1beta1               true         EndpointSlice
events                            ev           events.k8s.io/v1                       true         Event
ingresses                         ing          extensions/v1beta1                     true         Ingress
flowschemas                                    flowcontrol.apiserver.k8s.io/v1beta1   false        FlowSchema
prioritylevelconfigurations                    flowcontrol.apiserver.k8s.io/v1beta1   false        PriorityLevelConfiguration
helmchartconfigs                               helm.cattle.io/v1                      true         HelmChartConfig
helmcharts                                     helm.cattle.io/v1                      true         HelmChart
addons                                         k3s.cattle.io/v1                       true         Addon
nodes                                          metrics.k8s.io/v1beta1                 false        NodeMetrics
pods                                           metrics.k8s.io/v1beta1                 true         PodMetrics
ingressclasses                                 networking.k8s.io/v1                   false        IngressClass
ingresses                         ing          networking.k8s.io/v1                   true         Ingress
networkpolicies                   netpol       networking.k8s.io/v1                   true         NetworkPolicy
runtimeclasses                                 node.k8s.io/v1                         false        RuntimeClass
poddisruptionbudgets              pdb          policy/v1beta1                         true         PodDisruptionBudget
podsecuritypolicies               psp          policy/v1beta1                         false        PodSecurityPolicy
clusterrolebindings                            rbac.authorization.k8s.io/v1           false        ClusterRoleBinding
clusterroles                                   rbac.authorization.k8s.io/v1           false        ClusterRole
rolebindings                                   rbac.authorization.k8s.io/v1           true         RoleBinding
roles                                          rbac.authorization.k8s.io/v1           true         Role
priorityclasses                   pc           scheduling.k8s.io/v1                   false        PriorityClass
csidrivers                                     storage.k8s.io/v1                      false        CSIDriver
csinodes                                       storage.k8s.io/v1                      false        CSINode
storageclasses                    sc           storage.k8s.io/v1                      false        StorageClass
volumeattachments                              storage.k8s.io/v1                      false        VolumeAttachment

devkulkarni · January 22, 2021, 4:29pm

I don’t know what api version is Terraform generating for the resources. I hear that there is a ‘dry run’ option for Terraform. Try that to see if it gives a clue.
If not, I would suggest checking on Terraform forums as this issue is most likely because the apiVersion of the k8s objects that Terraform is generating is not present on your cluster.

Just to verify that is the case try these two things:

try doing a deployment directly with extensions/v1beta1 as apiVersion.
try using a different cluster in which extensions/v1beta is present (older k8s version) and use that with your terraform template above.

MizanIftee · February 4, 2021, 6:13am

Could you please check if kubernetes provider is given, mentioned Like below:

provider “kubernetes” {
host = hostsinfo
client_certificate = base64decode(hostcertificate)
client_key = base64decode(hostskey)
cluster_ca_certificate = base64decode(hostclientcertificate)
}
you will get all info from ~/.kube/config

PALLAPOTHU_MANOJ_SAI · October 26, 2021, 3:10pm

Hi guys I am getting the same error could you please help me to resolve it.

eocern · November 8, 2021, 1:53pm

Hi,

I had similar problem and it took me a bit of time to realize what was missing. I had the
provider “kubernetes” {
host = hostsinfo
client_certificate = base64decode(hostcertificate)
client_key = base64decode(hostskey)
cluster_ca_certificate = base64decode(hostclientcertificate)
}

but it was in a separate submodule. In my case I had to add the provider in the submodule causing me the original problem and all started to work fine.

Topic		Replies	Views
Getting "Error from server (NotFound): while deploying kustomize Windows	0	383	August 19, 2022
Error: failed to create deployment: General Discussions	1	2369	December 7, 2022
I am able to create deployment from yaml file. But nodes are not getting created General Discussions	2	659	April 3, 2019
Error from server (NotFound) General Discussions development	8	6552	August 1, 2019
FailedCreatePodSandBox General Discussions network	0	1037	August 24, 2023

Deployment failing on kubernetes with terraform (the server could not find the requested resource)

Cluster information:

Related Topics