Nni frameworkcontroller error (Failed to put CRD: the server could not find the requested resource)

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version: 1.22.1
Cloud being used: (put bare-metal if not on a public cloud)
Installation method:
Host OS: Ubuntu 18.04
CNI and version:
CRI and version:

You can format your yaml by highlighting it and pressing Ctrl-Shift-C, it will make your output easier to read.

I1001 07:08:06.475855       8 controller.go:207] Initializing frameworkcontroller
I1001 07:08:06.480273       8 controller.go:210] With Config:
kubeApiServerAddress: ""
kubeConfigFilePath: ""
kubeClientQps: 80
kubeClientBurst: 120
workerNumber: 200
largeFrameworkCompression: false
crdEstablishedCheckIntervalSec: 1
crdEstablishedCheckTimeoutSec: 60
objectLocalCacheCreationTimeoutSec: 300
frameworkCompletedRetainSec: 2592000
frameworkMinRetryDelaySecForTransientConflictFailed: 60
frameworkMaxRetryDelaySecForTransientConflictFailed: 900
logObjectSnapshot:
  framework:
    onFrameworkRetry: true
    onFrameworkDeletion: true
  task:
    onTaskRetry: true
    onTaskDeletion: true
  pod:
    onPodDeletion: true
podFailureSpec:
- code: -1000
  phrase: PodEvicted
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^Evicted$
    messageRegex: (?ms).*
- code: -1001
  phrase: PodNodeLost
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^NodeLost$
    messageRegex: (?ms).*
- code: -1002
  phrase: PodScheduledToInsufficientResourceNode
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^OutOf\S+$
    messageRegex: (?ms).*
- code: -1003
  phrase: PodPreemptedForCriticalPod
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^Preempting$
    messageRegex: (?ms).*
- code: -1004
  phrase: PodDeadlineExceeded
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - reasonRegex: (?i)^DeadlineExceeded$
    messageRegex: (?ms).*
- code: -1005
  phrase: PodNodeAdmissionForbidden
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - reasonRegex: (?i)^Forbidden$
    messageRegex: (?ms).*
- code: -1006
  phrase: PodNodeAdmissionUnexpectedError
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - reasonRegex: (?i)^UnexpectedAdmissionError$
    messageRegex: (?ms).*
  - reasonRegex: (?i)^UnknownReason$
    messageRegex: (?ms).*
  - reasonRegex: (?i)^InvalidNodeInfo$
    messageRegex: (?ms).*
  - reasonRegex: (?i)^UnexpectedPredicateFailureType$
    messageRegex: (?ms).*
- code: -1200
  phrase: ContainerDockerOOMKilled
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^OOMKilled$
      messageRegex: (?ms).*
      codeRange:
        min: 1
- code: -1201
  phrase: ContainerDockerCreationError
  type:
    name: Failed
    attributes: []
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?ms).*
      codeRange:
        min: 125
        max: 125
- code: -1202
  phrase: ContainerDockerStartCmdPermissionDenied
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?ms).*
      codeRange:
        min: 126
        max: 126
- code: -1203
  phrase: ContainerDockerStartCmdNotFound
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?ms).*
      codeRange:
        min: 127
        max: 127
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?msi).*(not found|cannot find|no such).*
      codeRange:
        min: 128
        max: 128
- code: -1204
  phrase: ContainerDockerStartUnknownError
  type:
    name: Failed
    attributes: []
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      reasonRegex: (?i)^ContainerCannotRun$
      messageRegex: (?ms).*
      codeRange:
        min: 128
        max: 128
- code: 130
  phrase: ContainerSigIntReceived
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 130
        max: 130
- code: 131
  phrase: ContainerSigQuitReceived
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 131
        max: 131
- code: 132
  phrase: ContainerSigIllReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 132
        max: 132
- code: 134
  phrase: ContainerSigAbrtReceived
  type:
    name: Failed
    attributes: []
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 134
        max: 134
- code: 135
  phrase: ContainerSigBusReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 135
        max: 135
- code: 136
  phrase: ContainerSigFpeReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 136
        max: 136
- code: 137
  phrase: ContainerSigKillReceived
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 137
        max: 137
- code: 139
  phrase: ContainerSigSegvReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 139
        max: 139
- code: 141
  phrase: ContainerSigPipeReceived
  type:
    name: Failed
    attributes:
    - Permanent
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 141
        max: 141
- code: 143
  phrase: ContainerSigTermReceived
  type:
    name: Failed
    attributes:
    - Transient
  podPatterns:
  - containers:
    - nameRegex: (?ms).*
      messageRegex: (?ms).*
      codeRange:
        min: 143
        max: 143
W1001 07:08:06.480346       8 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1001 07:08:06.483368       8 controller.go:427] Recovering frameworkcontroller
I1001 07:08:06.500651       8 utils.go:108] Create CRD frameworks.frameworkcontroller.microsoft.com
E1001 07:08:06.506182       8 runtime.go:69] Observed a panic: &errors.errorString{s:"Failed to put CRD: the server could not find the requested resource"} (Failed to put CRD: the server could not find the requested resource)
/go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/go/src/github.com/microsoft/frameworkcontroller/pkg/internal/utils.go:66
/go/src/github.com/microsoft/frameworkcontroller/pkg/controller/controller.go:428
/go/src/github.com/microsoft/frameworkcontroller/cmd/frameworkcontroller/main.go:35
/usr/local/go/src/runtime/proc.go:200
/usr/local/go/src/runtime/asm_amd64.s:1337
E1001 07:08:06.506227       8 panic.go:522] Stopping frameworkcontroller
panic: Failed to put CRD: the server could not find the requested resource [recovered]
        panic: Failed to put CRD: the server could not find the requested resource

goroutine 1 [running]:
github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
        /go/src/github.com/microsoft/frameworkcontroller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x11f84e0, 0xc0004160e0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/microsoft/frameworkcontroller/pkg/internal.PutCRD(0xc0002d70e0, 0xc0004fe000, 0xc00035d0a8, 0xc00035d0b0)
        /go/src/github.com/microsoft/frameworkcontroller/pkg/internal/utils.go:66 +0x173
github.com/microsoft/frameworkcontroller/pkg/controller.(*FrameworkController).Run(0xc0000ed3f0, 0xc0000cb140)
        /go/src/github.com/microsoft/frameworkcontroller/pkg/controller/controller.go:428 +0x157
main.main()
        /go/src/github.com/microsoft/frameworkcontroller/cmd/frameworkcontroller/main.go:35 +0x47

I try to set up nni frameworkcontroller.

When run “kubectl logs frameworkcontroller”, I got this message.