Wrong "exp" date in projected serviceaccount token

General Discussions
1

Cluster information:

Kubernetes version: 1.21
Cloud being used: Amazon EKS
Installation method: Pulumi
Host OS: amazon-eks-node-1.21-v20211004 (AMI ID: ami-0c073420a05707690
)

I’m using AWS EKS 1.21 with service account discovery enabled (it’s enabled by default by EKS, I can’t change it).
The .well-known/openid-configuration endpoint returns:

{
  "issuer": "https://oidc.eks.eu-west-1.amazonaws.com/id/***",
  "jwks_uri": "https://ip-***.eu-west-1.compute.internal:443/openid/v1/jwks",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}

Created a ServiceAccount for one of my deployments and the pod gets this as projected volume.

  volumes:
  - name: kube-api-access-b4xt9
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

The projected token mounted into the pod contains this:

{
  "aud": [
    "https://kubernetes.default.svc"
  ],
  "exp": 1664448004,
  "iat": 1632912004,
  "iss": "https://oidc.eks.eu-west-1.amazonaws.com/id/***",
  "kubernetes.io": {
    "namespace": "sbx",
    "pod": {
      "name": "dliver-site-config-service-77494b8fdd-45pxw",
      "uid": "0dd440a6-1213-4faa-a69e-398b83d2dd6b"
    },
    "serviceaccount": {
      "name": "dliver-site-config-service",
      "uid": "c26ad760-9067-4d90-a327-b3d6e32bce42"
    },
    "warnafter": 1632915611
  },
  "nbf": 1632912004,
  "sub": "system:serviceaccount:sbx:dliver-site-config-service"
}

Kubernetes renew the projected token every hour, so everything looks fine.
Except the projected token “exp” field:
"iat": 1632912004 which is Wednesday, September 29, 2021 10:40:04 AM
"exp": 1664448004 which is Thursday, September 29, 2022 10:40:04 AM

So the problem is, that the projected token expiry time is 1 year, instead of around 1 hour.

If I set automountServiceAccountToken: false and configuring the volume manually to the same as above, the token exp will be the same, 1 year.
But when I change expirationSeconds to another value, e.g.: 3606, 3608, 3600, 3700, etc, then the token exp will be correct, based on the expirationSeconds, so it will be around 1 hour later than iat.

Looks likes something magical happens, when the value is the default 3607, but I can’t find any related information or issue, so I don’t really know it’s a kubernetes, EKS or OIDC issue.
I assume it’s kubernetes or EKS, because the projected token and the default expirationSeconds value provided by them and the exp is correct in any other case, so the OIDC provides should work fine.

Any idea why is this happening?

2

Finally got an answer elsewhere.

cluster operators can specify flag --service-account-extend-token-expiration=true to kube apiserver to allow tokens have longer expiration temporarily during the migration. Any usage of legacy token will be recorded in both metrics and audit logs.

The “3607” magic number is part of the Bound Service Account Tokens safe rollout plan, described in this kep.
The actual number hardcoded in the source code.
The --service-account-extend-token-expiration flag was set to true by default from 1.20.

The mentioned metric/log info can be found in the kep too and was implemented here.
To see these logs in EKS need to enable audit logging on the cluster, then check Cloudwatch for related log entries.

I used this query in Cloudwatch Log Insight to find which pods don’t reload the token periodically:

filter @logStream like 'kube-apiserver-audit'
 | filter ispresent(`annotations.authentication.k8s.io/stale-token`)
 | parse `annotations.authentication.k8s.io/stale-token` "subject: *," as subject
 | stats count(*) as staleCount by subject, `user.username`
 | sort staleCount desc