Manually created ServiceAccount tokens "expiring" unexpectedly

General Discussions
, ,
1

Cluster information:

Kubernetes version:

Client Version: v1.29.9-dbx2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.9-dbx2

Cloud being used: bare-metal
Installation method: hand-rolled deployment platform
Host OS: Linux -- 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

I’ve created a ServiceAccount, a ClusterRole and a ClusterRoleBinding between them.

I’ve also created a long-lived token like so

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: <some_token_name>
  annotations:
    kubernetes.io/service-account.name: <some_sa>
type: kubernetes.io/service-account-token
EOF

This token only seems to last 18-20 days. After which, when I try to curl it returns a 401:

TOKEN=$(kubectl get secret <my_secret_token_name> -n default -o jsonpath='{.data.token}' | base64 --decode)

APISERVER=https://my.api.server:5000

curl -k -H "Authorization: Bearer $TOKEN" $APISERVER/api/v1/namespaces/default/pods

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

At this point, when I recreate another token like so:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: <some_new_token_name>
  annotations:
    kubernetes.io/service-account.name: <some_sa>
type: kubernetes.io/service-account-token
EOF

And curl with the above, I get a valid response. Obviously the expected behaviour is such that the first token never expires or becomes invalidated.

Things I’ve ruled out (preliminarily):

  • Admission Controller – I’m not running any funky plugins that might be filtering these requests
  • etcd defrag – the fact that I can kubectl get secrets on the “invalid” token means it still exists (and also the ServiceAccount was never deleted/recreated)
  • Some weird spill-over behaviour from token auto cleanup – I checked the labels/annotations of the tokens and while there is this label kubernetes.io/legacy-token-last-used, docs indicate this doesn’t apply to manually generated tokens

What could be the culprit here?

2

Hi,
Try decoding data.token field and check its structure. It should not have the exp field.