Cluster information:
Kubernetes version: v1.23.16-eks-48e63af
Cloud being used: AWS
Installation method: EKS
Host OS: Linux/UNIX
CNI and version: N/A
CRI and version: N/A
Hello, guys. I have one question that want to ask.
I have tried to find how much time container get from start to ready status and found a problem that can’t solve.
Example response from k8s API by python3:
'status': {'conditions': [{'last_probe_time': None,
'last_transition_time': datetime.datetime(2023, 1, 21, 18, 4, 26, tzinfo=tzutc()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Initialized'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2023, 2, 21, 5, 12, 56, tzinfo=tzutc()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Ready'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2023, 2, 21, 5, 12, 56, tzinfo=tzutc()),
'message': None,
'reason': None,
'status': 'True',
'type': 'ContainersReady'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2023, 1, 21, 18, 4, 26, tzinfo=tzutc()),
'message': None,
'reason': None,
'status': 'True',
'type': 'PodScheduled'}],
'container_statuses': [{'container_id': <containerID>,
'image': <image>,
'image_id': <imageID>,
'last_state': {'running': None,
'terminated': {'container_id': <containerID>,
'exit_code': 137,
'finished_at': datetime.datetime(2023, 2, 11, 17, 47, 13, tzinfo=tzutc()),
'message': None,
'reason': 'OOMKilled',
'signal': None,
'started_at': datetime.datetime(2023, 1, 21, 18, 4, 28, tzinfo=tzutc())},
'waiting': None},
'name': <name>,
'ready': True,
'restart_count': 1,
'started': True,
'state': {'running': {'started_at': datetime.datetime(2023, 2, 11, 17, 47, 14, tzinfo=tzutc())},
'terminated': None,
'waiting': None}}],
In this example, container started at 21 JAN 2023 18:04:28 (datetime.datetime(2023, 1, 21, 18, 4, 28, tzinfo=tzutc())), then it restarted at 11 FEB 2023 17:47:14 ({‘started_at’: datetime.datetime(2023, 2, 11, 17, 47, 14, tzinfo=tzutc())}), but “ContainersReady” condition was at 21 FEB 2023 05:12:56 (datetime.datetime(2023, 2, 21, 5, 12, 56, tzinfo=tzutc())). As I found here kubernetes/util.go at master · kubernetes/kubernetes · GitHub condition can be updated during life of pod. And I think, problem is that container had one failed request for Liveness Probe (to restart it needs 3 failed request) and didn’t restart, but updated condition “ContainersReady” time.
How I can find starttime for this container? Is it possible or not?
Also I have pods with “started_at” and “ContainersReady” time the same. How it even possible? because app starting 20-30 seconds.
'status': {'conditions': [{'last_probe_time': None,
'last_transition_time': datetime.datetime(2023, 3, 27, 3, 15, 29, tzinfo=tzutc()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Initialized'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2023, 3, 27, 3, 18, 7, tzinfo=tzutc()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Ready'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2023, 3, 27, 3, 18, 7, tzinfo=tzutc()),
'message': None,
'reason': None,
'status': 'True',
'type': 'ContainersReady'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2023, 3, 27, 3, 15, 29, tzinfo=tzutc()),
'message': None,
'reason': None,
'status': 'True',
'type': 'PodScheduled'}],
'container_statuses': [{'container_id': <containerID>,
'image': <image>,
'image_id': <imageID>,
'last_state': {'running': None,
'terminated': None,
'waiting': None},
'name': <name>,
'ready': True,
'restart_count': 0,
'started': True,
'state': {'running': {'started_at': datetime.datetime(2023, 3, 27, 3, 18, 7, tzinfo=tzutc())},
'terminated': None,
'waiting': None}}],