CM pod is fluctuating in k8s

General Discussions
1

Asking for help? Comment out what you need so we can get more information to help you!

Cluster information:

Kubernetes version:5.6.2
Cloud being used: (put bare-metal if not on a public cloud)
Installation method: On premises
Host OS: Linux
Error in CM POD

More Information:

Events:
Type Reason Age From Message

Warning Unhealthy 2m17s (x123 over 150d) kubelet Readiness probe failed: command “/usr/local/bin/tml-event-handler.sh isRegistryReady” timed out

Error in Logs:

Failed to get D-Bus connection: Operation not permitted
Starting lighttpd: [ OK ]
Container is build for restricted with UID 1001
Running container with UID 1001 for user restricted …
Using UID 1001 for user restricted
Failed to get D-Bus connection: Operation not permitted
Starting cluster-manager-svc: [ OK ]
Registry address [cass-svc-0.mashery.svc.cluster.local:8080] is not reachable, trying again. Err [dial tcp: i/o timeout]
Registry address [cass-svc-0.mashery.svc.cluster.local:8080] is not reachable, trying again. Err [dial tcp: i/o timeout]
Registry address [cass-svc-0.mashery.svc.cluster.local:8080] is not reachable, trying again. Err [dial tcp: i/o timeout]
Registry address [cass-svc-0.mashery.svc.cluster.local:8080] is not reachable, trying again. Err [dial tcp: i/o timeout]
Registry address [cass-svc-0.mashery.svc.cluster.local:8080] is not reachable, trying again. Err [dial tcp: i/o timeout]
Registry address [cass-svc-0.mashery.svc.cluster.local:8080] is not reachable, check the network. Quitting the CLI process
containeragent is stopped
Starting containeragent:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/mashery/containeragent/lib/tpcl/ch.qos.logback.classic_1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/mashery/containeragent/lib/tpcl/ch.qos.logback.slf4j_1.2.3.v20200428-2012.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See SLF4J Error Codes for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/mashery/containeragent/lib/tpcl/com.google.inject_4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte,int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

2

Pod Unhealthy
The pod’s readiness probe is failing because the tml-event-handler.sh script cannot verify if the registry is ready.

D-Bus Error

The application is unable to get a D-Bus connection. The error message, “Operation not permitted,” indicates a permissions issue.

Connectivity Issues

The pod is failing to connect to cass-svc-0.mashery.svc.cluster.local:8080, resulting in an I/O timeout. This suggests a problem with network connectivity to the specified service.

Security Context

The pod is running as a restricted user with UID 1001. This security context may be causing some of the other issues, such as the D-Bus error, due to a lack of necessary permissions.

Agent Termination

The containeragent is repeatedly stopping and then attempting to restart. This indicates a potential stability problem or a fatal error within the agent itself.