Sidecar communication

General Discussions
1

Hello All, I’m currently attempting to deploy the JMX exporter alongside my Java application (Tomcat), either as a sidecar or as separate pods with the Java application in one pod and the JMX exporter in another. However, I’ve encountered an error message after trying to deploy my application as a sidecar.

As a newcomer to Kubernetes with limited knowledge, I’m reaching out to our community for assistance in resolving this issue. My goal is to monitor the Java application pod and retrieve metrics from it. Any help would be greatly appreciated.

Appliaction Yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
name: gitomcat-deployment
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: gitomcat
template:
metadata:
labels:
app: gitomcat
spec:
containers:
- name: gitomcat-container
image: tomcat:9-jdk8
ports:
- containerPort: 2081
env:
- name: CATALINA_OPTS
value: “-javaagent:/usr/local/jmx/jmx_prometheus_javaagent-0.20.0.jar =8088:/usr/local/jmx/conf/prometheus-jmx-config.yaml”
volumeMounts:
- name: local-gi
mountPath: /Apps/GameCode/GI
- name: agent-volume
mountPath: /usr/local/jmx
- name: jmx-prometheus-javaagent
mountPath: “/Apps/GameCode/GI/jmx_prometheus_javaagent-0.20.0.jar”
- name: prometheus-jmx-config
mountPath: “/usr/local/jmx/conf”

        # - name: local-gi
        #   mountPath: /usr/local/tomcat/bin/catalina.sh
        # # - name: local-gi
        # #   mountPath: /usr/local/tomcat/conf/context.xml
        # # - name: local-gi
        # #   mountPath: /usr/local/tomcat/conf/server.xml
        - name: catalina-log
          mountPath: /usr/local/tomcat/logs
    - name: catalina-log
      image: busybox:1.28
      args:
        - "/bin/sh"
        - "-c"
        - "sleep 20; tail -n+1 -F /usr/local/tomcat/logs/catalina.out"
      volumeMounts:
        - name: catalina-log
          mountPath: /usr/local/tomcat/logs
    - name: fluent-bit
      image: public.ecr.aws/aws-observability/aws-for-fluent-bit:stable
      volumeMounts:
        - name: config-volume
          mountPath: /fluent-bit/etc
        - name: catalina-log
          mountPath: /usr/local/tomcat/logs
    # - name: jmx-exporter
    #   image: public.ecr.aws/bitnami/jmx-exporter:0.20.0
    #   ports:
    #     - name: metrics
    #       containerPort: 8081
    #   volumeMounts:
    #     - name: jmx-config
    #       mountPath: /opt/jmx-exporter/config
  volumes:
      - name: catalina-log
        emptyDir: {}
      - name: local-gi
        hostPath:
          path: /Users/pavanreddy/Desktop/GI/
      - name: config-volume
        configMap:
          name: fluent-bit-config
      - name: jmx-config
        configMap:
          name: jmx-config
      - name: agent-volume
        configMap:
          name: jmx-javaagent-configmap
      - name: jmx-prometheus-javaagent
        hostPath:
          path: /Users/pavanreddy/Desktop/jmx_prometheus_javaagent-0.20.0.jar 
      - configMap: 
          name: prometheus-jmx-config
        name: prometheus-jmx-config

Error message:

kubectl logs gitomcat-deployment-7697b5c994-dx99l
Defaulted container “gitomcat-container” out of: gitomcat-container, catalina-log, fluent-bit
Exception in thread “main” java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.NullPointerException
at java.util.regex.Matcher.getTextLength(Matcher.java:1283)
at java.util.regex.Matcher.reset(Matcher.java:309)
at java.util.regex.Matcher.(Matcher.java:229)
at java.util.regex.Pattern.matcher(Pattern.java:1093)
at io.prometheus.jmx.JavaAgent.parseConfig(JavaAgent.java:89)
at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:52)
… 6 more
FATAL ERROR in native method: processing of -javaagent failed

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f445e1ff898, pid=1, tid=0x00007f445e1d6640

JRE version: OpenJDK Runtime Environment (8.0_402-b06) (build 1.8.0_402-b06)

Java VM: OpenJDK 64-Bit Server VM (25.402-b06 mixed mode linux-amd64 compressed oops)

Problematic frame:

C [libc.so.6+0x28898] abort+0x178

Core dump written. Default location: /usr/local/tomcat/core or core.1

An error report file with more information is saved as:

/usr/local/tomcat/hs_err_pid1.log

If you would like to submit a bug report, please visit:

Issues · adoptium/adoptium-support · GitHub

I’ve attempted various troubleshooting methods, such as pulling the image from the repository and testing it successfully. I can pull the images from Docker Hub without any issues. However, when I try to run it as a sidecar, I encounter errors.

If anyone has experience with and has successfully implemented this setup in their environment, I would greatly appreciate your assistance and knowledge sharing.