How to connect a stateful service from logstash?

General Discussions
1

I want to use one logstash send data to multiple logstash services.

If I use this setting in k8s, it can work.

service.yml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: logstash
  name: logstash
spec:
  selector:
    app: logstash
  ports:
    - name: "5044"
      port: 5044
      targetPort: 5044

filebeat.yml

    filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - /var/log/log1.json
        tags: ["log1"]

    processors:
      - decode_json_fields:
          fields: ["message"]
          process_array: true
          max_depth: 1
          target: ""
          overwrite_keys: true
          add_error_key: false

    output.logstash:
      hosts: ["logstash-log1.default.svc.cluster.local:5044"]

logstash.yml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-logstash
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
    xpack.monitoring.enabled: false
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }

    filter {
    }

    output {
      stdout { codec => rubydebug }
    }

But if I change to another way, set output from another logstash instance as:

k8s services:

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: logstash-log1
  name: logstash-log1
spec:
  selector:
    app: logstash-log1
  ports:
    - name: "5044"
      port: 5044
      targetPort: 5044

logstash.yml

    output {
      if "log1" in [tags] {
        stdout { codec => rubydebug }
        tcp {
          mode => "client"
          host => "logstash-log1.default.svc.cluster.local"
          port => 5044
          codec => "json"
        }
      }
    }

target logstash’s logstash.yml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: configmap-logstash
data:
  logstash.yml: |
    http.host: "0.0.0.0"
    path.config: /usr/share/logstash/pipeline
    xpack.monitoring.enabled: false
  logstash.conf: |
    input {
      tcp {
        port => 5044
      }
    }

    filter {
    }

    output {
      stdout { codec => rubydebug }
    }

The communication can be confimed, it works. But the data seems didn’t send to logstash-log1.default.svc.cluster.local:5044.

If use curl to test from origin logstash,

curl logstash-log1.default.svc.cluster.local:5044

Can get log data in the target logstash:

{
  "port" => 37424,
  "host" => "10.12.8.213"
}
{
  "port" => 37424,
  "host" => "10.12.8.213"
}
{
  "port" => 37424,
  "host" => "10.12.8.213"
}
{
  "port" => 37424,
  "host" => "10.12.8.213"
}

So the problem may be the configuration between 2 logstash. Doesn’t tcp input/output methods work?