I try to install fluentd with helm in k8s. But when i install, error occurs ‘Could not communicate to Elasticsearch, resetting connection and trying again. EOFError (EOFError)’. All k8s object related elasticsearch and fluentd is in same namespace efk.
curl to Elasticsearch cluster in curlimage pod in same namespace
$ curl --cacert ca.crt -u elastic:xxxx https://elasticsearch-master:9200
{
"name" : "elasticsearch-master-0",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "3VGn0y_bRDmwWLj-jGgq5g",
"version" : {
"number" : "8.5.1",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "c1310c45fc534583afe2c1c03046491efba2bba2",
"build_date" : "2022-11-09T21:02:20.169855900Z",
"build_snapshot" : false,
"lucene_version" : "9.4.1",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
fluentd values.yaml
kind: "Deployment"
# variant: elasticsearch7 # >> default
replicaCount: 1
service:
ports:
- name: "forwarder"
protocol: TCP
containerPort: 24224
volumes:
- name: elasticsearch-certs
secret:
secretName: elasticsearch-master-certs
defaultMode: 0777
volumeMounts:
- name: elasticsearch-certs
mountPath: /etc/fluent/certs
readOnly: true
fileConfigs:
01_sources.conf: |-
<source>
@type forward
@label @KUBERNETES
bind 0.0.0.0
port 24224
@log_level debug
</source>
02_filters.conf: |-
<label @KUBERNETES>
<match **>
@type relabel
@label @OUTPUT
</match>
</label>
04_outputs.conf: |-
<label @OUTPUT>
<match **>
@type elasticsearch
host elasticsearch-master
schema https
port 9200
path ""
user elastic
password u6g7fHaIM1JDth1m
ca_file /etc/fluent/certs/ca.crt
client_cert /etc/fluent/certs/tls.crt
client_key /etc/fluent/certs/tls.key
@log_level debug
</match>
</label>
fluentd logs
2023-10-05 07:36:32 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-10-05 07:36:32 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/../../../etc/fluent/fluent.conf"
2023-10-05 07:36:35 +0000 [info]: gem 'fluentd' version '1.16.2'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.5'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.2.0'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-10-05 07:36:35 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-10-05 07:36:35 +0000 [debug]: 'host elasticsearch-master' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master' doesn't have tag placeholder
2023-10-05 07:36:35 +0000 [debug]: 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2023-10-05 07:36:35 +0000 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-10-05 07:36:35 +0000 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2023-10-05 07:36:35 +0000 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2023-10-05 07:36:35 +0000 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-10-05 07:36:35 +0000 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-10-05 07:36:35 +0000 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-10-05 07:36:35 +0000 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-10-05 07:36:35 +0000 [debug]: Need substitution: false
2023-10-05 07:36:35 +0000 [debug]: 'host_placeholder elasticsearch-master' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master' doesn't have tag placeholder
2023-10-05 07:36:35 +0000 [info]: using configuration file: <ROOT>
<label @FLUENT_LOG>
<match **>
@type null
@id ignore_fluent_logs
</match>
</label>
<source>
@type forward
@label @KUBERNETES
bind "0.0.0.0"
port 24224
@log_level "debug"
</source>
<label @KUBERNETES>
<match **>
@type relabel
@label @OUTPUT
</match>
</label>
<label @DISPATCH>
<filter **>
@type prometheus
<metric>
name fluentd_input_status_num_records_total
type counter
desc The total number of incoming records
<labels>
tag ${tag}
hostname ${hostname}
</labels>
</metric>
</filter>
<match **>
@type relabel
@label @OUTPUT
</match>
</label>
<label @OUTPUT>
<match **>
@type elasticsearch
host "elasticsearch-master"
schema https
port 9200
path ""
user "elastic"
password xxxxxx
ca_file "/etc/fluent/certs/ca.crt"
client_cert "/etc/fluent/certs/tls.crt"
client_key "/etc/fluent/certs/tls.key"
@log_level "debug"
</match>
</label>
</ROOT>
2023-10-05 07:36:35 +0000 [info]: starting fluentd-1.16.2 pid=7 ruby="3.1.4"
2023-10-05 07:36:36 +0000 [info]: spawn command to main: cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd", "-c", "/fluentd/etc/../../../etc/fluent/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "-r", "/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.2.5/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2023-10-05 07:36:37 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-10-05 07:36:37 +0000 [info]: adding match in @FLUENT_LOG pattern="**" type="null"
2023-10-05 07:36:37 +0000 [info]: adding match in @KUBERNETES pattern="**" type="relabel"
2023-10-05 07:36:37 +0000 [info]: adding filter in @DISPATCH pattern="**" type="prometheus"
2023-10-05 07:36:37 +0000 [info]: adding match in @DISPATCH pattern="**" type="relabel"
2023-10-05 07:36:37 +0000 [info]: adding match in @OUTPUT pattern="**" type="elasticsearch"
2023-10-05 07:36:37 +0000 [debug]: #0 'host elasticsearch-master' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master' doesn't have tag placeholder
2023-10-05 07:36:37 +0000 [debug]: #0 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2023-10-05 07:36:37 +0000 [debug]: #0 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-10-05 07:36:37 +0000 [debug]: #0 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2023-10-05 07:36:37 +0000 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2023-10-05 07:36:37 +0000 [debug]: #0 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-10-05 07:36:37 +0000 [debug]: #0 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-10-05 07:36:37 +0000 [debug]: #0 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-10-05 07:36:37 +0000 [debug]: #0 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-10-05 07:36:37 +0000 [debug]: #0 Need substitution: false
2023-10-05 07:36:37 +0000 [debug]: #0 'host_placeholder elasticsearch-master' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master' doesn't have tag placeholder
2023-10-05 07:36:39 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. EOFError (EOFError)
2023-10-05 07:36:39 +0000 [warn]: #0 Remaining retry: 14. Retry to communicate after 2 second(s).
2023-10-05 07:36:42 +0000 [info]: Received graceful stop
2023-10-05 07:36:43 +0000 [warn]: #0 Could not communicate to Elasticsearch, resetting connection and trying again. EOFError (EOFError)
2023-10-05 07:36:43 +0000 [warn]: #0 Remaining retry: 13. Retry to communicate after 4 second(s).
I am using the self-signed certificate generated by Elastic Search helm chart, so I set the ca, but I think I did something wrong there.
I accessed the Pod and read the mounted certificate file and found that it exists.
When I modify variant in fluentd values.yaml to elasticsearch8, error logs like below and then repeat:
2023-10-05 07:43:39 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-10-05 07:43:39 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/../../../etc/fluent/fluent.conf"
2023-10-05 07:43:41 +0000 [info]: gem 'fluentd' version '1.16.2'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.3.0'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '3.2.0'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-10-05 07:43:41 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-10-05 07:43:41 +0000 [debug]: 'host elasticsearch-master' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: elasticsearch-master' doesn't have tag placeholder
2023-10-05 07:43:41 +0000 [debug]: 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2023-10-05 07:43:41 +0000 [debug]: 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2023-10-05 07:43:41 +0000 [debug]: 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2023-10-05 07:43:41 +0000 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' has timestamp placeholders, but chunk key 'time' is not configured
2023-10-05 07:43:41 +0000 [debug]: 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2023-10-05 07:43:41 +0000 [debug]: 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2023-10-05 07:43:41 +0000 [debug]: 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2023-10-05 07:43:41 +0000 [debug]: 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2023-10-05 07:43:41 +0000 [debug]: Need substitution: false
2023-10-05 07:43:41 +0000 [debug]: 'host_placeholder elasticsearch-master' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: elasticsearch-master' doesn't have tag placeholder
2023-10-05 07:43:41 +0000 [info]: using configuration file: <ROOT>
<label @FLUENT_LOG>
<match **>
@type null
@id ignore_fluent_logs
</match>
</label>
<source>