Fluentd error: Unable to push logs to [elasticsearch]
After application deployments, Kibana stopped showing logs exactly after 7 days. The error "Fluentd error: Unable to push logs to [elasticsearch]" was shown in the fluentd logs. The initial response was to increase the buffer limits for fluentd as follows:
chunk_limit_size 10M queue_limit_length 256
The behavior occurred again after two weeks, which led to the same error. On closer investigation, the error was preceded by the statement "Failed to write to the buffer." This led me to inspect the fluentd configuration again and found the following code in the buffer part which caused the fluentd buffers to be filled as per the official documentation on Fluentd :
The fix for this overflow_action is to change from block to drop_oldest_chunk, allowing the fluentd logs to flow seamlessly to the elastic search by dropping the oldest logs in the buffer.
<buffer> @type file path /var/log/fluentd-buffers/kubernetes.system.buffer flush_mode interval retry_type exponential_backoff flush_thread_count 2 flush_interval 5s retry_forever retry_max_interval 30 chunk_limit_size 10M queue_limit_length 256 overflow_action drop_oldest_chunk </buffer>