Latest Post

Fluentd error: Unable to push logs to [elasticsearch]

After application deployments, Kibana stopped showing logs exactly after 7 days. The error "Fluentd error: Unable to push logs to [elasticsearch]" was shown in the fluentd logs. The initial response was to increase the buffer limits for fluentd as follows:

chunk_limit_size 10Mqueue_limit_length 256
The behavior occurred again after two weeks, which led to the same error. On closer investigation, the error was preceded by the statement "Failed to write to the buffer." This led me to inspect the fluentd configuration again and found the following code in the buffer part which caused the fluentd buffers to be filled as per the official documentation on Fluentd :
overflow_action blockThe fix for this overflow_action is to change from block to drop_oldest_chunk, allowing the fluentd logs to flow seamlessly to the elastic search by dropping the oldest logs in the buffer.
  <buffer> @type file path /var/log/fluentd-buffers/kubernetes.system.buffer flush_mode inter…

On-board Linux computers to Azure Log Analytics

For on-boarding linux servers to Azure log analytics, just execute the command on the respective server:

wget https://raw.githubusercontent.com/Microsoft/OMS-Agent-for-Linux/master/installer/scripts/onboard_agent.sh && sh onboard_agent.sh -p [protocol://][user:password@]proxyhost[:port] -w <YOUR WORKSPACE ID> -s <YOUR WORKSPACE PRIMARY KEY>

In few cases like mine, the servers will not have access to internet and we will need to install the downloaded oms agent package with the proxy information. 

sh omsagent-1.6.0-42.universal.x64.sh --install -w <YOUR WORKSPACE ID> -s <YOUR WORKSPACE PRIMARY KEY> -p [protocol://][user:password@]proxyhost[:port]

The proxy can have username and password for authentication and even if no proxy authentication is required, we will need to enter a dummy username and password which in the example below is azure and azure.

sh omsagent-1.6.0-42.universal.x64.sh --install -w <YOUR WORKSPACE ID> -s <YOUR WORKSPACE PRIMARY KEY> -p http://azure:azure@proxyhost[:port]

If you have SCOM agent already installed then instead of --install we will need to specify --upgrade.

sh omsagent-1.6.0-42.universal.x64.sh --upgrade -w <YOUR WORKSPACE ID> -s <YOUR WORKSPACE PRIMARY KEY> -p http://azure:azure@proxyhost[:port]

Comments

Post a Comment

Popular posts from this blog

Nutanix Calm Blueprint for Single Linux VM

Fluentd error: Unable to push logs to [elasticsearch]