td-agent(1.1.21)をEC2インスタンスにインストールし、fluent-plugin-s3でログをS3に送ろうとしたら、ログに以下のようなエラーが吐き出されていました。
2015-01-19 15:46:58 +0900 [error]: failed to configure/start sub output s3: aws_key_id or aws_sec_key is invalid. Please check your configuration
2015-01-19 15:46:58 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.4.1/lib/fluent/plugin/out_s3.rb:190:in `rescue in check_apikeys'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.4.1/lib/fluent/plugin/out_s3.rb:188:in `check_apikeys'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.4.1/lib/fluent/plugin/out_s3.rb:111:in `start'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:133:in `block in plant'
<internal:prelude>:10:in `synchronize'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:33:in `next'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/buffer.rb:172:in `block in emit'
/usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/buffer.rb:168:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:427:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:33:in `next'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/out_copy.rb:73:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/match.rb:36:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/engine.rb:160:in `emit_stream'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:227:in `receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `wrap_receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:343:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `on_change'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run_once'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:212:in `run'
2015-01-19 15:46:58 +0900 [error]: Cannot output messages with tag 'foo.syslog.secure'
2015-01-19 15:47:28 +0900 [error]: failed to configure sub output s3: Other '' plugin already use same buffer_path: type = , buffer_path = /var/log/td-agent/buffer/s3-foo.syslog.secure.*.buffer
2015-01-19 15:47:28 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/buf_file.rb:94:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:188:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:471:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.4.1/lib/fluent/plugin/out_s3.rb:47:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
<internal:prelude>:10:in `synchronize'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:33:in `next'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/buffer.rb:172:in `block in emit'
/usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/buffer.rb:168:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:427:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:33:in `next'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/out_copy.rb:73:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/match.rb:36:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/engine.rb:160:in `emit_stream'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:227:in `receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `wrap_receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:343:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `on_change'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run_once'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:212:in `run'
各種パッケージのバージョンは以下の通りです。
- td-agent 1.1.21
- fluentd 0.10.55
- fluent-plugin-s3 0.4.1
以下のように設定していました。
# /etc/td-agent/td-agent.conf
<match **>
<store>
type forest
subtype s3
<template>
log_level info
aws_key_id xxxxxxxxxxxxxx
aws_sec_key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
s3_bucket foo_bucket
s3_endpoint s3-ap-northeast-1.amazonaws.com
s3_region ap-northeast-1
# S3バケットの保存パスプレフィックス
# e.g. logs/nginx.access/2015/01/access-
path logs/${tag_parts[0]}/%Y/%m/${tag_parts[1]}-
# バッファのログを格納するパスプレフィックス
buffer_path /var/log/td-agent/buffer/s3-${tag}.*.buffer
# formatで指定した時間毎にログファイルをS3へ格納する
time_slice_format %Y%m%d
time_slice_wait 10m
# 指定したサイズに達するとログを格納する
buffer_chunk_limit 500m
# バッファの最大キュー数
buffer_queue_limit 2
flush_at_shutdown true
# 出力するログフォーマット
format json
# jsonログに時間を入れる
include_time_key true
# 時間のタグ名
time_key time
# jsonログにタグを入れる
include_tag_key true
# "タグ"のタグ名
tag_key tag
# 時刻をUTCにする
utc
</template>
</store>
<store>
# Elasticsearchなどの設定
...
</store>
</match>
調べたところ、以下の記事が今回の事象と一致していました。
nginxのaccess.logをfluentdでS3に格納するまで
今回の場合、ログ格納先のS3バケット(foo_bucket)にはすでに他のログファイルなどが格納していました。
fluent-plugin-s3のソースを見ると、S3のバケットの中身が存在する場合にエラーになるようでした。
# https://github.com/fluent/fluent-plugin-s3/blob/v0.4.1/lib/fluent/plugin/out_s3.rb#L187
def check_apikeys
@bucket.empty?
rescue
raise "aws_key_id or aws_sec_key is invalid. Please check your configuration"
end
0.5.1で対応していますが、td-agentというパッケージでインストールしてしまっているのと、Chefからインストールしているため、わざわざs3プラグインのみ単体でアップデートするのが大変かなと思ったので今回はbar_bucketなどのような空のS3バケットを新規作成することで対応しました。
最後に
td-agentでお手軽にfluentdなどのパッケージがインストールできて便利ですが、最新バージョンに反映されるのに時間がかかるので、fluentd単体をインストールしても良いかもしれませんね。
