Fluentd in Co-Work
•
3 likes•9,349 views
Fluentd is a data collector that can unify logging and metrics formats and enable real-time extraction, transformation, and storage of data. It will be used at 10xLab to collect logging data from their Co-Work app and infrastructure components and enable real-time analysis and long-term storage. Fluentd makes it easy to set up log collection pipelines and extend functionality through plugins. 10xLab plans to use Fluentd with Resque to reliably queue and process job data, store logs in S3, analyze logs in Treasure Data, and monitor systems. Fluentd will be installed via AWS cloud-init and managed using Chef.
Fluentd in Co-Work
- 1. Fluentd in Fluentd meetup in Fukuoka 2013/03/07 @Spring_MT
- 2. Profile twitter : @Spring_MT Company : 10xLab Engineer
- 3. Agenda Fluentd in Co-Work app in Co-Work infra
- 5. SNS for Companies (like Yammer) During Closed β
- 8. 10xlab Infra ALL AWS! Use VPC Ruby, Fluentd and Chef in all servers gem install fluentd
- 9. f fluentd public subnet net work Gateway NAT SSLの変換 subnet subnet subnet utility subnet fluentd storage Co-Work deliver proxy APP Watch f f redis f f f worker varnish f DB(MySQL on EC2) f + sinatra f f f worker f f f f
- 10. Co-Work WAF Ruby on Rails DB MySQL on EC2 KVS redis
- 11. Fluentd in Co-Work App
- 13. Queuing
- 14. Queuing Resque Ruby based job queue liblary using github Multiprocessing Backend is redis https://github.com/defunkt/resque
- 15. fluent-plugin-resque Fluentd output plugin for sending resque- queue to redis https://github.com/hotchpotch/fluent- plugin-resque
- 16. Why Fluentd?
- 17. worker worker app worker worker r r queue e e worker worker worker s s app worker q q worker u u queue e e worker worker worker app worker worker
- 18. worker worker app worker worker X If queing fail, App should retry queuing. You write a retry processing?
- 19. worker worker app worker f f worker l l u u e e r queue n n e worker worker t t worker - - s app worker l r q worker o e u queue g s g q e worker worker e u worker r e app worker worker
- 20. worker app worker worker X X worker If queuing fail, Fluentd retry queuing ! Fluentd(and Logger) Buffering internally Retrying
- 21. Send Queue Data to Fluentd
- 23. Send Resque- Queue to Redis
- 26. Logging
- 27. Logging Access log(nginx) PV log(rails) Ref. WEB†DB PRESS vol. App Log(rails)
- 28. Why Fluentd?
- 29. Log File Transfer Forwarding server is full m9(^Д^)ﾌﾟｷﾞｬｰ Operate transfer script? Real-Time analysis ? Delete log file
- 30. Log File Format and Analysis Disjointed format (CSV? TSV? LTSV? JSON?) Operate analysis script?
- 31. Fluentd Unified Format(JSON) Real-Time transfer and High Availability Easy Setup and Easy extension Well Tested At Big services(○HN, ○ookpad)
- 32. Just Do It ! http://d.hatena.ne.jp/tagomoris/20121029/1351491111
- 33. Just Do It ! し ょ 今 で http://d.hatena.ne.jp/tagomoris/20121029/1351491111
- 34. access proxy log in_tail deliver S3 in_tail PV log Growth app app log logger Forecast
- 35. Analysis
- 36. copy out_route fluent-plugin-rewrite-tag-filter analysis fluent-plugin-datacounter fluent-plugin-numeric-counter fluent-plugin-numeric-monitor fluent-plugin-amplifier-filter
- 37. Output
- 38. GrowthForecast fluent-plugin-growthforecast For visualize data GrowthForecast Web API for drawing graph Perl
- 41. Treasure Data fluent-plugin-td For data analysis (mainly access log) TreasureData Hadoop-based data store and analysis service
- 43. S3 fluent-plugin-s3 For backup logs S3 Data Store Service in AWS Eleven Nine
- 44. ikachan fluent-plugin-ikachan IRC notification(Ex.fatal log) ikachan Web API for IRC Perl
- 46. Analysis in TD SQL Like 500 GB ($299/M) Analysis using Ruby!
- 50. Collect Lib
- 51. Fluentd in Co-Work Infra
- 52. Watch For resource watch Fluentd collect data and push data each notification tools.
- 53. fluent-plugin-mysqlslowquery Collect mysql slow query log Notify IRC through fluent-plugin-ikachan + ikachan
- 55. fluent-resque-stat Collect Resque status pending キューの残数 processed 取り出したキューの総数 queues 登録されてるキューの名前の総数 workers キューを取り出すworkerの数 working キューを処理中のworker数 failed 失敗したキュー数 servers resqueが動いてるサーバーのhostやportの情報 resque_statが動いているサーバーの RAILS_ENV or environment RACK_ENV(なければdevelopmentが入る)
- 57. Thank You !
- 58. Fluentd Setup
- 59. Install and Setup Installattion of Fluentd is AWS cloud- init(user_data) Setting Fluentd is chef-sole and capistrano
- 60. I Extend a little above system for specifying cookbook. $ cap -S subnet=fluentd -S hosts=xfdeliver001 chef
- 61. Launch and Monitoring Process supervisor