|
| 1 | +A Python structured logger for Fluentd |
| 2 | +====================================== |
| 3 | + |
| 4 | +.. image:: https://travis-ci.org/fluent/fluent-logger-python.svg?branch=master |
| 5 | + :target: https://travis-ci.org/fluent/fluent-logger-python |
| 6 | + :alt: Build Status |
| 7 | + |
| 8 | +.. image:: https://coveralls.io/repos/fluent/fluent-logger-python/badge.svg |
| 9 | + :target: https://coveralls.io/r/fluent/fluent-logger-python |
| 10 | + :alt: Coverage Status |
| 11 | + |
| 12 | +Many web/mobile applications generate huge amount of event logs (c,f. |
| 13 | +login, logout, purchase, follow, etc). To analyze these event logs could |
| 14 | +be really valuable for improving the service. However, the challenge is |
| 15 | +collecting these logs easily and reliably. |
| 16 | + |
| 17 | +`Fluentd <https://github.com/fluent/fluentd>`__ solves that problem by |
| 18 | +having: easy installation, small footprint, plugins, reliable buffering, |
| 19 | +log forwarding, etc. |
| 20 | + |
| 21 | +**fluent-logger-python** is a Python library, to record the events from |
| 22 | +Python application. |
| 23 | + |
| 24 | +Requirements |
| 25 | +------------ |
| 26 | + |
| 27 | +- Python 2.6 or greater including 3.x |
| 28 | + |
| 29 | +Installation |
| 30 | +------------ |
| 31 | + |
| 32 | +This library is distributed as 'fluent-logger' python package. Please |
| 33 | +execute the following command to install it. |
| 34 | + |
| 35 | +.. code:: sh |
| 36 | +
|
| 37 | + $ pip install fluent-logger |
| 38 | +
|
| 39 | +Configuration |
| 40 | +------------- |
| 41 | + |
| 42 | +Fluentd daemon must be launched with a tcp source configuration: |
| 43 | + |
| 44 | +:: |
| 45 | + |
| 46 | + <source> |
| 47 | + type forward |
| 48 | + port 24224 |
| 49 | + </source> |
| 50 | + |
| 51 | +To quickly test your setup, add a matcher that logs to the stdout: |
| 52 | + |
| 53 | +:: |
| 54 | + |
| 55 | + <match app.**> |
| 56 | + type stdout |
| 57 | + </match> |
| 58 | + |
| 59 | +Usage |
| 60 | +----- |
| 61 | + |
| 62 | +Event-Based Interface |
| 63 | +~~~~~~~~~~~~~~~~~~~~~ |
| 64 | + |
| 65 | +First, you need to call ``logger.setup()`` to create global logger |
| 66 | +instance. This call needs to be called only once, at the beggining of |
| 67 | +the application for example. |
| 68 | + |
| 69 | +By default, the logger assumes fluentd daemon is launched locally. You |
| 70 | +can also specify remote logger by passing the options. |
| 71 | + |
| 72 | +.. code:: python |
| 73 | +
|
| 74 | + from fluent import sender |
| 75 | +
|
| 76 | + # for local fluent |
| 77 | + sender.setup('app') |
| 78 | +
|
| 79 | + # for remote fluent |
| 80 | + sender.setup('app', host='host', port=24224) |
| 81 | +
|
| 82 | +Then, please create the events like this. This will send the event to |
| 83 | +fluent, with tag 'app.follow' and the attributes 'from' and 'to'. |
| 84 | + |
| 85 | +.. code:: python |
| 86 | +
|
| 87 | + from fluent import event |
| 88 | +
|
| 89 | + # send event to fluentd, with 'app.follow' tag |
| 90 | + event.Event('follow', { |
| 91 | + 'from': 'userA', |
| 92 | + 'to': 'userB' |
| 93 | + }) |
| 94 | +
|
| 95 | +Python logging.Handler interface |
| 96 | +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 97 | + |
| 98 | +This client-library also has ``FluentHandler`` class for Python logging |
| 99 | +module. |
| 100 | + |
| 101 | +.. code:: python |
| 102 | +
|
| 103 | + import logging |
| 104 | + from fluent import handler |
| 105 | +
|
| 106 | + custom_format = { |
| 107 | + 'host': '%(hostname)s', |
| 108 | + 'where': '%(module)s.%(funcName)s', |
| 109 | + 'type': '%(levelname)s', |
| 110 | + 'stack_trace': '%(exc_text)s' |
| 111 | + } |
| 112 | +
|
| 113 | + logging.basicConfig(level=logging.INFO) |
| 114 | + l = logging.getLogger('fluent.test') |
| 115 | + h = handler.FluentHandler('app.follow', host='host', port=24224) |
| 116 | + formatter = handler.FluentRecordFormatter(custom_format) |
| 117 | + h.setFormatter(formatter) |
| 118 | + l.addHandler(h) |
| 119 | + l.info({ |
| 120 | + 'from': 'userA', |
| 121 | + 'to': 'userB' |
| 122 | + }) |
| 123 | + l.info('{"from": "userC", "to": "userD"}') |
| 124 | + l.info("This log entry will be logged with the additional key: 'message'.") |
| 125 | +
|
| 126 | +You can also customize formatter via logging.config.dictConfig |
| 127 | + |
| 128 | +.. code:: python |
| 129 | +
|
| 130 | + import logging.config |
| 131 | + import yaml |
| 132 | +
|
| 133 | + with open('logging.yaml') as fd: |
| 134 | + conf = yaml.load(fd) |
| 135 | +
|
| 136 | + logging.config.dictConfig(conf['logging']) |
| 137 | +
|
| 138 | +A sample configuration ``logging.yaml`` would be: |
| 139 | + |
| 140 | +.. code:: python |
| 141 | +
|
| 142 | + logging: |
| 143 | + version: 1 |
| 144 | +
|
| 145 | + formatters: |
| 146 | + brief: |
| 147 | + format: '%(message)s' |
| 148 | + default: |
| 149 | + format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s' |
| 150 | + datefmt: '%Y-%m-%d %H:%M:%S' |
| 151 | + fluent_fmt: |
| 152 | + '()': fluent.handler.FluentRecordFormatter |
| 153 | + format: |
| 154 | + level: '%(levelname)s' |
| 155 | + hostname: '%(hostname)s' |
| 156 | + where: '%(module)s.%(funcName)s' |
| 157 | +
|
| 158 | + handlers: |
| 159 | + console: |
| 160 | + class : logging.StreamHandler |
| 161 | + level: DEBUG |
| 162 | + formatter: default |
| 163 | + stream: ext://sys.stdout |
| 164 | + fluent: |
| 165 | + class: fluent.handler.FluentHandler |
| 166 | + host: localhost |
| 167 | + port: 24224 |
| 168 | + tag: test.logging |
| 169 | + formatter: fluent_fmt |
| 170 | + level: DEBUG |
| 171 | + null: |
| 172 | + class: logging.NullHandler |
| 173 | +
|
| 174 | + loggers: |
| 175 | + amqp: |
| 176 | + handlers: [null] |
| 177 | + propagate: False |
| 178 | + conf: |
| 179 | + handlers: [null] |
| 180 | + propagate: False |
| 181 | + '': # root logger |
| 182 | + handlers: [console, fluent] |
| 183 | + level: DEBUG |
| 184 | + propagate: False |
| 185 | +
|
| 186 | +Testing |
| 187 | +------- |
| 188 | + |
| 189 | +Testing can be done using |
| 190 | +`nose <https://nose.readthedocs.org/en/latest/>`__. |
| 191 | + |
| 192 | +Contributors |
| 193 | +------------ |
| 194 | + |
| 195 | +Patches contributed by `those |
| 196 | +people <https://github.com/fluent/fluent-logger-python/contributors>`__. |
| 197 | + |
| 198 | +License |
| 199 | +------- |
| 200 | + |
| 201 | +Apache License, Version 2.0 |
0 commit comments