DarkMetrix Log

A fast network logging system powered by Apache Kafka.

Architecture

Quick start

Preperation

An Apache Kafka need to be installed previously and create a topic.

./kafka-topics.sh --zookeeper localhost:2181 --create --replication-factor 1 --partitions 10 --topic net_log

See DarkMetrix/log/agent or DarkMetrix/log/server to get more information about Kafka setting.

Install from source

git clone git@github.com:DarkMetrix/log.git 

Build agent & run (You will need a root user)

$cd DarkMetrix/log/agent/src
$go build -o ../bin/dm_log_agent
$../admin/start.sh

Build server & run

$cd DarkMetrix/log/server/src
$go build -o ../bin/dm_log_server
$../admin/start.sh

Protocol

We use google/protobuf as the protocol, so you shuold use protobuf whatever programming language you use to marshal the log and send via unix domain socket.

log.proto

syntax = "proto2";

package log_proto;

//Log package
message LogPackage
{
    optional string project = 1;    //Project name(Also used as the folder name)
    optional string service = 2;    //Service name(Also used in the log file name)
    optional uint32 level = 3;      //Log level(INFO = 0, WARNING = 1, ERROR = 2, FATAL = 3)
    optional bytes  log = 4;        //Log content
};

Configuration

All configuration file is in json.

agent

See DarkMetrix/log/agent to get more information.

server

See DarkMetrix/log/server to get more information.

Lisense

DarkMetrix Log

MIT license

Dependencies

• github.com/cihub/seelog BSD License
• github.com/golang/protobuf/proto BSD License
• github.com/spf13/viper MIT License
• github.com/Shopify/sarama MIT License