Skip to content

Commit 592879b

Browse files
author
Damien Dallimore
committed
HEC appenders
1 parent 1ff8d97 commit 592879b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

59 files changed

+2375
-2095
lines changed

README.md

Lines changed: 7 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -14,27 +14,28 @@ This framework contains :
1414
* Implementation of Splunk CIM(Common Information Model) and best practice logging semantics
1515
* java.util.logging handler for logging to Splunk REST endpoints
1616
* java.util.logging handler for logging to Splunk Raw TCP Server Socket
17+
* java.util.logging handler for logging to Splunk HEC Endpoint
1718
* Log4j appender for logging to Splunk REST endpoints
1819
* Log4j appender for logging to Splunk Raw TCP Server Socket
20+
* Log4j appender for logging to Splunk HEC Endpoint
1921
* Logback appender for logging to Splunk REST endpoints
2022
* Logback appender for logging to Splunk Raw TCP Server Socket
23+
* Logback appender for logging to Splunk HEC Endpoint
2124
* Example logging configuration files
2225
* Javadocs
2326

2427
If you want to use UDP to send events to Splunk , then Log4j 1.x and Logback already have Syslog Appenders.
2528
Log4j 2 has a UDP Appender and Syslog Appender.
2629
And of course you can still use any File appenders and have the file monitored by a Splunk Universal Forwarder.
2730

28-
I generally recommend using the raw TCP handlers/appenders I have provided , they perform the best, and have features coded into them for
29-
auto connection re-establishment and configurable buffering of log events which will get flushed upon reconnection.
31+
I generally recommend using the raw TCP or HEC handlers/appenders I have provided , they perform the best, and have features coded into them for auto connection re-establishment and configurable buffering of log events which will get flushed upon reconnection.
3032

3133
## Logging frameworks galore
3234

3335
Log4j 2 and Log4j 1.x are very distinct from one another.
3436
Logback was actually the "new version" of Log4j 1.x , and then Log4J 2 attempted to improve upon Logback.
3537
This rather convoluted family tree has essentially transpired with 3 different logging frameworks in play, each with different characteristics.
36-
Log4j 1.x still has a very large legacy usage base in enterprise software therefore warrants addressing with its own custom appenders
37-
and example configurations.
38+
Log4j 1.x still has a very large legacy usage base in enterprise software therefore warrants addressing with its own custom appenders and example configurations.
3839

3940
## Splunk Universal Forwarder vs Splunk Java Logging
4041

@@ -48,7 +49,7 @@ semantic format.
4849

4950
## Resilience
5051

51-
The HTTP REST and Raw TCP handler/appenders have autonomous socket reconnection logic in case of connection failures.
52+
The HTTP REST ,Raw TCP and HEC handler/appenders have autonomous socket reconnection logic in case of connection failures.
5253
There is also internal event queuing that is loosely modelled off Splunk's outputs.conf for Universal Forwarders.
5354
You can set these propertys :
5455
* maxQueueSize : defaults to 500KB , format [integer|integer[KB|MB|GB]]
@@ -86,7 +87,7 @@ Details can be found in the file LICENSE.
8687

8788
## Quick Start
8889

89-
1. Untar releases/splunklogging-1.0.tar.gz
90+
1. Untar releases/splunklogging-1.3.tar.gz
9091
2. All the required jar files are in the lib directory..
9192
3. Assume you know how to setup your classpath to use your preferred logging framework implementation.
9293
4. There is a simple code example here https://github.com/damiendallimore/SplunkJavaLogging/blob/master/src/com/splunk/logging/examples/Example.java
@@ -106,19 +107,6 @@ resources to your computer. For example, use the following command:
106107

107108
> git clone https://github.com/damiendallimore/SplunkJavaLogging.git
108109
109-
## Resources
110-
111-
Splunk Common Information Model
112-
113-
* http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/UnderstandandusetheCommonInformationModel
114-
115-
Splunk Best Practice Logging Semantics
116-
117-
* http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6
118-
119-
Splunk documentation
120-
121-
* http://docs.splunk.com/Documentation/Splunk
122110

123111
## Contact
124112

RELEASE_NOTES.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,13 @@
1+
1.3
2+
---
3+
4+
Added HEC appenders
5+
16
1.2
27
---
38

4-
Set deafult socket buffer sizes for TCP inputs to 8192
9+
Set default socket buffer sizes for TCP inputs to 8192
10+
511
1.1
612
---
713
Added new Splunk SDK jar with TLS support

build/build.properties

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
version=1.2
1+
version=1.3
22

33
src=src
44
classes=classes
@@ -12,4 +12,4 @@ main_jar=splunklogging.jar
1212
suffix=.tar.gz
1313
compressionType=gzip
1414
id=splunklogging
15-
javaClassTarget=1.5
15+
javaClassTarget=1.6

config/jdklogging.properties

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,20 @@ com.splunk.logging.jdk.handler.SplunkRawTCPHandler.port=5151
3030
com.splunk.logging.jdk.handler.SplunkRawTCPHandler.maxQueueSize=5MB
3131
com.splunk.logging.jdk.handler.SplunkRawTCPHandler.dropEventsOnQueueFull=false
3232

33+
# Set the default logging level for new SplunkHECHandler instances
34+
com.splunk.logging.jdk.handler.SplunkHECHandler.level=INFO
35+
com.splunk.logging.jdk.handler.SplunkHECHandler.host=somehost
36+
com.splunk.logging.jdk.handler.SplunkHECHandler.port=8088
37+
com.splunk.logging.jdk.handler.SplunkHECHandler.token=YOURTOKEN
38+
com.splunk.logging.jdk.handler.SplunkHECHandler.https=false
39+
com.splunk.logging.jdk.handler.SplunkHECHandler.poolsize=1
40+
com.splunk.logging.jdk.handler.SplunkHECHandler.index=main
41+
com.splunk.logging.jdk.handler.SplunkHECHandler.source=foo
42+
com.splunk.logging.jdk.handler.SplunkHECHandler.sourcetype=bar
43+
com.splunk.logging.jdk.handler.SplunkHECHandler.maxQueueSize=5MB
44+
com.splunk.logging.jdk.handler.SplunkHECHandler.dropEventsOnQueueFull=false
45+
46+
3347
# Set the default logging level for new ConsoleHandler instances
3448
java.util.logging.ConsoleHandler.level = INFO
3549
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

config/log4j.properties

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,22 @@ log4j.appender.splunkrawtcp.layout.ConversionPattern=%m%n
3838
#log4j.appender.splunkrawtcp.layout.ConversionPattern=%d{ABSOLUTE} %m%n
3939
#log4j.appender.splunkrawtcp.layout.ConversionPattern=%m loglevel="%p"%n
4040

41+
# Send log events to a Splunk HEC Endpoint
42+
log4j.appender.splunkhec=com.splunk.logging.log4j.appender.SplunkHECAppender
43+
log4j.appender.splunkhec.host=somehost
44+
log4j.appender.splunkhec.port=8088
45+
log4j.appender.splunkhec.token=YOURTOKEN
46+
log4j.appender.splunkhec.https=false
47+
log4j.appender.splunkhec.poolsize=1
48+
log4j.appender.splunkhec.index=main
49+
log4j.appender.splunkhec.source=foo
50+
log4j.appender.splunkhec.sourcetype=bar
51+
log4j.appender.splunkhec.maxQueueSize=5MB
52+
log4j.appender.splunkhec.dropEventsOnQueueFull=false
53+
log4j.appender.splunkhec.layout=org.apache.log4j.PatternLayout
54+
log4j.appender.splunkhec.layout.ConversionPattern=%m%n
55+
56+
4157
#Console appender
4258
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
4359
log4j.appender.stdout.Target=System.out

config/logback.xml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,27 @@
4040
</layout>
4141
</appender>
4242

43+
<!--Send log events to a Splunk HEC Endpoint-->
44+
<appender name="splunkrawtcp" class="com.splunk.logging.logback.appender.SplunkHECAppender">
45+
<port>8088</port>
46+
<host>somehost</host>
47+
<token>YOUR TOKEN</token>
48+
<https>false</https>
49+
<poolsize>1</poolsize>
50+
<index>main</index>
51+
<source>foo</source>
52+
<sourcetype>bar</sourcetype>
53+
<maxQueueSize>5MB</maxQueueSize>
54+
<dropEventsOnQueueFull>false</dropEventsOnQueueFull>
55+
<layout class="ch.qos.logback.classic.PatternLayout">
56+
<pattern>%m%n</pattern>
57+
<!-- optionally you can enrich the messages with formatting tokens from the logging framework
58+
<pattern>%d %m%n</pattern>
59+
<pattern>%m loglevel="%p"%n</pattern>
60+
-->
61+
</layout>
62+
</appender>
63+
4364
<!--Send log events to a console-->
4465
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
4566
<Target>System.out</Target>

javadocs/allclasses-frame.html

Lines changed: 4 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,12 @@
22
<!--NewPage-->
33
<HTML>
44
<HEAD>
5-
<!-- Generated by javadoc (build 1.6.0_43) on Fri Feb 07 13:06:07 NZDT 2014 -->
5+
<!-- Generated by javadoc (build 1.6.0_65) on Tue Aug 25 18:25:18 ICT 2015 -->
66
<TITLE>
77
All Classes
88
</TITLE>
99

10-
<META NAME="date" CONTENT="2014-02-07">
10+
<META NAME="date" CONTENT="2015-08-25">
1111

1212
<LINK REL ="stylesheet" TYPE="text/css" HREF="stylesheet.css" TITLE="Style">
1313

@@ -21,32 +21,20 @@
2121

2222
<TABLE BORDER="0" WIDTH="100%" SUMMARY="">
2323
<TR>
24-
<TD NOWRAP><FONT CLASS="FrameItemFont"><A HREF="com/splunk/logging/examples/Example.html" title="class in com.splunk.logging.examples" target="classFrame">Example</A>
24+
<TD NOWRAP><FONT CLASS="FrameItemFont"><A HREF="com/splunk/logging/HECTransportConfig.html" title="class in com.splunk.logging" target="classFrame">HECTransportConfig</A>
2525
<BR>
2626
<A HREF="com/splunk/logging/RestEventData.html" title="class in com.splunk.logging" target="classFrame">RestEventData</A>
2727
<BR>
28-
<A HREF="com/splunk/logging/jdk/handler/SplunkFormatter.html" title="class in com.splunk.logging.jdk.handler" target="classFrame">SplunkFormatter</A>
28+
<A HREF="com/splunk/logging/SplunkHECInput.html" title="class in com.splunk.logging" target="classFrame">SplunkHECInput</A>
2929
<BR>
3030
<A HREF="com/splunk/logging/SplunkInput.html" title="class in com.splunk.logging" target="classFrame">SplunkInput</A>
3131
<BR>
3232
<A HREF="com/splunk/logging/SplunkLogEvent.html" title="class in com.splunk.logging" target="classFrame">SplunkLogEvent</A>
3333
<BR>
3434
<A HREF="com/splunk/logging/SplunkLogEventFactory.html" title="class in com.splunk.logging" target="classFrame">SplunkLogEventFactory</A>
3535
<BR>
36-
<A HREF="com/splunk/logging/log4j/appender/SplunkRawTCPAppender.html" title="class in com.splunk.logging.log4j.appender" target="classFrame">SplunkRawTCPAppender</A>
37-
<BR>
38-
<A HREF="com/splunk/logging/logback/appender/SplunkRawTCPAppender.html" title="class in com.splunk.logging.logback.appender" target="classFrame">SplunkRawTCPAppender</A>
39-
<BR>
40-
<A HREF="com/splunk/logging/jdk/handler/SplunkRawTCPHandler.html" title="class in com.splunk.logging.jdk.handler" target="classFrame">SplunkRawTCPHandler</A>
41-
<BR>
4236
<A HREF="com/splunk/logging/SplunkRawTCPInput.html" title="class in com.splunk.logging" target="classFrame">SplunkRawTCPInput</A>
4337
<BR>
44-
<A HREF="com/splunk/logging/log4j/appender/SplunkRestAppender.html" title="class in com.splunk.logging.log4j.appender" target="classFrame">SplunkRestAppender</A>
45-
<BR>
46-
<A HREF="com/splunk/logging/logback/appender/SplunkRestAppender.html" title="class in com.splunk.logging.logback.appender" target="classFrame">SplunkRestAppender</A>
47-
<BR>
48-
<A HREF="com/splunk/logging/jdk/handler/SplunkRestHandler.html" title="class in com.splunk.logging.jdk.handler" target="classFrame">SplunkRestHandler</A>
49-
<BR>
5038
<A HREF="com/splunk/logging/SplunkRestInput.html" title="class in com.splunk.logging" target="classFrame">SplunkRestInput</A>
5139
<BR>
5240
</FONT></TD>

javadocs/allclasses-noframe.html

Lines changed: 4 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,12 @@
22
<!--NewPage-->
33
<HTML>
44
<HEAD>
5-
<!-- Generated by javadoc (build 1.6.0_43) on Fri Feb 07 13:06:07 NZDT 2014 -->
5+
<!-- Generated by javadoc (build 1.6.0_65) on Tue Aug 25 18:25:18 ICT 2015 -->
66
<TITLE>
77
All Classes
88
</TITLE>
99

10-
<META NAME="date" CONTENT="2014-02-07">
10+
<META NAME="date" CONTENT="2015-08-25">
1111

1212
<LINK REL ="stylesheet" TYPE="text/css" HREF="stylesheet.css" TITLE="Style">
1313

@@ -21,32 +21,20 @@
2121

2222
<TABLE BORDER="0" WIDTH="100%" SUMMARY="">
2323
<TR>
24-
<TD NOWRAP><FONT CLASS="FrameItemFont"><A HREF="com/splunk/logging/examples/Example.html" title="class in com.splunk.logging.examples">Example</A>
24+
<TD NOWRAP><FONT CLASS="FrameItemFont"><A HREF="com/splunk/logging/HECTransportConfig.html" title="class in com.splunk.logging">HECTransportConfig</A>
2525
<BR>
2626
<A HREF="com/splunk/logging/RestEventData.html" title="class in com.splunk.logging">RestEventData</A>
2727
<BR>
28-
<A HREF="com/splunk/logging/jdk/handler/SplunkFormatter.html" title="class in com.splunk.logging.jdk.handler">SplunkFormatter</A>
28+
<A HREF="com/splunk/logging/SplunkHECInput.html" title="class in com.splunk.logging">SplunkHECInput</A>
2929
<BR>
3030
<A HREF="com/splunk/logging/SplunkInput.html" title="class in com.splunk.logging">SplunkInput</A>
3131
<BR>
3232
<A HREF="com/splunk/logging/SplunkLogEvent.html" title="class in com.splunk.logging">SplunkLogEvent</A>
3333
<BR>
3434
<A HREF="com/splunk/logging/SplunkLogEventFactory.html" title="class in com.splunk.logging">SplunkLogEventFactory</A>
3535
<BR>
36-
<A HREF="com/splunk/logging/log4j/appender/SplunkRawTCPAppender.html" title="class in com.splunk.logging.log4j.appender">SplunkRawTCPAppender</A>
37-
<BR>
38-
<A HREF="com/splunk/logging/logback/appender/SplunkRawTCPAppender.html" title="class in com.splunk.logging.logback.appender">SplunkRawTCPAppender</A>
39-
<BR>
40-
<A HREF="com/splunk/logging/jdk/handler/SplunkRawTCPHandler.html" title="class in com.splunk.logging.jdk.handler">SplunkRawTCPHandler</A>
41-
<BR>
4236
<A HREF="com/splunk/logging/SplunkRawTCPInput.html" title="class in com.splunk.logging">SplunkRawTCPInput</A>
4337
<BR>
44-
<A HREF="com/splunk/logging/log4j/appender/SplunkRestAppender.html" title="class in com.splunk.logging.log4j.appender">SplunkRestAppender</A>
45-
<BR>
46-
<A HREF="com/splunk/logging/logback/appender/SplunkRestAppender.html" title="class in com.splunk.logging.logback.appender">SplunkRestAppender</A>
47-
<BR>
48-
<A HREF="com/splunk/logging/jdk/handler/SplunkRestHandler.html" title="class in com.splunk.logging.jdk.handler">SplunkRestHandler</A>
49-
<BR>
5038
<A HREF="com/splunk/logging/SplunkRestInput.html" title="class in com.splunk.logging">SplunkRestInput</A>
5139
<BR>
5240
</FONT></TD>

javadocs/com/splunk/logging/RestEventData.html

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,12 @@
22
<!--NewPage-->
33
<HTML>
44
<HEAD>
5-
<!-- Generated by javadoc (build 1.6.0_43) on Fri Feb 07 13:06:06 NZDT 2014 -->
5+
<!-- Generated by javadoc (build 1.6.0_65) on Tue Aug 25 18:25:17 ICT 2015 -->
66
<TITLE>
77
RestEventData
88
</TITLE>
99

10-
<META NAME="date" CONTENT="2014-02-07">
10+
<META NAME="date" CONTENT="2015-08-25">
1111

1212
<LINK REL ="stylesheet" TYPE="text/css" HREF="../../../stylesheet.css" TITLE="Style">
1313

@@ -37,8 +37,7 @@
3737
<A NAME="navbar_top_firstrow"><!-- --></A>
3838
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="3" SUMMARY="">
3939
<TR ALIGN="center" VALIGN="top">
40-
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="../../../overview-summary.html"><FONT CLASS="NavBarFont1"><B>Overview</B></FONT></A>&nbsp;</TD>
41-
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="package-summary.html"><FONT CLASS="NavBarFont1"><B>Package</B></FONT></A>&nbsp;</TD>
40+
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="../../../com/splunk/logging/package-summary.html"><FONT CLASS="NavBarFont1"><B>Package</B></FONT></A>&nbsp;</TD>
4241
<TD BGCOLOR="#FFFFFF" CLASS="NavBarCell1Rev"> &nbsp;<FONT CLASS="NavBarFont1Rev"><B>Class</B></FONT>&nbsp;</TD>
4342
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="class-use/RestEventData.html"><FONT CLASS="NavBarFont1"><B>Use</B></FONT></A>&nbsp;</TD>
4443
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="package-tree.html"><FONT CLASS="NavBarFont1"><B>Tree</B></FONT></A>&nbsp;</TD>
@@ -55,8 +54,8 @@
5554

5655
<TR>
5756
<TD BGCOLOR="white" CLASS="NavBarCell2"><FONT SIZE="-2">
58-
&nbsp;PREV CLASS&nbsp;
59-
&nbsp;<A HREF="../../../com/splunk/logging/SplunkInput.html" title="class in com.splunk.logging"><B>NEXT CLASS</B></A></FONT></TD>
57+
&nbsp;<A HREF="../../../com/splunk/logging/HECTransportConfig.html" title="class in com.splunk.logging"><B>PREV CLASS</B></A>&nbsp;
58+
&nbsp;<A HREF="../../../com/splunk/logging/SplunkHECInput.html" title="class in com.splunk.logging"><B>NEXT CLASS</B></A></FONT></TD>
6059
<TD BGCOLOR="white" CLASS="NavBarCell2"><FONT SIZE="-2">
6160
<A HREF="../../../index.html?com/splunk/logging/RestEventData.html" target="_top"><B>FRAMES</B></A> &nbsp;
6261
&nbsp;<A HREF="RestEventData.html" target="_top"><B>NO FRAMES</B></A> &nbsp;
@@ -543,8 +542,7 @@ <H2>
543542
<A NAME="navbar_bottom_firstrow"><!-- --></A>
544543
<TABLE BORDER="0" CELLPADDING="0" CELLSPACING="3" SUMMARY="">
545544
<TR ALIGN="center" VALIGN="top">
546-
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="../../../overview-summary.html"><FONT CLASS="NavBarFont1"><B>Overview</B></FONT></A>&nbsp;</TD>
547-
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="package-summary.html"><FONT CLASS="NavBarFont1"><B>Package</B></FONT></A>&nbsp;</TD>
545+
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="../../../com/splunk/logging/package-summary.html"><FONT CLASS="NavBarFont1"><B>Package</B></FONT></A>&nbsp;</TD>
548546
<TD BGCOLOR="#FFFFFF" CLASS="NavBarCell1Rev"> &nbsp;<FONT CLASS="NavBarFont1Rev"><B>Class</B></FONT>&nbsp;</TD>
549547
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="class-use/RestEventData.html"><FONT CLASS="NavBarFont1"><B>Use</B></FONT></A>&nbsp;</TD>
550548
<TD BGCOLOR="#EEEEFF" CLASS="NavBarCell1"> <A HREF="package-tree.html"><FONT CLASS="NavBarFont1"><B>Tree</B></FONT></A>&nbsp;</TD>
@@ -561,8 +559,8 @@ <H2>
561559

562560
<TR>
563561
<TD BGCOLOR="white" CLASS="NavBarCell2"><FONT SIZE="-2">
564-
&nbsp;PREV CLASS&nbsp;
565-
&nbsp;<A HREF="../../../com/splunk/logging/SplunkInput.html" title="class in com.splunk.logging"><B>NEXT CLASS</B></A></FONT></TD>
562+
&nbsp;<A HREF="../../../com/splunk/logging/HECTransportConfig.html" title="class in com.splunk.logging"><B>PREV CLASS</B></A>&nbsp;
563+
&nbsp;<A HREF="../../../com/splunk/logging/SplunkHECInput.html" title="class in com.splunk.logging"><B>NEXT CLASS</B></A></FONT></TD>
566564
<TD BGCOLOR="white" CLASS="NavBarCell2"><FONT SIZE="-2">
567565
<A HREF="../../../index.html?com/splunk/logging/RestEventData.html" target="_top"><B>FRAMES</B></A> &nbsp;
568566
&nbsp;<A HREF="RestEventData.html" target="_top"><B>NO FRAMES</B></A> &nbsp;

0 commit comments

Comments
 (0)