Load Balancing

OnBase

Technical White Paper

Document Version Foundation LTR 1.0
Last Updated: January 21, 2020

Research and Development Performance Team

Hyland Software, Inc.
28500 Clemens Road
Westlake, OH 44145
Phone: +1 440.788.5000
https://www.onbase.com/
Copyright
Information in this document is subject to change without notice. The OnBase® software (the “Software")
described in this document is furnished only under a separate license agreement and may be used or copied only
according to the terms of such agreement. It is against the law to copy the Software except as specifically allowed
in the license agreement. This document or accompanying materials contains certain information which is
confidential information of Hyland Software, Inc. and which is subject to the confidentiality provisions agreed to by
you.
All data, names, and formats used in this document’s examples are fictitious unless noted otherwise. Complying
with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright law,
no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any
form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without
the express written permission of Hyland Software, Inc.
©2018 Hyland Software, Inc. All rights reserved.
Depending on the modules licensed, the OnBase® software may include software developed and copyrighted by
third parties, including but not limited to the following:
A2iA CheckReader™ by A2iA Corp;
Adobe® PDF Library™ by Adobe Systems Incorporated;
dtSearch® Text Retrieval Engine by dtSearch Corp.;
software or other content adapted from Smart Client – Composite UI Application Block by Microsoft
Corporation © 2005 Microsoft Corporation;
software or other content adapted from Microsoft patterns & practices ObjectBuilder © 2006 Microsoft
Corporation;
Nuance™ OCR © 1994-2012 Nuance Communications;
portions of imaging code owned and copyrighted by Pegasus Imaging Corporation, Tampa, FL;
Imaging Technology copyrighted by Snowbound Software Corporation, Snowbound.com;
CD-R technology by Sonic Solutions, Inc.;
full-text indexing technology by Autonomy;
IDSMail © 2005 by Intuitive Data Solutions;
jLex Copyright 1996-2003 by Elliot Joel Berk and C. Scott Ananian;
Rumba by NetManage;
AutoVue by Oracle America, Inc.
Streaming Powered by Wowza Streaming software
All rights reserved.
Further information regarding third-party software included in OnBase can be found in the About OnBase box
within the Software.
Hyland, Hyland Software®, and OnBase® are registered and/or unregistered trademarks of Hyland Software, Inc. in
the United States and other countries. A2iA CheckReader™ is a trademark of A2iA Corporation.
Adobe® PDF Library™ is a trademark of Adobe Systems Incorporated.
All other trademarks, service marks, trade names and products of other companies are the property of their
respective owners.
Intended Audience
This document is intended for IT Professionals and Solutions Designers that deploy OnBase and have an
understanding of OnBase Web/Application Server architecture.

Abstract
This document discusses OnBase’s support for load balancing and the ability to load-balance between
the Web and Application Servers.

Version Statement
This document has been updated for OnBase Foundation but is applicable to versions 10 and later.
Table of Contents
COPYRIGHT ................................................................................................................................................. 2
INTENDED AUDIENCE ................................................................................................................................ 4
ABSTRACT................................................................................................................................................... 4
VERSION STATEMENT ............................................................................................................................... 4
TABLE OF CONTENTS ............................................................................................................................... 5
INTRODUCTION........................................................................................................................................... 7
LOAD BALANCER CONFIGURATION ....................................................................................................... 8
PERSISTENCE ............................................................................................................................................. 8
COOKIE-BASED LOAD BALANCING ................................................................................................................ 8
TIMEOUTS – SESSION, IDLE, AND REQUEST .................................................................................................. 8
LIMITS – REQUESTS, CONNECTIONS, BANDWIDTH, ETC. ................................................................................ 9
CASE SENSITIVITY ....................................................................................................................................... 9
SSL/HTTPS OFFLOADING .......................................................................................................................... 9
LOAD BALANCING METHOD .......................................................................................................................... 9
ONBASE CONFIGURATION ..................................................................................................................... 10
LOAD-BALANCING MULTIPLE W EB SERVERS ............................................................................................... 10
LOAD-BALANCING MULTIPLE APPLICATION SERVERS................................................................................... 10
Web Server Configuration ................................................................................................................... 10
Client Configuration ............................................................................................................................. 11
RESPONDING TO NODE FAILURES .............................................................................................................. 12
REDUNDANCY, SCALABILITY, AND HIGH AVAILABILITY .................................................................................. 13
Redundancy ........................................................................................................................................ 13
Scalability ............................................................................................................................................ 13
High Availability ................................................................................................................................... 13
HOW LAYER 7 LOAD BALANCING W ORKS WITH ONBASE.............................................................................. 14
TROUBLESHOOTING LOAD BALANCING WITH ONBASE ................................................................... 16
CHECK DIAGNOSTICS CONSOLE ................................................................................................................. 16
CHECK THE HTTP(S) TRAFFIC FOR PROPER COOKIE HANDLING ................................................................. 16
Using Fiddler to Verify the Load Balancer Cookie .............................................................................. 16
Using Wireshark to Verify the Load Balancer Cookie ......................................................................... 18
CHECK THE HTTP(S) TRAFFIC FOR ERRORS .............................................................................................. 21
UNITY CLIENT AUTOMATIC RECONNECT...................................................................................................... 22
REMOVE OR REDUCE COMPLEXITIES .......................................................................................................... 22
REMOVE OR BYPASS THE LOAD BALANCER................................................................................................. 23
CONTACTING YOUR FIRST LINE OF SUPPORT.............................................................................................. 23
LOAD-BALANCED EXAMPLES ................................................................................................................ 24
INTERNAL CORE CLIENTS........................................................................................................................... 24
EXTERNAL W EB AND CORE CLIENTS .......................................................................................................... 25
EXTERNAL AND INTERNAL CLIENTS ............................................................................................................. 27
Introduction
Load balancing is a deployment technique that can provide high availability, disaster recovery, and
additional scalability. Load balancing is a means of increasing capacity, making applications resistant to
hardware and software failures, and creating a system robust enough to support mission-critical
applications. OnBase Web and Application Servers can use load balancing in large, complex installations
to reliably support thousands of concurrent users.

In order to use load balancing for OnBase, persistent or “sticky” OnBase sessions must be maintained by
the load balancer. This requires the use of either cookie-based (Layer 7) or IP-based (Layer 3 or Layer 4)
load-balancing techniques.
If using cookie-based (Layer 7) load balancing, the following requirements also apply:
 Cookies must be generated using RFC 2109 or RFC 2965 formats.
 Web Servers must be configured to use SOAP communication.
OnBase supports hardware, software, and virtualized load balancers. Hyland Software has no preferred
or recommended load balancing vendor.

Load-balancing is a complex subject and this document is simply an account of OnBase’s requirements
for load balancing. Load balancing can be implemented incorrectly and requires skill and experience to
configure and troubleshoot.

Any references to “Web Server” are also applicable to the OnBase Web Server (AppNet), Gateway
Caching Server, Mobile Broker, Patient Window, Agenda, Plan Review, Public Access, etc. All core-based
modules use the same architecture when communicating with the Application Server through a load
balancer.
Load Balancer Configuration
Because Hyland does not offer preferences or recommendations for specific load balancing vendors, we
cannot provide device-specific support. You will need to reach out to your load balancing vendor for
support specific to their product.

Persistence
In order to load-balance your OnBase solution your load balancer must support cookie-based (Layer 7)
or IP-based (Layer 3 or Layer 4) load balancing.
The load balancer must also be configured to use persistent sessions in order to maintain users’ session
state information with a specific server. Persistent sessions are also referred to as “client affinity” or
“sticky” sessions. The initial server that a client is routed to is determined by the load balancer, which
issues the cookie. The load balancer reads the cookie on subsequent requests, routing those requests to
the correct server.

Cookie-Based Load Balancing

OnBase supports RFC 2109 or RFC 2965-based cookies. Any cookies generated by the load balancer must
be in one of these formats.
Using the ASP.NET_SessionID cookie for cookie-based load balancing is not supported and should not be
used. Instead, your load balancer should be providing a cookie to clients.

Note: OnBase is a stateful application and all requests the client need to be delivered to the correct
server. If your appliance is configured to use cookie-based load balancing, the traffic needs to be load
balanced based on that cookie. To improve performance, some appliances are shipped with a default
setting to load balance for each TCP connection, rather than each HTTP request. This setting can cause
adverse effects with OnBase. Please consult your vendor’s documentation to disable this feature if
enabled.

Timeouts – Session, Idle, and Request

Most load balancers have timeout settings. These settings can control how long a request/response is
allowed to take or how long a user can be idle before being removed from persistence. Some load
balancers may even have separate timeout values for client to load balancer and another for load
balancer to server. When a timeout from a load balancer does occur, each vendor might handle it
differently. Some will close the connection, while others may return an HTTP error. Load balancers also
each have their own default values for timeouts. It is important to know what timeout options are
available, how vendors react to a timeout, and how to log or identify when a timeout occurs.
Hyland recommends setting load balancer timeouts high enough so that OnBase will time out first.
Because every load balancer behaves differently, it is easier for us to troubleshoot an issue if the
timeout is triggered by our software instead of the load balancer. The default execution timeout for the
Application Server is 300 seconds (5 minutes), and the default idle timeout for ASP.NET sessions on the
Application Server is 20 minutes.

Limits – Requests, Connections, Bandwidth, etc.

Whether used to increase performance or reduce the risk of Denial of Service attacks, each load
balancing component has various limit settings that you should be aware of. Common limits include the
number of connections, connection rate, number of requests, and user bandwidth. Different load
balancers have their own configuration and behavior related to these limit settings.

Case Sensitivity
When configuring your load-balanced environment, pay special attention to the letter casing of text in
the various configurations. Often times there are vendor-specific casing rules. For example, the URL
root, cookie path, and cookie name are case-sensitive.

SSL/HTTPS Offloading
Many load balancers also provide SSL/HTTPS offloading capability. Offloading the encryption and
decryption of HTTPS traffic is supported.

Load Balancing Method

Many load balancers have different methods of balancing load among servers (round robin, least
connections, least requests, quickest response time, etc.). The best method to use depends on the
specifics of your environment and expected workloads.
OnBase Configuration
Load-Balancing Multiple Web Servers
If there is a load balancer between the client workstations and the Web Server, then modify the
dmsVirtualRoot setting in the Web Server’s Web.config file to point to the load balancer. This is the URL
users will go to when accessing the site.
1. In the Web Server’s Web.config, locate the dmsVirtualRoot setting.
2. Change the value to the load balancer’s IP address, hostname, or fully qualified domain name
(whichever is appropriate) rather than the Web Server’s.

In the following example, the value of the dmsVirtualRoot setting has been modified to point to a load
balancer with the hostname WebLoadBalancer:

Load-Balancing Multiple Application Servers

Web Server Configuration

If there is a load balancer between the Web Server and Application Server, then you must modify the
ApplicationServer settings in the Web Server’s Web.config file to point to the load balancer and to use
SOAP.
1. In the Web Server’s Web.config, locate the ApplicationServer setting.
2. Change the URL value to the load balancer’s IP address, hostname, or fully qualified domain
name (whichever is appropriate) rather than the Application Server’s.
3. If using cookie-based (Layer 7) load balancing, two additional steps must be taken:
a. Change the extension at the end of the URL to .asmx.
b. Change the ServiceClientType to SOAP.
In the following example, the ApplicationServer settings have been modified to point to a load balancer
with a hostname of AppLoadBalancer which uses cookie-based (Layer 7) balancing:

Client Configuration
If there is a load balancer between the client and the Application Server, then the URL used to connect
to the Application Server will need to point to the load balancer. Depending on the client, the
Application Server URL is configured in different locations.

The following example shows the Unity Client’s Application Server URL setting (ServicePath) modified to
point to a load balancer.
This example shows OnBase Studio connecting to load balanced Application Servers.

Responding to Node Failures

OnBase does not signal or update load balancers when a failure occurs. Removing failed nodes from
load balancing is highly implementation-specific, and outside of the scope of this document.
Except as noted for Unity Client sessions, sessions may be lost in a failure, requiring users to reconnect.
Redundancy, Scalability, and High Availability
The primary purpose of load balancing is to assist with providing redundancy, scalability, and high
availability.

Redundancy
The primary goal of redundancy is to ensure that applications are available to users when needed. Load-
balancing through persistent sessions allows for the creation of new sessions but by design it does not
allow for dynamic re-routing of sessions to other nodes. In the current architecture, sessions that are
established across tiers must log in again if one of the nodes that they have affinity to fails.
It is worth noting that clients such as the Unity Client have additional redundancy built in. The
Unity Client has the ability to detect loss of connectivity to its tiers. If there are available nodes,
the Unity Client will attempt to re-establish a session with the current credentials, assuming
there is at least one available Web Server and one available Application Server in a multi-tier
deployment, or at least one available Application Server in a single-tier deployment.

Scalability
The majority of the processing for client sessions through the Web and Application Servers, including for
the Web Client, is handled by the Application Server. Because the Web and Application Servers can be
scaled independently, greater workloads can be supported without additional Web Server licenses.

High Availability
The addition of multiple servers assures high availability. If a hardware failure occurs on one Web Server,
new sessions can be routed to an available Web Server, whose requests in turn are routed to an
available Application Server. If an Application Server experiences a hardware failure, new sessions
coming from Web Servers can be routed to an available Application Server.
How Layer 7 Load Balancing Works with OnBase
The following diagram depicts the lifetime of two requests coming from a client and the effect of load
balancing on those requests.

1. A client makes a request to the Web Server and is serviced through the first load balancer.
2. The request does not contain a cookie yet so the load balancer sends it to an available Web
Server, where a session is established.
3. The Web Server makes a request to the Application Server via the second load balancer.
4. The request from the Web Server does not contain a cookie yet so the load balancer sends the
request to an available Application Server, where a session is established.
5. The Application Server responds to the request from the Web Server and the load balancer
attaches a cookie to the response.
6. The second load balancer associates the cookie with the Application Server and sends the
response back to the requesting Web Server.
7. The Web Server responds to the request from the client and attaches its session cookie to the
response.
8. The first load balancer associates the cookie with the Web Server and sends the response back
to the requesting client.
9. The client sends a second request to the Web Server. This request now contains the cookie
associated with the session on the Web Server.
10. The load balancer recognizes the cookie and forwards the request to the Web Server that was
the source of the cookie.
11. The Web Server makes a request to the Application Server. This request now contains the
cookie associated with the session on the Application Server.
12. The load balancer recognizes the cookie and forwards the request to the Application Server that
was the source of the cookie.
Troubleshooting Load Balancing with OnBase
Check Diagnostics Console
Be sure to check Diagnostics Console on all servers at the same time when using load balancing. If the
load balancer is sending traffic to the wrong Application Server, you should see a failed session error.

Beware of false positives from “failed to get session for session id” errors. These messages can
also be thrown when an end-user leaves a client open that has lost its session. The clients will
heartbeat/ping the server with a stale session id that will not be valid on the server. This can be
caused by a session timeout or after an IIS reset.

Check the HTTP(S) Traffic for Proper Cookie Handling

In cookie-based load balancing it is important that cookies are handled properly. To verify this, you’ll
want to look for the cookie that you receive from the load balancer and verify that it’s the same cookie
used on all subsequent requests throughout the OnBase session. It is important to monitor the traffic at
every tier, not from only one side. Ideally you want to capture information about traffic from the client,
load balancer, and server side at the same time.

Using Fiddler to Verify the Load Balancer Cookie

Fiddler is a free web debugging proxy and can be found at http://www.telerik.com/fiddler. Fiddler is run
on the client to view HTTP/HTTPS communication. It is useful to see the cookie received from the load
balancer and confirm that the client is using it properly.
Begin by looking at the headers for the request and responses. In Fiddler, click on the Inspectors tab and
then Headers on the top (request) and bottom (response).
The following example capture shows logging into OnBase and when the cookie is applied.
In the initial request headers from the capture above you can see that there are no cookies present,
which is expected. However, notice in the response that new cookies are being set, shown here as Set-
Cookie. In this example, the X-Mapping-cijgbboe cookie is set by the load balancer. The name of the
cookie varies by vendor. Its value is generated by the load balancer and is used to identify which server
the request should be routed to.
Moving down in the session list, you can see that the OnBase client provided the load balancer cookie.
This allows the request to be routed to the same server. Selecting requests further down the Fiddler
session list allows you to verify that the same cookie is being used on all requests throughout this
OnBase session.

Using Wireshark to Verify the Load Balancer Cookie

Wireshark is a free packet analyzer tool that is used in network troubleshooting, network analysis, and
verification of software/communication protocols. Wireshark can be found at
https://www.wireshark.org/.
Wireshark, like Fiddler, shows the cookie being set and then applied on all subsequent requests. It has
the advantage of being able to run from both the client and server. It can often open up packet captures
from network devices such as a load balancer. However, Wireshark cannot easily decrypt and display
HTTPS traffic.
The example capture below shows logging into OnBase and when the cookie is applied.

At the top of the list you can see the first POST that the client makes to the server. Notice there are no
cookies in the header.
Looking at the response, the first 200 OK, notice that it contains the load balancer cookie. It is named X-
Mapping-cijgbboe in this example.
By choosing the next POST in the packet list you can see that it now contains the cookie in the request to
the server. Selecting frames through the Wireshark packet list allows you to verify that the same cookie
is being used on all requests throughout this OnBase session.

Check the HTTP(S) Traffic for Errors

Check for any HTTP error codes coming back in a response or any error pages that might contain details
about what went wrong. Every load balancer can behave differently, so use a tool such as Fiddler or
Wireshark to check for any problems.
In the following example, the client received a HTTP 500 error page that was not the default 500 error
page from IIS. Instead it came from the load balancer after 30 seconds (its default timeout).
Unity Client Automatic Reconnect
The Unity Client has the ability to automatically reconnect when it gets a failed session error from the
server. Because the reconnect happens without the user knowing, it can sometimes provide a false
positive that everything is working as it keeps retrying. Check Diagnostics Console for errors to be sure.

Remove or Reduce Complexities

Remove some of the complexities of the load balancer configuration such as HTTPS, Windows
Authentication, etc. This will help narrow down if a particular configuration is having problems.
Remove or Bypass the Load Balancer
Completely remove the load balancer from the environment so the client connects directly to the Web
Server or Application Server. This will verify whether the load balancer is the problem or not.

Contacting Your First Line of Support

If you cannot figure out a load balancing issue on your own, contact your first line of support for
OnBase. Hyland usually requires a Fiddler capture with Diagnostic Console (Error and Service tabs) to
assist with troubleshooting load balancing issues.
Load-Balanced Examples
The following diagrams depict multi-tiered load balancing configurations.

Internal Core Clients

The above diagram depicts a set of Core Clients and Application Servers internal to the network. The
clients are load-balanced directly to the Application Serves.
External Web and Core Clients
The above diagrams depict a set of external Web and Core Clients, internal Web Servers, and internal
Application Servers. The clients are load-balanced directly to the Web Server and then load-balanced
(top diagram) or channelized (bottom diagram) to the Application Servers.
External and Internal Clients

This scenario is a hybrid of previous scenarios. There are internal Core Clients (green), as well as external
Web and Core Clients (blue). The internal Core Clients are load-balanced directly to the Application
Server tier while the external Web and Core Clients are load-balanced to the Web Server tier which in
turn is load-balanced to the Application Server tier. In this scenario, the Web and external Core clients
are deployed over the internet, so you would not want them load-balanced directly to the Application
Server tier.
The Application Server requires an ADO.Net connection to the database and file access to the disk
groups and would typically not sit in the DMZ. The Web Server tier requires only HTTP access. This
warrants putting the Web Server tier and Application Server tier in separate zones separated by a
firewall. In highly secure environments, there is typically a firewall from the Application Server tier into
the core network as well.