Content Management

<Insert Picture Here> Webinar

Dec 2008 – Content Server Performance Tuning

Cordell Melgaard Jason Ganske
Principal Technical Support Engineer Principal Technical Support Engineer
Global Customer Service Global Customer Service
 Welcome Content Management Customers!

Five Steps to a Faster Server

• Design Benchmarks

• Run Benchmarks

• Analyze Benchmarks

• Adjust the core configuration

• Adjust add-on configuration

 General Issues

• HELP!! My Content Server is Slow

• Searching
• Batchloading / Contribution
• Page Rendering (HSCP, JSP, ASP)
• Publishing

• Tuning is the same as any web application

• Adjust configuration flags for your specific needs
• Optimize Queries
• Maximize hardware
• Cache! Cache! Cache!
 Underlying Problem Areas

• All web applications have similar problems

• File system: too small or too slow
• Bandwidth: too slow to clients or database
• Processor: too small or overused
• Memory: too little or underused
• Database queries are not optimized
• Cache: too small, underused, or ignored

• What’s the solution?

• There’s a reason its called ‘tuning’…
• There is no magic button to fix performance problems
 Five Steps to a Faster Server

1. Design Benchmarks
 Gather setup and usage information
2. Run Benchmarks

3. Analyze Benchmarks

4. Adjust the core configuration

5. Adjust add-on configuration

 1) Design Benchmarks - Intro

• Speed is Relative
• Need something to compare against
1) Design Benchmarks - Intro

• Too few make benchmarks, but it’s vital to:

• Quantify performance
• Clean-room vs. real-world
• Avoid side-effects
• Avoid fixing the wrong problem

• Basic benchmarks can be quick and painless

• Thorough benchmarks can be slow and painful

 1) Design Benchmark – Initial Steps

• Document and verify setup

• Hardware: CPUs, memory, manufacturer
• Software: database, web server, CMS version
• Network: firewalls, routers, switches, bandwidth
• Security: LDAP, Active Directory, SSO

• Define where it’s slow

• Searching, Contributing, Viewing

• Check the Server Output for errors

• Logging even simple errors slows down the system
 1) Design Benchmarks – Web and
Batch Tests
• Design a web script for a typical user’s session
• Navigate to common pages
• Site Studio, Publisher, or standard templates
• Submit some forms
• Search, update, workflow, check in
• Recreate a typical workload for a user
 1) Design Benchmark - Pitfalls

• Make benchmark as real-world as possible

• Use the production server, or
• Use an exact replica of production server

• Isolate benchmark server

• Only allow testers to access
• Isolation required to quantify performance changes
• …but isolation does not resemble real-world

• Document Document Document

• Every test should be well documented to find consistency
 Five Steps to a Faster Server

1. Design Benchmarks

2. Run Benchmarks
 Tools, execution, logging

3. Analyze Benchmarks

4. Adjust the core configuration

5. Adjust add-on configuration

 2) Run Benchmarks – General Tools

• Free and Simple

• Content Server Health Monitor
• Content Server System Audit Page
• Process Explorer
• taskmgr (Win32) or top (Unix)

• Complex
• Mercury LoadRunner
• Java Profilers
• Perl / Python scripts
• PamIE, SamIE, WWW::Mechanize
 2) Run Benchmarks – Specific Tools

• Database performance analysis tools

• HUNDREDS of tools available
• Check with your DBA

• Apache mod_log_config
• Customize logs to record page response time
• Prioritize effort on popular pages

• Network analysis
• Ethereal/Wireshark
• trafshow
• tracert / traceroute
 2) Run Benchmarks – Setup and
Execution

• Number of users for web benchmark

• Concurrent = 1 to 10% of total users
• Peak usage = 10 to 80% weekly users

• Monitor all processes on all servers

• CPU and RAM usage, IO Reads
• Content Server: java, mkvdk, htmlexport
• Database: oracle, mssql, etc.
 2) Run Benchmarks – Log Files

• Web log files

• Refinery, Archiver, and Server logs
• Filter Administration for security filter logs

• Tracing output
• oracle/bin/IdcServerNT.log (Windows)
• oracle/etc/log (Unix)
• System Audit Info tracing sections:
indexer, pagecreation, requestaudit
socketrequests, searchquery
systemdatabase, userstorage
 2) Run Benchmarks – Pitfalls

• Adjustments
• Best-case: turn off logging, don’t allow contribution
• Bad-case: turn off search cache
• Worst-case: run batch and web tests simultaneously

• Double-check resource usage

• Is a virus scanner running?
• Is anybody else accessing the server?
• Who else is using CS dependent resources?
• File server, database, web server, LDAP/AD/NTLM
 Five Steps to a Faster Server

1. Design Benchmarks

2. Run Benchmarks

3. Analyze Benchmarks
 Find possible bottlenecks and retest

4. Adjust the core configuration

5. Adjust add-on configuration

 3) Analyze Benchmarks

• Review benchmarks
• Read, Analyze, Read, Analyze
• Time required to run batch tests
• Requests per second per user for web tests

• What is slow?

• EXAMPLE goals for business content:

• Read-only requests: 20 per second per GHz CPU
• Raw Verity queries: 4 per second per GHz CPU
• Contribution: 4 per second per GHz CPU
• Assume: 100k files and 1GB RAM
 3) Analyze Benchmarks – Find the
bottleneck
• Where is the bottleneck?
• Search engine / indexer
• Database
• Network
• Java VM (Content Server, Refinery, add-ons)
• File system
• Web Security Filter

• First benchmarks offer hints

• Need verification
• Enable more tracing sections, and retest
3) Analyze Benchmarks – Search or
Indexer Bottleneck Clues

• Template pages are fast

• Simple queries take over one second
• Navigation pages execute highly complex queries
• Analyze requestaudit & searchquery trace
• The system has a complex security model
• Search cache isn’t utilized
• System Audit Info: small cache, few hits
• Verity collection is not local
• Slow network file system?
• Database queries not optimized when using DB search
 3) Analyze Benchmarks – Database
Bottleneck Clues
Common Error
• System Audit Info page
•AnActive
error has
andoccurred. The stack trace below shows more
waiting connections
information.
• Dropped !$gpescm2:
connection !csIndexerAbortedMsg
errors
intradoc.common.ServiceException: !
• Too many connection errors
csIndexerUnableToBuildChanges!
• Most of request time is spent in database
csNoConnectionsAvailable,SystemDatabase
at intradoc.indexer.IndexerBuildChanges.buildChanges(Unknown
• Database CPU is overworked
Source)
• SCS and MKVDK underworked
at intradoc.indexer.IndexerBuildChanges.doWork(Unknown Source)
at intradoc.indexer.Indexer.doIndexing(Unknown Source)
• Hint:at intradoc.indexer.Indexer.buildIndex(Unknown Source)
at intradoc.server.IndexerMonitor.doIndexing(Unknown Source)
• Monitor searchquery tracing for timestamps on queries
at intradoc.server.IndexerMonitor$4.run(Unknown Source)
• Watch standard web logs for connection errors
 3) Analyze Benchmarks – Network
Bottleneck Clues

• Remote connections (databases) are slow

• …but remote services aren’t working very hard
• Use trafshow or Ethereal/Wireshark to determine saturation

• Hint:
• Monitor searchquery tracing for timestamps on queries
• Compare with running the query directly on the database
 3) Analyze Benchmarks – Core
Engine / Java VM Bottleneck Clues

• The Stellent engine runs pretty lean, but

• Publishing or page rendering can be slow

• It may be a Java VM problem if:

• It’s a consumption server
• It renders a lot of HCSPs or JSPs
• It runs a lot of workflows or subscriptions
• Java VM uses all the CPU or Memory
• Even if searching isn’t performed
• Multiple CPUs not being fully utilized
• Using outdated JVM
 3) Analyze Benchmarks – File System
Bottleneck Clues

•FAST
• File system is remote, but not high-performance
• SAN: Very fast, expensive, long setup
•RELIABLE
• NAS: Slower, cheaper, easy setup
•CHEAP
• Larger than expected file activity
• Monitor I/O reads, and I/O read Bytes
• File system is nearing capacity

PICK TWO
• Hint:
• Note: some remote file system problems indicate an
underlying network problem
3) File System Architecture
Performance Location
Web File Database
Server System
Fastest Local Local Local

Local Local Remote

Local SAN Remote

Local NAS Remote

Local Remote Remote

Remote Local Remote

Remote SAN Remote

Remote NAS Remote

Slowest Remote Remote Remote

 3) Analyze Benchmarks – Web
Security Filter Bottleneck Clues
• User’s first login takes several seconds
• Subsequent page delivery is fast
• Many users login as their browser’s home page
• Pay login cost, but don’t navigate further
• Unmanaged content is much faster
• Use only one thread to test

• Hint:
• Monitor verbose userstorage trace for timestamps
• Note: stack traces are not always errors in this section
 3) Analyze Benchmarks - Pitfalls

• Java profiler reports misleading

• Logging and virus scanners slow systems down

• Try to establish real-world benchmarks

• No search cache, do contribution and consumption

• Watch the search engine carefully

• Note the frequency and complexity of queries
• Substring queries KILL performance
 Five Steps to a Faster Server

1. Design Benchmarks

2. Run Benchmarks

3. Analyze Benchmarks

4. Adjust the core configuration

 Configuration, caching, database, and verity

5. Adjust add-on configuration

 4) Adjust Core
• Easy:
• Configuration flags
• Java VM tuning

• Difficult:
• IdocScript caching and tuning
• Simplify security model

• Harder:
• Verity Index Tuning
• Database Indexes and tuning (OracleQueryOptimizer)
 4) Adjust Core – Config Flags

• Indexer (Verity)
• AdditionalIndexBuildParams, DoAutoMaxMerge,
MaxMergeBaseCount, TimeoutPerOneMegInSec
• Search
• MaxSearchConnections, CachedResultRowCount
• Database
• NumConnections
• Only increase for testing can have opposite effect
• General
• UserCacheTimeout, IdcServerSocketQueueDepth
 4) Adjust Core – Java Tuning

• JVM is self-tuning, but benefits from hints

• Modify values in intradoc.cfg
• Optimize memory and compiler
• MetaLink Note : Note 460039.1
• JAVA_MAX_HEAP_SIZE=[Size in megabytes]
• Multiple CPU systems
• Use JVM 1.5.1 or better
• Tune GC for multiple CPUs, platform specific
• Different tuning options for CS versions
 4) Adjust Core – IdocScript Tuning

• Simplify pages
• Some pages needlessly do multiple requests
• Reduce number and complexity of queries
• Avoid queries that don’t cache, such as timestamps

• Resolve simple errors

• CS Log will constantly log IdocScript errors unnecessarily

• Use database search for exact matches

• SearchEngineName=DATABASE.METADATA
• QueryText=dDocType LIKE ‘ADACCT‘
 4) Adjust Core – IdocScript Caching

• cacheInclude
• Include a dynamichtml fragment from a cache
• For fragments that are used multiple times
• Global cache, named cache, or session cache

• setMaxAge
• Uses Cache-Control HTTP header for page caching
• For pages that one uses hits frequently
 4) Adjust Core – Simplify security
• Searches attach security clause
• The more complex the security, the longer the query
• In general, longer queries take longer to run
• Worst performance will be from users with mixed security

• Monitor searchquery trace in System Audit

• User input:
• dDocName <matches> `test`

• Actual query run:

• (dDocName <matches> `test`) <and>
(dSecurityGroup <not> `Secure`)
 4) Adjust Core – Verity Tuning

• Tune and houseclean the Verity collection

• MKVDK command-line options
• Leverage ‘zone search’ on common fields
• Avoid automatically adding every field to index
• Tune the number of ‘parts file’
• Many parts, fast search; Few parts, fast index
• Put commonly searched fields into separate parts
• Security group, title, author
• Plan ahead!
• Adding metadata fields requires an index rebuild
• Use dummy or non-indexed fields, or database search
 4) Adjust Core – Database Tuning
• OracleQueryOptimizer
• Free component for tuning Oracle DB queries
• Add extra indexes to commonly searched columns
• Update database statistics as a database grows
• Before & after inserting lots of content
• Maintains indexes, helps query optimizer
• Optimize row-locking algorithm
• ORACLE: optimizer_mode = first_rows
• Consider custom services or queries
• Database-specific queries for better performance
• Simplify check-ins when running batches
 4) Adjust Core - Pitfalls

• Be sure to re-run the benchmarks

• Record new performance data
• Watch for side-effects!

• Test server improvements misleading

• Production server will not see exact benefits…
• unless the benchmark was 100% real-world
 Five Steps to a Faster Server
1. Design Benchmarks

2. Run Benchmarks

3. Analyze Benchmarks

4. Adjust the core configuration

5. Adjust add-on configuration

 Conversion, publishing, J2EE, and other extras
 5) Adjust Add-ons

• Batchloader
• Dynamic Converter
• Site Studio
• PDF Converter
• Content Tracker
• CIS
 5) Adjust Add-ons - Batchloader

• Temporarily disable other activity

• Inbound Refinery / PDF Converter
• Auto update indexer cycle – Repository Manager
• Optimize the search collection (if using Verity)
prior to batch insert.
• Analyze your database usage during a
batchload. Goal is to help the database query
optimizer.
• Oracle: ANALYZE TABLE COMPUTE STATISTICS
• SQL Server: CREATE STATISTICS
• DB2: RUNSTATS
5) Adjust Add-ons - Dynamic
Converter
Dynamic Converter enables on-demand, ad-hoc publishing of native business
documents (255 formats) to HTML, XML or wireless formats.

• Create Template Selection Rules

• Predetermines the conversion and layout templates
• Upon submission, content is converted and cached.
• Cache location:
<stellent_dir>\weblayout\groups\<security_group>\documents\<typ
e>\~export\<content id>~<rev>~<template name>~<layout name>
 5) Adjust Add-ons - Site Studio
Site Studio automates Web site creation and contribution methodologies.
Additionally, Site Studio enables business users to contribute content in a user-
friendly WYSIWYG form-based environment with regional-level security and in-
context preview, workflow, and approval.

• Analyze page architecture

• Turn on tracing and watch the output
• Search, Database, Script Errors
• Optimize the queries in dynamic lists
• Simple
• Operators (Matches)
• Indexed
• Avoid using time and Substring
 5) Adjust Add-ons - Site Studio (cont.)

• Use front side caching

• Reverse proxy server
• Use IdocScript caching
• Optimize services executed on pages
• Consolidate actions. Remove unnecessary work/data.
• Client-side vs. server-side page construction
• e.g. out-of-the-box fragments use client-side mechanisms to
build navigation.
• Static lists vs. dynamic lists
• Static Lists – XML content
• Dynamic Lists – Search Queries
5) Adjust Add-ons – Inbound Refinery

Enables the automatic publishing of native business content to web viewable

PDF.

• 10gR3 Refinery is another Content Server

• Separate the refinery machine from content
server machine
• Run multiple refineries and make use of
selective queue processing
• Analyze formats being used and adjust rules
5) Adjust Add-ons - Content Tracker

Oracle Content Tracker monitors content and server access and performs data
extraction to populate a database repository. The data is then viewable through
out of the box reports, as well as standalone reporting tools (e.g. Crystal Reports).

• Content Tracker is performance intensive

• Consider changing reduction schedule
• Discover what content is “inactive”
• Analyze searches being performed
• Helps optimize metadata usage
5) Adjust Add-ons - Content
Integration Suite (CIS)
Oracle Content Integration Suite provides for a reliable and scalable integration
infrastructure for integrations with Enterprise Applications.

• Cache! Cache! Cache!

• Service-level caching exists, but out-of-the-box only
caches DOC_INFO
• Cache data that is public information and not tied to
users or security. e.g. Announcements
• Caching mechanisms
• Page-level HTTP caching
• App server specific caching
• OSCache toolkit
 Professional Services

• Oracle Professional Services is best suited to assist with

performance issues
• Oracle Support is not capable of analyzing and tuning of
performance issues
• Support can assist with pointing customers in general direction
• Support can initiate the request for Consulting Services
• Slow does not always equal Broken
• May not be a patch or component that can help the issue
• May need hardware or architecture solution
 Questions?

• Cordell Melgaard
Principal Technical Support Engineer
Global Customer Service
• cordell.melgaard@oracle.com
• Jason Ganske
Principal Technical Support Engineer
Global Customer Service
• jason.ganske@oracle.com

• Thank you!