Evolving FPGA Verification Capabilities
Code Coverage
Ray Salemi
Application Engineer Consultant
academy@mentor.com
www.verificationacademy.com
Overview
• In this session we will
• Discuss the use of code coverage
- An easy, automated, measurable metric to help assess
the Verification Complete milestone
• Examine different types of code coverage
- Introduce the use of exclusions to help achieve 100%
coverage
Target Audience
• Novice FPGA engineer or student
• Who are interested in learning more
verification techniques
• Experienced FPGA engineers
• Who are considering introducing verification on a future
project
• FPGA Project Managers
• Whose goal is to improve the capability maturity of their
organization through industry best practices and processes
Seven Steps to the Ultimate Testbench
Functional Coverage
Automatic Stimulus
Self-Checking Testbench
Transactions
Assertions
Test Planning
Code Coverage
Code Coverage
• The old adage goes…
• If you haven’t tested it,
then it’s likely broken!
• Code coverage is an
automated technique
that identifies code that
hasn’t been tested
Code coverage is automated and easy!
• Does not require a change
to your current design and
simulation approach
• A command line switch
applied at compilation and
another at simulation
Code Coverage Goal
• Some say 95%
• If your roof is 95% waterproof, would you be pleased?
• 100% is not obtainable
• Without exclusions!
• e.g., unreachable default
branch in a case statement
• The review process is invaluable!
Types of Code Coverage
• Statement
• Where all statements executed?
• Branch
• Did all branches exercised?
• Condition
• Did we take all branches for all reasons?
• Expression
• Where sub-expressions of a concurrent assignment tested?
• Finite State Machine
• Where all states and transitions visited?
The TinyCache State-Machine Example Design
• We now introduce a simple example
• A state-machine controller called TinyCache
- Sits between a CPU and a memory
- Simple write-through cache
CPU2CACHE CACHE2MEM
• We use this example to
demonstrate the five types Cache
CACHE2CPU MEM2CACHE
of code coverage
• Goal: ensure that our design is thoroughly simulated
TinyCache Flow Chart
START
Case
current_state
RESET HIT WRITE MISS OTHERS
next_state <= HIT; next_state <= HIT; next_state <= RESET;
F
(cpuwait_int=‘1’) next_state <= HIT;
F
T (cpuwait_wr=‘1’)
T
next_state <= MISS; next_state <= HIT;
next_state <= WRITE;
End Case
END
Statement Coverage
• Goal: Verify that every
statement has been
executed during simulation
• A statement is a line of
code that ends in a
semicolon
• e.g., assignments, task calls,
procedure calls
Statement Coverage Report Example
Branch Coverage
• Goal: Verify that every CASE
and IF branch alternative has
been exercised
Branch Coverage Report Example
Branch Coverage Report Example
Condition Coverage
• Goal: Verify that every CASE and IF condition
sub-expression is evaluated
• Consider the following:
if (A || B)
inc_sum = sum+1;
• Neither branch nor statement
coverage tells us which
sub-expression triggered
the evaluation
Complex Condition Example
• Consider the TinyCache example
always @(key_ram[cache_ram] or cpu_address or cpu_rd)
if ( ( (key_ram[cache_address] != cpu_address) ||
(invalid[cache_address]) ) && cpu_rd )
cpuwait = 1;
else
cpuwait = 0;
• The signal cpuwait tells the CPU that there is a
cache miss
Complex Condition Example
• The condition expression can be viewed as
(A or B) and C
A—Is this a cache miss?
(key_ram[cache_address] != cpu_address)
B—Is the cache entry invalid?
(invalid[cache_address])
C—Are we doing a read operation
(cpu_rd)
• Branch coverage tells us that we executed both
hits and misses, but not why we executed them
Condition Coverage Report Example
Expression Coverage
• Goal: Ensure that all sub-expression are exercised
• RTL contains two types
of statements
• Procedural statements
• Concurrent assignments
• Expression coverage helps
identify sub-expressions within concurrent
assignments that have not been exercised
Complex Continuous Assignment Example
• Consider the TinyCache example
assign cpuwait = ( (key_ram[cache_address] != cpu_address) ||
(invalid[cache_address]) ) && cpu_rd;
Expression Coverage Report Example
Finite State Machine Coverage
• Goal: Ensure that we
have visited every state
and followed every
transition in a finite
state machine
Finite State Machine Example
• Consider the TinyCache simple example
(cpuwait_int = ‘1’)
MISS
RESET
HIT
WRITE
(cpu_wr = ‘1’)
Finite State Machine Coverage Report Example
Toggle Coverage
• For completeness, we mention toggle coverage
• Goal: Ensure signals transition
from 0->1 and 1->0
• Generally not used in RTL
simulation, with the exception
of possibly block or cluster interfaces
Exclusions
• Designate portions of the code that need not be covered
• Block Exclusion
- For example, debugging code that you want to excluded in the final
coverage report
• Default Exclusion
- Some simulators, such as ModelSim, have the option to automatically
exclude all default and OTHERS alternatives in case statements
• Conditional and Expression Row Exclusion
- For example, a mode bit set to 1 that never changes
• FSM State and Transition Exclusion
- My want to exclue certain error states or unused transitions
Code coverage, not perfect, but certainly valuable!
• No metric is perfect
• Code coverage measures
stimulus activation
• Not propagation
• Code coverage not linked to
design intent
• Even with these limitations,
code coverage helps identify
what has not been tested!
Summary
• In this session we
• Discussed the use of code coverage
- An easy, automated, measurable metric to help assess
the Verification Complete milestone
• Examined different types of code coverage
- Introduce the use of exclusions to help achieve 100%
coverage
Resources
• Mentor Graphics Training
• Scalable Verification Courses
- A wide range of instructor led classes
- Located in public training centers in many major cities or onsite at
your workplace
- Web-based events with live instructors are also available.
• Mentor Graphics Consulting
• Questa® Verification Methodology JumpStart
• Knowledge-Sourcing Model
- Infuse knowledge into your organization while addressing your
immediate product development challenges
For More Information
• FPGA Simulation
Ray Salemi,
Boston Light Press,
March 2, 2009
www.fpgasimulation.com
Evolving FPGA Verification Capabilities
Code Coverage
Ray Salemi
Application Engineer Consultant
academy@mentor.com
www.verificationacademy.com
