CONFIGURING AND REBUILDING

THE FORECAST SYSTEM METRICS DASHBOARD IN GRAFANA

TECHNICAL DOCUMENTATION
For the Naval Research Laboratory (NRL)
Team: Alberto Lucas, Josh Hansen, Michael Garber
_______________________
California State University, Monterey Bay
CST499 - Computer Science Capstone
Tim Whitcomb - Product Client
Bude Su - Faculty Capstone Advisor
_______________________
Summer 2022
 2

Table of Contents

Introduction 3
Data Source Configuration 3
Prebuilt Dashboard Configuration 6
Rebuilding the Dashboard 8
Resizing and Repositioning Panels 22
View Modes 23
Viewing Options 24
 3
I. Introduction
As outlined in the Prometheus and Docker Compose documentation here, Grafana is a metrics
and data visualization tool that allows for a myriad of types of data sources and visual outputs
and graphs. The steps below outline how the Grafana Dashboard, the FSMD.json file located
here, was configured to visualize random test data funneled through Prometheus. The
information below includes details to create a new dashboard, but also outlines how the existing
dashboard can be reconfigured to fit another data source. The steps below assume that Grafana is
installed on the local system in question. Typically, as in the configuration mentioned above,
Grafana can be accessed from this URL: http://localhost:3000/

II. Data Source Configuration

First, it’s necessary to ensure that the data source Grafana is using is configured correctly. In the
test setup through Docker Compose outlined here, the Docker Compose config file,
docker-compose.yml, contains a volume that presets the data source:

- ./provisioning/datasources:/etc/grafana/provisioning/datasources

This copies the following prometheus_ds.yml file to the location in the Grafana volume located
above:

datasources:
- name: ${DS_PROMETHEUS}
access: proxy
type: prometheus
url: http://prometheus:9090
isDefault: true

Make sure that “url” correctly matches the output URL for your Prometheus setup.

To utilize a preset data source file like this in the same way with our preset dashboard, the
“name” above must also match the “uid” value in the FSMD.json file. Look for a datasource
with the type prometheus. Copy the character string next to the “uid” entry that matches the
“name” above. We chose “${DS_PROMETHEUS}” because, when we had the data source name
set to “PROMETHEUS,” this string kept overwriting the uid in the JSON file whenever we
saved a new copy of the dashboard.
 4

In any case, if the steps above are followed and Docker provides Grafana with the preset data
source in a volume as listed, this data source will be available to Grafana automatically at
runtime, without further data source configuration required.

Alternatively, you may manually set the data source using the following steps:

1. Along the left edge of the Grafana window there is a cluster of 5 icons at the top and 4
icons at the bottom. Mouse over the cog wheel icon to show the configuration menu.
Click Data Sources.

2. Click the add data source button and select Prometheus

3. Choose a name that matches the uid in FSMD.json, “${DS_PROMETHEUS}”, or

modify FSMD.json to match whatever name you choose in the uid fields, as outlined
above. If you are building a new dashboard from scratch, choose any name.
 5

4. Make sure that the HTTP URL correctly matches the output URL for your Prometheus
setup.

5. Keep the Access method to the default “Server” (previously Proxy).

6. Keep all other defaults and save the data source.

 6
III. Prebuilt Dashboard Configuration
If using the preconfigured FSMD.json dashboard, use the following steps to ensure the
dashboard works with your Prometheus source:

1. Import the FSMD.json file to your Grafana though the Dashboards menu on the top left.

2. If your data source name is set differently than the uid values in the dashboard for the
Prometheus data source, make sure to fix this by editing the JSON file directly. You can
do find and replace all for the string “${DS_PROMETHEUS}”.
 7
3. Once relaunching the dashboard, you should see your data source chosen for each panel
appear as selected:

4. If you have an I/O Fraction metric, ensure that each metric that utilizes this metric is set
to use the actual name of the metric that is being pushed from your Prometheus. Select
the header bar at the top of each panel, and search for “I_O_Fraction” in the queries
listed. Replace each instance with the correct name.

5. Repeat the above for the “Initial_RMS_pressure_tendency" and "Total_Runtime", saving

each panel as you go.

6. Refresh the dashboard and configure the time range on the top right to match the desired
span for the data coming from your Prometheus.

7. You should now see your data being populated in the prebuilt dashboard.
 8
IV. Rebuilding the Dashboard
The following outlines how to make a similar dashboard to the prebuilt one here. Make sure you
have first followed the steps at “Data Source Configuration” to have your Prometheus data
source ready to go.

Creating a new dashboard and panel

1. First, third icon in the top left group is the dashboards menu. Mouse over it and click
New dashboard.

2. Click “Add a new panel”. Under the placeholder graph is a drop down menu for the
available data sources. Select the data source that you added.
 9

3. Choose the desired name for the panel on the top right, and a description if desired.

4. Click the “+ Query” button. Click on the “A” and edit the name of the query to represent
the metric query you will be choosing.

5. Each of the custom panels in the dashboard outlined below will start with the steps above,
follow the appropriate steps below for the type of panel you want to add.
 10
Most recent values panel

1. To reproduce the panel we made that shows the most recent values for all metrics, create
a new panel and set the type of panel on the top right to “Stat”.

2. Add a query that represents each metric that should appear in this overhead, most recent
values panel. We created three queries, by selecting each of our three metrics,
“I_O_Fraction”, “Total_Runtime”, and “Initial_RMS_pressure_tendency”.
 11
3. Add an operation by clicking “+ Operation” on each query, and add the “Greater or equal
to” condition. Set this value to zero, or whatever minimum represents a valid value for
each of your metrics.

4. In the side panel, under “Value options” add “Last*” to the Calculation field to ensure
that the values selected are the most recent:

5. In the side panel under “Stat styles”, we recommend setting “Graph mode” to “None” to
prevent a line graph of most recent values from appearing behind the stats in the panel.
We also recommend setting “Text alignment” to “Center” to appear better with multiple
values in an enlarged panel.

6. In the side panel under “Standard options”, set “Color scheme” to “From thresholds (by
value)” to have the colors of each metric value apply the thresholds set below.

7. In the side panel, choose “Overrides” near the top and add the following overrides for
each metric:
a. Select the metric name in the “Fields Returned by Query” menu.
b. Click “+ Add Override Property” and select “Display name”. Choose the desired
name for each metric to appear in the panel.
c. Add another override property for “Standard options > Unit” for each metric,
setting the desired metric.
 12
d. Add another override property for “Thresholds” for each metric, setting the
desired threshold values and colors. Note that each threshold represents the value
above which, the value shown in the dashboard will have the chosen color, as
long as the value is less than the next highest threshold. We recommend setting
the following thresholds:
i. Set the lowest threshold to the minimum valid value for each metric. This
was zero for each of our test metrics and was red.
ii. Then, the next threshold we set to be equal to the median minus two
standard deviations, set to yellow.
iii. The next is median minus one standard deviation, set to green.
iv. The next, median plus one standard deviation and set to yellow.
v. Finally, median plus two standard deviations and set to red.
vi. Thus, values greater than two standard deviations off of the means would
be represented as red. Values less than two stddev off of the median, but
more than one stddev off, would be yellow. Values within one stddev of
the median will be green.

8. Save the queries, settings, and overrides by clicking “Apply” on the top right. The panel
should be complete.
 13
Single metric historical panel

1. To reproduce the panels we made that show historical values for a single metric at a time,
create a new panel and set the type of panel on the top right to “Time series”. Repeat this
and all the steps below for each desired metric.

2. Click the “+ Query” button. Give the query a descriptive name representing the metric to
be used in this panel.

3. From the Metric menu, find and choose the metric to be used in this panel.

4. Add an operation on this panel for “Greater or equal to”. Set this value to zero, or
whatever minimum represents a valid value for each of your metrics. If missing or null
data paired with timestamps is passed in from Prometheus, this will be represented as a
missing bar in the graph.
 14
5. Create another query, giving it a descriptive name that represents a rolling average and
the length of the rolling average you will choose, such as “Rolling Average (4d)”.

6. Choose “Code” on top right of the query section to set a raw query.

7. In the Metrics browser enter the following:

avg_over_time(([metric name] > 0)[4d:])

8. This query will generate a rolling average over the range set. Replace “[metric name]”
with the raw metric name. This should be the same as that appearing under “Metric” for
the first query in this panel.

9. Make sure to replace “4d” in the query to represent the range that you would like the
rolling average to use. To change the time frame of the rolling average you can change
the number to whatever value you wish. To change the unit of time refer to this list:

● Seconds s
● Minutes m
● Hours h
● Days d
● Weeks w
● Months M
● Quarters Q
● Years y

10. In the side panel under “Graph styles”, we recommend selecting bars for a single metic
historical view. We also recommend centered bar alignment and a line width of 2 for a 30
day view. Make sure to set “Fill opacity” to maximum. The rest of the options can be left
to default.
 15

11. Make sure to set the correct “Unit” for the given metric under “Standard options” in the
sidebar.

12. Also under “Standard options”, set “Display name” to the desired metric name to be
shown in the legend for the actual metric values (Y-axis), and “Color scheme” to “Single
color”. We chose blue as a neutral value and to not conflict visually with thresholds.

13. Set desired thresholds in the side panel. Slightly different from the values set above, we
recommend setting the following thresholds to allow for meaningful threshold lines to be
superimposed on the single metric graph.
a. Set the lowest threshold to be equal to the median minus two standard deviations,
set to red.
b. The next is median minus one standard deviation, set to yellow.
c. The next is the value of the median, set to green.
d. The next, median plus one standard deviation and set to yellow.
e. Finally, median plus two standard deviations and set to red.
 16
f. Thus, there will be red lines representing the bounds of two stddev from the
median, yellow lines showing bounds of one stddev from the median, and a green
line for the median itself.

14. Make sure to set “Show thresholds” in the “Thresholds” section of the side panel to “As
lines”. This will ensure that the lines above appear as expected.

15. If filled regions are desired instead, then thresholds will need to be set as follows instead
in order to logically shade the colored regions correctly:
a. Set the lowest threshold to the minimum valid value for each metric, set to red.
For the FSMD test metrics this would be zero for each of our test metrics.
b. Then, the next threshold we set to be equal to the median minus two standard
deviations, set to yellow.
c. The next is median minus one standard deviation, set to green.
d. Optionally, another threshold can be set to the value of median itself, set to green
e. Next, set a threshold to the median plus one standard deviation and set to yellow.
f. Finally, set a threshold to the median plus two standard deviations and set to red.
 17

16. In the side panel, choose “Overrides” near the top and add the following overrides for the
second query:
a. Select the second query name in the “Fields Returned by Query” menu, which
represents the rolling average.
b. Click “+ Add Override Property” and select “Display name”. Choose the desired
name for each metric to appear in the panel. We chose “Rolling average (4d)”.
c. Add another override property for “Standard options > Color scheme”, select
“Single color” and choosing a distinct color that will not clash with the bars or the
threshold lines. We recommend purple or pink.
d. Add an override for “ Graph styles > Style” and choose “Lines”.
e. Finally, add an override for “ Graph styles > Fill opacity” and set the value to 0.
 18
17. To save each panel, select the blue “Apply” button in the top right corner.

Data acquired panel

The final panel type is a smaller module that shows whether or not data was acquired for each
time segment, for each metric. As configured, in our prebuilt dashboard and in the steps below, a
less than zero value, paired with an actual timestamp, represents a missing value. This will still
need to be fed into Grafana from Prometheus for these modules to work correctly. For setups
involving null data paired with timestamps being passed in, a different configuration would be
required than that below, involving overrides on null and non-null values.

1. To reproduce these panels that show whether or not data was acquired for a given metric,
create a new panel and set the type of panel on the top right to “Time series”. Repeat this
and all the steps below for each desired metric.
 19

2. Click the “+ Query” button. This query will represent the condition that data was indeed
acquired. We named this query “True”.

3. From the Metric menu, find and choose the metric to be used in this panel.

4. Set a “Greater or equal to” operation on the metric and this time check the box for
“Bool”. This operation will output a value of 0 or 1 based on whether or not each value is
less than zero.

5. Next, add another operation for “Equal to” and set it to 1. This means that the entire
query will filter for only values that are greater than or equal to zero.

6. Add an inverse query that is exactly the same, and uses the same metric, but name it
“False” and set the “Equal to” field to 0. This means that the entire query will filter for
only values that are less than zero.
 20

7. In the side panel under “Graph styles”, set “Style” to “Points”. We recommend a point
size of 6.

8. In the side panel under “Standard options”, set “Unit” to “Boolean / Yes / No”.

9. In the side panel under “Standard options”, set “Color scheme” to “Single color”, and set
to green.

10. In the side panel, choose “Overrides” near the top and add an override, choosing “Fields
Returned by Query”.
 21
a. Select the “False” query in the menu.
b. Add an override property for “Standard options > Color scheme”. Set this to
“Single color” and choose red.

11. To save each panel, select the blue “Apply” button in the top right corner.
 22
V. Resizing and Repositioning Panels
To reposition a panel on the dashboard, mouse over the top portion of the panel.

When your mouse changes to 4 arrows pointing different directions, you can click and drag that
panel to wherever you wish.

To resize a panel, move the cursor to the bottom right corner until the cursor changes to a double
sided arrow pointing top-left and bottom-right.

Click and drag to adjust the height and width of the panel as you wish.

Clicking on the title of the panel will open a menu with the following options:

● View Full screen view of the current panel.

● Edit: Exposes all of the functional options of the panel.
● Share: Has several different options to share the panel.
● Explore: Displays the Grafana query strings and window to display the info in a few
different graphs.
 23
● Inspect: Displays the data values being displayed.
● Duplicate: Creates a copy of this panel on this dashboard.
● Copy: Copies the panel to the clipboard, allowing a copy of the panel to be created
elsewhere.
● Create Library Panel: Adds the current panel to a library allowing it to be easily recreated
later.
● Remove: Deletes the panel.

VI. View Modes

In the top-right corner of the dashboard is the cycle view mode button. Pushing this button will
cycle between three different view modes. Reducing how much of the Grafana user-interface is
shown along the top and left edges of the window.

Pressing the Escape key will reveal the full UI regardless of what view mode you happen to be
in.
 24
VII. Viewing Options
There is an option to set the background transparency in each panel. The graph on the left had
background transparency disabled. The graph on the right has background transparency enabled
The difference is mostly aesthetic, however disabling this option better defines the borders of the
panel for resizing purposes.

● From the panel title menu select the edit option.

● The Transparent background option toggle is in the side panel under the description field.

Finally if you would like to color the regions of the graph denoting the thresholds, click the edit
option from the title bar menu

.
In the side panel select an option from the show thresholds menu
● Off where thresholds are not displayed.
● As lines, only the lines are displayed.
● As filled regions, the areas are colored with no lines displayed.
● As filled regions and lines, both lines and regions are displayed.