FortiOS-7.4.5-Administration - Guide (2) - Parte4

SD-WAN
2. Drill down on an application, such as YouTube, then select the Sessions tab.
To verify applications identified by Application Control in SD-WAN:
# diagnose sys sdwan internet-service-app-ctrl-list
Steam(16518 4294838108): 23.6.148.10 6 443 Thu Apr 15 08:51:54 2021

Netflix(18155 4294837589): 54.160.93.182 6 443 Thu Apr 15 09:13:25 2021
Netflix(18155 4294837589): 54.237.226.164 6 443 Thu Apr 15 10:04:37 2021
Minecraft(27922 4294837491): 65.8.232.41 6 443 Thu Apr 15 09:12:19 2021
YouTube(31077 4294838537): 74.125.69.93 6 443 Thu Apr 15 08:52:59 2021
YouTube(31077 4294838537): 108.177.112.136 6 443 Thu Apr 15 09:33:53 2021
YouTube(31077 4294838537): 142.250.1.93 6 443 Thu Apr 15 10:35:13 2021
...
DSCP tag-based traffic steering in SD-WAN
Differentiated Services Code Point (DSCP) tags can be used to categorize traffic for quality of service (QoS). SD-WAN
traffic steering on an edge device can be provided based on the DSCP tags.
This section provides an example of using DSCP tag-based traffic steering using secure SD-WAN. Traffic from the
customer service and marketing departments at a headquarters are marked with separate DSCP tags by the core switch
and passed to the edge FortiGate. The edge FortiGate reads the tags, then steers traffic to the preferred interfaces
based on the defined SD-WAN rules.
FortiOS 7.4.5 Administration Guide 901

Fortinet Inc.
SD-WAN
VoIP and social media traffic are steered. VoIP traffic from the customer service department is more important than
social media traffic. The edge FortiGate identifies the tagged traffic based on SD-WAN rules then steers the traffic:
l VoIP traffic is marked with DSCP tag 011100 and steered to the VPN overlay with the lowest jitter, to provide the
best quality voice communication with the remote PBX server.
l Social media traffic is marked with the DSCP tag 001100 and steered to the internet connection with the lowest cost.
The following is assumed to be already configured:
l Two IPsec tunnels (IPsec VPN on page 2029):
l Branch-HQ-A on Internet_A (port 1)
l Branch-HQ-B on Internet_B (port 5)
l Four SD-WAN members in two zones (Configuring the SD-WAN interface on page 789):
l Overlay zone includes members Branch-HQ-A and Branch-HQ-B
l virtual-wan-link zone includes members Internet_A and Internet_B
Internet_A has a cost of 0 and Internet_B has a cost of 10. When using the lowest cost strategy, Internet_A will
be preferred. Both members are participants in the Default_DNS performance SLA.
l A static route that points to the SD-WAN interface (Adding a static route on page 791).
l Two firewall policies:
Name SD-WAN-OUT Overlay-OUT

Fortinet Inc.
SD-WAN
From port3 port3
To virtual-wan-link Overlay
Source all all
Destination all all
Schedule always always
Service all all
Action Accept Accept
NAT enabled enabled
After the topology is configured, you can proceed with the configuration of the edge FortiGate:
l Configuring SD-WAN rules on page 903
l Results on page 905
Configuring SD-WAN rules
Configure SD-WAN rules to govern the steering of DSCP tag-based traffic to the appropriate interfaces. Traffic is steered
based on the criteria that are configured in the SD-WAN rules.
In this example, three SD-WAN rules are configured to govern DSCP tagged traffic:
l VoIP-Steer for VoIP traffic.
l Facebook-DSCP-steer for Social media traffic.
l All-traffic for all of the Other web traffic.
After configuring the rules, go to Network > SD-WAN and select the SD-WAN Rules tab to check the rules.
VoIP traffic
VoIP traffic is steered to the Overlay zone.

DSCP values are usually 6-bit binary numbers that are padded with zeros at the end. VoIP traffic with DSCP tag 011100
will become 01110000. This 8-bit binary number is represented in its hexadecimal form, 0x70, as the type of service bit
pattern (tos) value. The type of service evaluated bits (tos-mask) hexadecimal value of 0xf0 (11110000 in binary) is
used to check the four most significant bits in the tos value. The four most significant bits of the tos (0111) are used to
match the first four bits of the DSCP tag. Only the non-zero bit positions in the tos-mask are used for comparison; the
zero bit positions are ignored.
The Best quality (priority mode) strategy is used to select the preferred interface, with the Quality criteria (link-
cost-members) set to Jitter. The interface with the lowest amount of jitter is selected. For more information about
configuring SD-WAN rules with the Best Quality strategy, see Best quality strategy on page 870.
To configure the rule for DSCP tagged VoIP traffic using the CLI:
config sys sdwan

config service
edit 5
set name "VoIP-Steer"
set mode priority

Fortinet Inc.
SD-WAN
set tos 0x70

set tos-mask 0xf0
set dst "all"
set health-check "Default_DNS"
set link-cost-factor jitter
set priority-members 4 3
next
end
end
Social media traffic
Social media traffic is steered to the virtual-wan-link zone.

DSCP values are usually 6-bit binary numbers that are padded with zeros at the end. Social media traffic traffic with
DSCP tag 001100 will become 00110000. This 8-bit binary number is represented in its hexadecimal form, 0x30, as the
tos value. The tos-mask hexadecimal value of 0xf0 (11110000 in binary) is used to check the four most significant bits
in the tos value. The four most significant bits of the tos (0011) are used to match the first four bits of the DSCP tag. Only
the non-zero bit positions in the tos-mask are used for comparison; the zero bit positions are ignored.
The Manual (manual mode) strategy is used to select the preferred interface. Internet_B (port5, priority member 2) is set
as the preferred interface to steer all social media traffic to. For more information about configuring SD-WAN rules with
the manual strategy, see Manual strategy on page 867.
To configure SD-WAN rule for DSCP tagged social media traffic using the CLI:
config system sdwan

config service
edit 3
set name "Facebook-DSCP-steer"
set mode manual
set tos 0x30
set tos-mask 0xf0
set dst "all"
next
end
end
Other web traffic
Other web traffic is steered to the virtual-wan-link zone.

The Lowest Cost (SLA) strategy (sla mode) is used to select the preferred interface. The interface that meets the
defined SLA targets (Default_DNS in this case) is selected. If there is a tie, the interface with the lowest cost is selected,
Internet_A (port1) in this case.
For more information about configuring SD-WAN rules with the Lowest Cost (SLA) strategy, see Lowest cost (SLA)
strategy on page 874.
To configure SD-WAN rule for all other web traffic using the CLI:
config system sdwan

config service
edit 2

Fortinet Inc.
SD-WAN
set name "All-traffic"

set mode sla
set dst "all"
config sla
edit "Default_DNS"
set id 1
next
end
next
end
end
Results
These sections show the function of SD-WAN with respect to DSCP tagged traffic steering, and can help confirm that it is
running as expected:
l Verifying the DSCP tagged traffic on FortiGate on page 905
l Verifying the service rules on page 906
l Verifying traffic steering on the SD-WAN rules on page 907
l Verifying that steered traffic is leaving from the expected interface on page 907
Verifying the DSCP tagged traffic on FortiGate
Packet sniffing is used to verify the incoming DSCP tagged traffic. See Using the FortiOS built-in packet sniffer for more
information.
Wireshark is used to verify that VoIP traffic is tagged with the expected DSCP tag, 0x70 or 0x30.
VoIP traffic marked with DSCP tag 0x70:
# diagnose sniffer packet any '(ip and ip[1] & 0xfc == 0x70)' 6 0 l

Fortinet Inc.
SD-WAN
Web traffic marked with DSCP tag 0x30:
# diagnose sniffer packet any '(ip and ip[1] & 0xfc == 0x30)' 6 0 l
Verifying the service rules
To check that the expected DSCP tags and corresponding interfaces are used by the SD-WAN rules to
steer traffic:
# diagnose sys sdwan service4
Service(5): Address Mode(IPV4) flags=0x0

Gen(1), TOS(0x70/0xf0), Protocol(0: 1->65535), Mode(manual)
Members:
1: Seq_num(4 Branch-HQ-B), alive, selected
Dst address:
0.0.0.0-255.255.255.255

Gen(1), TOS(0x30/0xf0), Protocol(0: 1->65535), Mode(manual)
Members:
1: Seq_num(2 port5), alive, selected
Dst address:
0.0.0.0-255.255.255.255

Gen(1), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla), sla-compare-order
Members:
1: Seq_num(1 port1), alive, sla(0x1), cfg_order(0), cost(0), selected
Dst address:
0.0.0.0-255.255.255.255

Fortinet Inc.
SD-WAN
Verifying traffic steering on the SD-WAN rules
Go to Network > SD-WAN and select the SD-WAN Rules tab to check the Hit Count on the SD-WAN interfaces.
Verifying that steered traffic is leaving from the expected interface
To confirm that web traffic (port 443) flows through the correct underlay interface members, and VoIP traffic flows
through the correct overlay interface members, go to Dashboard > FortiView Policies and double click on the policy
name.
Web traffic is expected to leave on Interface_A (port1) or Interface_B (port5):
VoIP traffic is expected to leave on the preferred VPN_B_Tunnel (Branch-HQ-B) interface:

Fortinet Inc.
SD-WAN
ECMP support for the longest match in SD-WAN rule matching
The longest match SD-WAN rule can match ECMP best routes. The rule will select the egress ports on ECMP specific
routes, and not the less specific routes, to transport traffic.
The service mode determines which egress port on the ECMP specific routes is selected to forward traffic:
l Manual (manual): The first configured alive port is selected.
l Best Quality (priority): The best quality port is selected.
l Lowest Cost (sla): The first configured or lower cost port in SLA is selected.
Example
By default, SD-WAN selects the outgoing interface from all of the links that have valid routes to the destination. In some
cases, it is required that only the links that have the best (or longest match) routes (single or ECMP) to the destination
are considered.
In this example, four SD-WAN members in two zones are configured. The remote PC (PC_2 - 10.1.100.22) is accessible
on port15 and port16, even though there are valid routes for all of the SD-WAN members. A single SD-WAN service rule
is configured that allows traffic to balanced between all four of the members, but only chooses between port15 and
port16 for the specific 10.1.100.22 address.
A performance SLA health check is configured to monitor 10.1.100.2. An SD-WAN service rule in Lowest Cost (SLA)
mode is configured to select the best interface to steer the traffic. In the rule, the method of selecting a member if more
than one meets the SLA (tie-break) is configured to select members that meet the SLA and match the longest prefix
in the routing table (fib-best-match). If there are multiple ECMP routes with the same destination, the FortiGate will
take the longest (or best) match in the routing table, and choose from those interface members.
To configure the SD-WAN:
config system sdwan

config zone
edit "virtual-wan-link"
next
edit "z1"
next
end
config members
edit 1
set interface "port1"
set gateway 172.16.200.2
next
edit 2
set interface "dmz"
next

Fortinet Inc.
SD-WAN
edit 3
set zone "z1"
next
edit 4
set zone "z1"
next
end
config health-check
edit "1"
set server "10.1.100.2"
set members 0
config sla
edit 1
next
end
next
end
config service
edit 1
set name "1"
set mode sla
set dst "all"
set src "172.16.205.0"
config sla
edit "1"
set id 1
next
end
set priority-members 1 2 3 4
set tie-break fib-best-match
next
end
end
To check the results:
1. The debug shows the SD-WAN service rule. All of the members meet SLA, and because no specific costs are
attached to the members, the egress interface is selected based on the interface priority order that is configured in
the rule:
FGT_A (root) # diagnose sys sdwan service4
Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla

Members(4):
1: Seq_num(1 port1), alive, sla(0x1), gid(0), cfg_order(0), cost(0), selected
2: Seq_num(2 dmz), alive, sla(0x1), gid(0), cfg_order(1), cost(0), selected
Src address(1):
172.16.205.0-172.16.205.255

Fortinet Inc.
SD-WAN
Dst address(1):
0.0.0.0-255.255.255.255
2. The routing table shows that there are ECMP default routes on all of the members, and ECMP specific (or best)
routes only on port15 and port16:
FGT_A (root) # get router info routing-table static
Routing table for VRF=0
S* 0.0.0.0/0 [1/0] via 172.16.200.2, port1
[1/0] via 172.16.208.2, dmz
[1/0] via 172.16.209.2, port15
[1/0] via 172.16.210.2, port16
S 10.1.100.22/32 [10/0] via 172.16.209.2, port15
[10/0] via 172.16.210.2, port16
Because tie-break is set to fib-best-match, the first configured member from port15 and port16 is selected to
forward traffic to PC_2. For all other traffic, the first configured member from all four of the interfaces is selected to
forward traffic.
3. On PC-1, generate traffic to PC-2:
ping 10.1.100.22
4. On FGT_A, sniff for traffic sent to PC_2:

# diagnose sniffer packet any 'host 10.1.100.22' 4
interfaces=[any]
filters=[host 10.1.100.22]
2.831299 port5 in 172.16.205.11 -> 10.1.100.22: icmp: echo request
2.831400 port15 out 172.16.205.11 -> 10.1.100.22: icmp: echo request
Traffic is leaving on port15, the first configured member from port15 and port16.
Override quality comparisons in SD-WAN longest match rule matching
In SD-WAN rules, the longest match routes will override the quality comparisons when all of the specific routes are out of
SLA.
With this feature in an SD-WAN rule:
l Lowest Cost (sla): Even though all of the egress ports on specific routes (longest matched routes) are out of SLA,
the SD-WAN rule still selects the first configured or lower-cost port from the egress ports to forward traffic.
l Best Quality (priority): Even though the egress ports on specific routes (longest matched routes) have worse
quality that all other ports on less specific routes, the SD-WAN rule still selects the best quality port from the ports on
specific routes to forward traffic.
This features avoids a situation where, if the members on specific routes (longest matched routes) are out of SLA or
have worse quality, the traffic might be forwarded to the wrong members in SLA (higher quality) on the default or
aggregate routes.

Fortinet Inc.
SD-WAN
Example
In this example, four SD-WAN members in two zones are configured. The remote PC (PC_2 - 10.1.100.22) is accessible
on port15 and port16, even though there are valid routes for all of the SD-WAN members. A single SD-WAN service rule
is configured that allows traffic to balanced between all four of the members, but only chooses between port15 and
port16 for the specific 10.1.100.22 address. If neither port15 nor port16 meet the SLAs, traffic will be forwarded on one of
these interfaces, instead of on port1 or dmz.
A performance SLA health check is configured to monitor 10.1.100.2. An SD-WAN service rule in Lowest Cost (SLA)
mode is configured to select the best interface to steer the traffic. In the rule, the method of selecting a member if more
than one meets the SLA (tie-break) is configured to select members that meet the SLA and match the longest prefix
in the routing table (fib-best-match). If there are multiple ECMP routes with the same destination, the FortiGate will
take the longest (or best) match in the routing table, and choose from those interface members.
To configure the SD-WAN:
config system sdwan

config zone
next
edit "z1"
next
end
config members
edit 1
next
edit 2
set interface "dmz"
next
edit 3
set zone "z1"
next
edit 4
set zone "z1"
next
end
config health-check
edit "1"
set server "10.1.100.2"
set members 0
config sla

Fortinet Inc.
SD-WAN
edit 1
next
end
next
end
config service
edit 1
set name "1"
set mode sla
set dst "all"
set src "172.16.205.0"
config sla
edit "1"
set id 1
next
end
next
end
end
To check the results:
1. The debug shows the SD-WAN service rule. Both port15 and port16 are up, but out of SLA:
FGT_A (root) # diagnose sys sdwan service4
Members(4):
2: Seq_num(2 dmz), alive, sla(0x1), gid(0), cfg_order(1), cost(0), selected
Src address(1):
172.16.205.0-172.16.205.255
Dst address(1):
0.0.0.0-255.255.255.255
2. The routing table shows that there are ECMP default routes on all of the members, and ECMP specific (or best)
routes only on port15 and port16:
FGT_A (root) # get router info routing-table static
S* 0.0.0.0/0 [1/0] via 172.16.200.2, port1
[1/0] via 172.16.208.2, dmz
[1/0] via 172.16.209.2, port15
[1/0] via 172.16.210.2, port16
S 10.1.100.22/32 [10/0] via 172.16.209.2, port15
[10/0] via 172.16.210.2, port16
Because tie-break is set to fib-best-match, even though both port15 and port16 are out of SLA, the first
configured member of the two (port15) is selected to forward traffic to PC_2. For all other traffic, the first configured
member from all of the interfaces that are in SLA is selected to forward traffic (port1).
3. On PC-1, generate traffic to PC-2:

Fortinet Inc.
SD-WAN
ping 10.1.100.22
4. On FGT_A, sniff for traffic sent to PC_2:

interfaces=[any]
Traffic is leaving on port15, the first configured member from port15 and port16, even though both are out of SLA.
Internet service and application control steering
An application, application group, or application category can be selected as an SD-WAN service rule destination
criterion for IPv4 and IPv6 address modes.
To configure from the CLI:
config system sdwan
config service
edit <id>
set internet-service enable
set internet-service-app-ctrl <app id> [app id]
set internet-service-app-ctrl-group <app group> [app group]
set internet-service-app-ctrl-category <category id> [category id]
next
end
end
To configure for IPv6 addressing mode from the CLI, enable addr-mode ipv6:
config system sdwan
config service
edit <id>
set addr-mode ipv6
next
end
end
To view the detected application category details based on category ID, use diagnose sys sdwan internet-
service-app-ctrl-list cat-id <cat-id>.
This topic includes a GUI and CLI Example for application category on page 913 and a CLI Example for IPv6 on page
918.
Example for application category
In this example, traffic steering is applied to traffic detected as video/audio (category ID 5) or email (category ID 21) and
applies the lowest cost (SLA) strategy to this traffic. When costs are tied, the priority goes to member 1, dmz.

Fortinet Inc.
SD-WAN
To configure application categories as an SD-WAN rule destination in the GUI:
1. Enable the feature visibility:

a. Go to System > Feature Visibility.
b. In the Additional Features section, enable Application Detection Based SD-WAN.
c. Click Apply.
To enable GUI visibility of application detection based SD-WAN in the CLI:

config system global
set gui-app-detection-sdwan enable
end
2. Configure the SD-WAN members:

a. Go to Network > SD-WAN, select the SD-WAN Zones tab, and click Create New > SD-WAN Member.
b. Set the Interface to dmz, and set the Gateway to 172.16.208.2.
c. Click OK.
d. Repeat these steps to create another member for the vlan100 interface with gateway 172.16.206.2.
3. Configure the performance SLA (health check):
a. Go to Network > SD-WAN, and select the Performance SLAs tab, and click Create New.
b. Configure the following settings:
Name 1
Protocol DNS
Server 8.8.8.8
SLA Target Enable
c. Click OK.
4. Configure the SD-WAN rule to use the video/audio and email application categories:
a. Go to Network > SD-WAN, select the SD-WAN Rules tab, and click Create New.
b. In the Destination section, click the + in the Application field.
c. Click Category, and select Video/Audio and Email.

Fortinet Inc.
SD-WAN
d. Configure the other settings as needed.

e. Click OK.
5. Configure the firewall policy:
a. Go to Policy & Objects > Firewall Policy and click Create New.
b. Configure the following settings:
Incoming Interface port5
Outgoing Interface virtual-wan-link
Source 172.16.205.0
Destination all
Schedule always
Service ALL
Action ACCEPT
Application Control g-default
SSL Inspection certificate-inspection
c. Click OK.
To configure application categories as an SD-WAN rule destination in the CLI:
1. Configure the SD-WAN settings:

config system sdwan
set status enable
config zone

Fortinet Inc.
SD-WAN
next
end
config members
edit 1
set interface "dmz"
next
edit 2
set interface "vlan100"
next
end
config health-check
edit "1"
set server "8.8.8.8"
set protocol dns
set members 0
config sla
edit 1
next
end
next
end
end
2. Configure the SD-WAN rule to use application categories 5 and 21:

config system sdwan
config service
edit 1
set name "1"
set mode sla
set src "172.16.205.0"
set internet-service-app-ctrl-category 5 21
config sla
edit "1"
set id 1
next
end
next
end
end
3. Configure the firewall policy:

config firewall policy
edit 1
set srcintf "port5"
set dstintf "virtual-wan-link"
set action accept
set srcaddr 172.16.205.0
set dstaddr "all"
set schedule "always"
set service "ALL"

Fortinet Inc.
SD-WAN
set utm-status enable

set ssl-ssh-profile "certificate-inspection"
set application-list "g-default"
next
end
To test the configuration:
1. Verify that the traffic is sent over dmz:

# diagnose firewall proute list
list route policy info(vf=root):
id=2133590017(0x7f2c0001) vwl_service=1(1) vwl_mbr_seq=1 2 dscp_tag=0xff 0xff flags=0x0
tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0 dport=1-65535 path(2) oif=5(dmz)
oif=95(vlan100)
source(1): 172.16.205.0-172.16.205.255
destination wildcard(1): 0.0.0.0/0.0.0.0
internet service(2): (null)(0,5,0,0,0) (null)(0,21,0,0,0)
hit_count=469 last_used=2021-12-15 15:06:05
2. View some videos and emails on the PC, then verify the detected application details for each category:
# diagnose sys sdwan internet-service-app-ctrl-list cat-id 5
List App Ctrl Database Entry(IPv4) in Kernel:
Max_App_Ctrl_Size=32768 Num_App_Ctrl_Entry=4
YouTube(31077 4294838537): IP=142.250.217.110 6 443

YouTube(31077 4294838537): IP= 173.194.152.89 6 443
YouTube(31077 4294838537): IP= 173.194.152.170 6 443
YouTube(31077 4294838537): IP= 209.52.146.205 6 443
# diagnose sys sdwan internet-service-app-ctrl-list cat-id 21
List App Ctrl Database Entry(IPv4) in Kernel:
Max_App_Ctrl_Size=32768 Num_App_Ctrl_Entry=1
Gmail(15817 4294836957): IP=172.217.14.197 6 443
3. Verify that the captured email traffic is sent over dmz:

interfaces=[any]
filters=[host 172.217.14.197]
5.079814 dmz out 172.16.205.100.60592 -> 172.217.14.197.443: psh 2961561240 ack
2277134591
4. Edit the SD-WAN rule so that dmz has a higher cost and vlan100 is preferred.
5. Verify that the traffic is now sent over vlan100:
id=2134048769(0x7f330001) vwl_service=1(1) vwl_mbr_seq=2 1 dscp_tag=0xff 0xff flags=0x0
tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0 dport=1-65535 path(2) oif=95
(vlan100) oif=5(dmz)
source(1): 172.16.205.0-172.16.205.255

Fortinet Inc.
SD-WAN
internet service(2): (null)(0,5,0,0,0) (null)(0,21,0,0,0)

interfaces=[any]
filters=[host 172.217.14.197]
304.625168 vlan100 in 172.16.205.100.60592 -> 172.217.14.197.443: psh 2961572711 ack
2277139565
Example for IPv6
In this example, SD-WAN is configured to use an IPv6 service rule to steer traffic from FGT_A to FGT_B based on the
following application control options:
l Application Telnet
l An application group for ping
l An application category that includes SSH
When the rule is matched, traffic is steered based on the lowest cost SLA strategy. In this example, vlan100 is the
preferred interface, and traffic is routed to vlan100 on FGT_B.
To view the configuration:
1. View the SD-WAN configuration on FGT_A:

SD-WAN has four members in the default virtual-wan-link zone, each with an IPv4 and IPv6 gateway. The SD-WAN
service rule includes internet-service-app-ctrl 16091 for the Telnet, internet-service-app-ctrl-
group "network-Ping" for ping , and internet-service-app-ctrl-category 15 for SSH applications.
(sdwan) # show
config system sdwan
set status enable
config zone
next
end
config members
edit 1
set interface "dmz"
set gateway6 2000:172:16:208::2
next
edit 2
set interface "IPSec-1"
next
edit 3
set interface "agg1"
set gateway6 2000:172:16:203::2
next
edit 4
set interface "vlan100"
set gateway6 2000:172:16:206::2
next

Fortinet Inc.
SD-WAN
end
config health-check
edit "1"
set addr-mode ipv6
set server "2000::2:2:2:2"
set members 0
config sla
edit 1
next
end
next
end
config service
edit 1
set name "1"
set addr-mode ipv6
set mode sla
set internet-service-app-ctrl 16091
set internet-service-app-ctrl-group "network-Ping"
set internet-service-app-ctrl-category 15
config sla
edit "1"
set id 1
next
end
next
end
end
2. View the default route for FGT_A:

config router static
edit 5
set distance 1
set sdwan-zone "virtual-wan-link"
next
end
3. View the firewall policy for FGT_A:

The utm-status option is enabled to learn application 3T (3 tuple) information, and the default application profile
of g-default is selected.
edit 1
set uuid f09bddc4-def3-51ed-8517-0d8b6bc18f35
set srcintf "any"
set dstintf "any"
set action accept
set srcaddr6 "all"
set dstaddr6 "all"
set service "ALL"
set ssl-ssh-profile "certificate-inspection"
set application-list "g-default"

Fortinet Inc.
SD-WAN
next
end
To verify the configuration:
1. On FGT_A, check the routing table:

The routing table has ECMP applied to default gateways for each SD-WAN member.
# get router info routing-table static
S* 0.0.0.0/0 [1/0] via 172.16.203.2, agg1, [1/0]
[1/0] via 172.16.206.2, vlan100, [1/0]
[1/0] via 172.16.208.2, dmz, [1/0]
[1/0] via IPSec-1 tunnel 172.16.209.2, [1/0]
2. Check the SD-WAN service:

Based on the service rule, member 4 named vlan100 is preferred. Traffic must also match the highlighted internet
services.
# diagnose system sdwan service
Service(1): Address Mode(IPV6) flags=0x4200 use-shortcut-sla use-shortcut

Tie break: cfg
Members(4):
1: Seq_num(4 vlan100), alive, sla(0x1), gid(0), cfg_order(0), local cost(0),
selected 2: Seq_num(1 dmz), alive, sla(0x1), gid(0), cfg_order(1), local cost(0),
selected
3: Seq_num(2 IPSec-1), alive, sla(0x1), gid(0), cfg_order(2), local cost(0),
selected
4: Seq_num(3 agg1), alive, sla(0x1), gid(0), cfg_order(3), local cost(0), selected
Internet Service(3): Telnet(4294837974,0,0,0,0 16091) IPv6.ICMP(4294837087,0,0,0,0
16321) Network.Service(0,15,0,0,0)
3. Initiate traffic for ping, Telnet, and SSH to FGT_B, then FGT_A will learn 3T information for these applications, and
use the SD-WAN rule to route traffic for the applications to the preferred interface of vlan100.
l Following is the sniffer traffic for ping application. The ping traffic flows out of DMZ before 3T information is
recognized, then out from vlan100 after T3 traffic is recognized:
# diagnose sniffer packet any 'host 2000::2:0:0:4' 4
interfaces=[any]
filters=[host 2000::2:0:0:4]
16.952138 port5 in 2000:172:16:205::100 -> 2000::2:0:0:4: icmp6: echo request seq 1
[flowlabel 0x5080d]
16.954571 dmz out 2000:172:16:205::100 -> 2000::2:0:0:4: icmp6: echo request seq 1
[flowlabel 0x5080d]
16.954920 dmz in 2000::2:0:0:4 -> 2000:172:16:205::100: icmp6: echo reply seq 1
16.955086 port5 out 2000::2:0:0:4 -> 2000:172:16:205::100: icmp6: echo reply seq 1
[flowlabel 0x5080d]
17.953455 dmz out 2000:172:16:205::100 -> 2000::2:0:0:4: icmp6: echo request seq 2
[flowlabel 0x5080d]
17.953622 dmz in 2000::2:0:0:4 -> 2000:172:16:205::100: icmp6: echo reply seq 2
[flowlabel 0x5080d]

Fortinet Inc.
SD-WAN
18.960005 vlan100 out 2000:172:16:205::100 -> 2000::2:0:0:4: icmp6: echo request seq
3 [flowlabel 0x5080d]
18.960015 agg1 out 2000:172:16:205::100 -> 2000::2:0:0:4: icmp6: echo request seq 3
[flowlabel 0x5080d]
18.960024 port4 out 2000:172:16:205::100 -> 2000::2:0:0:4: icmp6: echo request seq 3
[flowlabel 0x5080d]
18.960295 vlan100 in 2000::2:0:0:4 -> 2000:172:16:205::100: icmp6: echo reply seq 3
[flowlabel 0x5080d]
l Following is the sniffer traffic for Telnet application group. The Telnet traffic flows out of agg1 before 3T
information is recognized, then out from vlan100 after T3 traffic is recognized:
# diagnose sniffer packet any 'host 2000::2:0:0:4 and dst port 23' 4 interfaces=
[any]
filters=[host 2000::2:0:0:4 and dst port 23]
4.096393 port5 in 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: syn 2723132265
[flowlabel 0xd4e65] 4.096739 agg1 out 2000:172:16:205::100.43128 ->
2000::2:0:0:4.23: syn 2723132265 [flowlabel 0xd4e65]
4.096752 port4 out 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: syn 2723132265
[flowlabel 0xd4e65]
.........
5.503679 port5 in 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: psh 2723132345 ack
544895389 [flowlabel 0xd4e65]
5.503894 vlan100 out 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: psh 2723132345
ack 544895389 [flowlabel 0xd4e65]
5.503907 agg1 out 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: psh 2723132345 ack
5.503918 port4 out 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: psh 2723132345 ack
5.504641 port5 in 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: ack 544895390
[flowlabel 0xd4e65]
5.504713 vlan100 out 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: ack 544895390
[flowlabel 0xd4e65]
5.504721 agg1 out 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: ack 544895390
[flowlabel 0xd4e65]
5.504728 port4 out 2000:172:16:205::100.43128 -> 2000::2:0:0:4.23: ack 544895390
[flowlabel 0xd4e65]
l Following is the sniffer traffic for SSH application category. The SSH traffic flows out of dmz before 3T
information is recognized, then out from vlan100 after T3 traffic is recognized:
# diagnose sniffer packet any 'host 2000::2:0:0:4 and dst port 22' 4
interfaces=[any]
filters=[host 2000::2:0:0:4 and dst port 22]
5.910752 port5 in 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: syn 980547187
[flowlabel 0xf1403]
5.911002 dmz out 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: syn 980547187
[flowlabel 0xf1403]
5.914550 port5 in 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: ack 583860244
[flowlabel 0xf1403]
5.914651 dmz out 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: ack 583860244
[flowlabel 0xf1403]
.....
8.116507 port5 in 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: psh 980549261 ack
583862554 [class 0x10] [flowlabel 0xf1403]

Fortinet Inc.
SD-WAN
8.116663 vlan100 out 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: psh 980549261

ack 583862554 [class 0x10] [flowlabel 0xf1403]
8.116674 agg1 out 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: psh 980549261 ack
8.116685 port4 out 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: psh 980549261 ack
8.118135 port5 in 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: ack 583862598
[class 0x10] [flowlabel 0xf1403]
8.118171 vlan100 out 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: ack 583862598
8.118179 agg1 out 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: ack 583862598
8.118189 port4 out 2000:172:16:205::100.35146 -> 2000::2:0:0:4.22: ack 583862598
4. View the IPv6 application control internet service ID list:

# diagnose system sdwan internet-service-app-ctrl6-list
Telnet(16091 4294837974): 2000::2:0:0:4 6 23 Thu Apr 20 17:43:00 2023

IPv6.ICMP(16321 4294837087): 2000::2:0:0:4 58 0 Thu Apr 20 17:43:00 2023
5. View the IPv6 application control internet service ID list by category:

# diagnose system sdwan internet-service-app-ctrl6-category-list
SSH(16060 4294837772): 2000::2:0:0:4 6 22 Thu Apr 20 17:43:00 2023
Use maximize bandwidth to load balance traffic between ADVPN shortcuts
When ADVPN is configured on a FortiGate spoke along with an SD-WAN rule set to Maximize Bandwidth SLA (GUI) or
load balance mode (CLI) as well as tie-break set to fib-best-match, then spoke-to-spoke traffic is load balanced
between multiple ADVPN shortcuts when the shortcuts are within the configured SLA conditions.
Following is an example configuration with load-balance enabled and tie-break set to fib-best-match:
config system sdwan
config service
edit 3
set mode sla
set load-balance enable
set dst "all"
config sla
edit "ping"
set id 1
next
end
next
end
end

Fortinet Inc.
SD-WAN
Example
In this example SD-WAN is configured between one hub and multiple spokes, and the SD-WAN configuration shows
SD-WAN rule 3 with the following required settings to enable spoke-to-spoke traffic between multiple ADVPN shortcuts:
l set load-balance enable
l set tie-break fib-best-match
show system sdwan
config system sdwan
set status enable
config zone
next
edit "zon2"
next
end
config members
edit 1
set interface "vd2-1"
set cost 10
next
edit 2
set cost 20
next
end
config health-check
edit "ping"
set server "11.11.11.11"
set members 1 2
config sla
edit 1
set latency-threshold 200
set jitter-threshold 50
next
edit 2
next
end
next
edit "1"
next
end
config service
edit 1

Fortinet Inc.
SD-WAN
set dst "033"

set priority-members 1
next
edit 2
set dst "133"
next
edit 3
set mode sla
set dst "all"
config sla
edit "ping"
set id 1
next
end
next
end
end
To trigger spoke-to-spoke communication, run an ICMP ping on PC A with IP address 22.1.1.22 behind spoke 1 that is
destined for PC B with IP address 33.1.1.33 behind spoke 2. The spoke-to-spoke traffic will be used to demonstrate load
balancing between shortcuts in the CLI output of this topic.
1. Confirm the ADVPN shortcuts are within the SLA conditions:

# diagnose system sdwan health-check
Health Check(ping):
Seq(1 vd2-1): state(alive), packet-loss(0.000%) latency(0.029), jitter(0.002), mos
(4.404), bandwidth-up(1999), bandwidth-dw(0), bandwidth-bi(1999) sla_map=0x3
Seq(1 vd2-1_0): state(alive), packet-loss(0.000%) latency(0.026), jitter(0.001), mos
2. Confirm the settings for SD-WAN rule 3:

# diagnose system sdwan service 3

Tie break: fib
Gen(1), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(load-balance hash-mode=round-robin)
Member sub interface(4):
1: seq_num(1), interface(vd2-1):
1: vd2-1_0(125)
1: vd2-2_0(127)
Members(4):
1: Seq_num(1 vd2-1), alive, sla(0x1), gid(2), num of pass(1), selected
2: Seq_num(1 vd2-1_0), alive, sla(0x1), gid(2), num of pass(1), selected
3: Seq_num(2 vd2-2), alive, sla(0x1), gid(2), num of pass(1), selected

Fortinet Inc.
SD-WAN
4: Seq_num(2 vd2-2_0), alive, sla(0x1), gid(2), num of pass(1), selected

Dst address(1):
0.0.0.0-255.255.255.255
3. Confirm firewall policing routing list:

# diagnose firewall proute list 2131230723
list route policy info(vf=vd2):
id=2131230723(0x7f080003) vwl_service=3 vwl_mbr_seq=1 1 2 2 dscp_tag=0xfc 0xfc

flags=0x90 load-balance hash-mode=round-robin fib-best-match tos=0x00 tos_mask=0x00
protocol=0 sport=0-65535 iif=0(any) dport=1-65535 path(4) oif=116(vd2-1) num_pass=1
oif=125(vd2-1_0) num_pass=1 oif=117(vd2-2) num_pass=1 oif=127(vd2-2_0) num_pass=1
destination(1): 0.0.0.0-255.255.255.255
source wildcard(1): 0.0.0.0/0.0.0.0
4. Confirm the routing table:

# get router info routing-table bgp
B* 0.0.0.0/0 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11),
01:26:14, [1/0]
[200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11),
01:26:14, [1/0]
B 1.1.1.1/32 [200/0] via 11.1.1.1 [2] (recursive via 12.1.1.1, vd2-vlan12),
01:26:14, [1/0]
B 11.11.11.11/32 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11),
01:26:14, [1/0]
01:26:14, [1/0]
B 33.1.1.0/24 [200/0] via 10.10.100.3 [2] (recursive is directly connected, vd2-1_
0), 01:19:41, [1/0]
[200/0] via 10.10.200.3 [2] (recursive is directly connected, vd2-2_
0), 01:19:41, [1/0]
01:26:14, [1/0]
01:26:14, [1/0]
5. Check the packet sniffer output for the default setting.

This step demonstrates routing for the default setting of set tie-break zone. The following packet sniffer
output of ICMP pings demonstrates how spoke-to-spoke traffic (ping from 22.1.1.22 to 33.1.1.13) is load balanced
between all parent tunnels and shortcuts, and is not limited to shortcuts within SLA.
# diagnose sniffer packet any "host 33.1.1.13" 4
interfaces=[any]
14.665232 vd22-vlan22 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
14.665234 npu0_vlink1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
14.665240 vd2-vlan22 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
14.665262 vd2-1_0 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
14.665274 vd3-1_0 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
14.665299 vd33-vlan33 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply

Fortinet Inc.
SD-WAN
14.665300 npu0_vlink1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply

14.665306 vd3-vlan33 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
14.665314 vd3-1_0 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
14.665326 vd2-1_0 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply

24.190982 vd2-2 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
24.190993 p2 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
24.191002 p2 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
24.191020 vd3-2 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
24.191063 vd3-2 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
24.191074 p2 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
24.191079 p2 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
24.191090 vd2-2 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply


67.257150 vd2-1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
67.257162 p1 in 22.1.1.22 -> 33.1.1.13: icmp: echo request
67.257170 p1 out 22.1.1.22 -> 33.1.1.13: icmp: echo request
67.257189 vd3-1 in 22.1.1.22 -> 33.1.1.13: icmp: echo request

Fortinet Inc.
SD-WAN

67.257234 vd3-1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
67.257245 p1 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
67.257250 p1 out 33.1.1.13 -> 22.1.1.22: icmp: echo reply
67.257261 vd2-1 in 33.1.1.13 -> 22.1.1.22: icmp: echo reply
^C
84 packets received by filter
0 packets dropped by kernel
6. Check the sniffer packet output after changing the setting to set tie-break fib-best-match.
The following packet sniffer output of ICMP pings demonstrates how load balancing of spoke-to-spoke is limited and
only occurs between shortcuts vd2-1_0 and vd2-2_0, which are within SLA.
interfaces=[any]


Fortinet Inc.
SD-WAN


^C
70 packets received by filter
0 packets dropped by kernel
7. Check SD-WAN heath.

When an ADVPN shortcut is out of SLA, traffic does not run on it. Shortcut vd2-1_0 is out of SLA.
# diagnose system sdwan health-check
Health Check(ping):
8. Check the sniffer packet:

No traffic runs on Shortcut vd2-1_0 because it is out of SLA.
interfaces=[any]

Fortinet Inc.
SD-WAN

Use SD-WAN rules to steer multicast traffic
SD-WAN rules can now steer multicast traffic. When an SD-WAN member is out of SLA, multicast traffic can fail over to
another SD-WAN member, and switch back when SLA recovers.
The new pim-use-sdwan option enables or disables the use of SD-WAN for PIM (Protocol Independent Multicast)
when checking RP (Rendezvous Point) neighbors and sending packets.
config router multicast
config pim-sm-global
set pim-use-sdwan {enable | disable}
end
end

Fortinet Inc.
SD-WAN
When SD-WAN steers multicast traffic, ADVPN is not supported. Use the set shortcut
option to disable shortcuts for the service:
config system sdwan
config service
edit <id>
set shortcut {enable | disable}
next
end
end
Example 1
In this hub and spoke example, the PIM source is behind the hub FortiGate, and the RP is set to internal port (port2) of
the hub firewall. Each spoke connects to the two WAN interfaces on the hub by using an overlay tunnel. The overlay
tunnels are members of SD-WAN.
Receivers behind the spoke FortiGates request a stream from the source to receive traffic on tunnel1 by default. When
the overlay tunnel goes out of SLA, the multicast traffic fails over to tunnel2 and continues to flow.
Following is an overview of how to configure the topology:

1. Configure the hub FortiGate in front of the PIM source. The RP is configured on internal port (port2) of the hub
FortiGate.
2. Configure the spoke FortiGates.
3. Verify traffic failover.

Fortinet Inc.
SD-WAN
To configure the hub:
1. On the hub, enable multicast routing, configure the multicast RP, and enable PIM sparse mode on each interface:
set multicast-routing enable
config rp-address
edit 1
set ip-address 172.16.205.1
next
end
end
config interface
edit "tport1"
set pim-mode sparse-mode
next
edit "tagg1"
next
edit "port2"
next
end
end
To configure each spoke:
1. Enable SD-WAN with the following settings:

l Configure the overlay tunnels as member of the SD-WAN zone.
l Configure a performance SLA health-check using ping.
l Configure a service rule for the PIM protocol with the following settings:
l Use the lowest cost (SLA) strategy.
l Monitor with the ping health-check.
l Disable ADVPN shortcut.
config system sdwan
set status enable
config zone
next
end
config members
edit 1
set interface "tunnel1"
next
edit 2
set interface "tunnel2"
next
end
config health-check
edit "ping"
set server "172.16.205.1"
set update-static-route disable

Fortinet Inc.
SD-WAN
set members 0
config sla
edit 1
next
end
next
end
config service
edit 1
set mode sla
set protocol 103
set dst "all"
config sla
edit "ping"
set id 1
next
end
set use-shortcut-sla disable
set shortcut disable
next
edit 2
set mode sla
set dst "all"
config sla
edit "ping"
set id 1
next
end
next
end
end
2. Enable multicast routing and configure the multicast RP. Enable PIM sparse-mode on each interface:
set spt-threshold disable
set pim-use-sdwan enable
config rp-address
edit 1
next
end
end
config interface
edit "tunnel1"
next
edit "tunnel2"
next
edit "port4"
next

Fortinet Inc.
SD-WAN
end
end
To verify traffic failover:
With this configuration, multicast traffic starts on tunnel1. When tunnel1 becomes out of SLA, traffic switches to tunnel2.
When tunnel1 is in SLA again, the traffic switches back to tunnel1.
The following health-check capture on the spokes shows tunnel1 in SLA with packet-loss (1.000%):
# diagnose sys sdwan health-check
Health Check(ping):
Seq(1 tunnel1): state(alive), packet-loss(0.000%) latency(0.056), jitter(0.002), mos(4.404),
bandwidth-up(999999), bandwidth-dw(1000000), bandwidth-bi(1999999) sla_map=0x1

Health Check(ping):
The following example shows tunnel1 out of SLA with packet-loss (3.000%):
Health Check(ping):
The following example shows tunnel1 back in SLA again:

Health Check(ping):

Health Check(ping):
The following example how traffic switches to tunnel2 while tunnel1 health-check is out of SLA. Source (172.16.205.11)
sends traffic to the multicast group. Later the traffic switches back to tunnel1 once SLA returns to normal:
195.060797 tunnel1 in 172.16.205.11 -> 225.1.1.1: icmp: echo request

Fortinet Inc.
SD-WAN

... ...
... ...
Example 2
In this hub and spoke example, the PIM source is behind spoke 1, and the RP is configured on the hub FortiGate. BGP is
used for routing. The hub uses embedded SLA in ICMP probes to determine the health of each tunnel, allowing it to
prioritize healthy IKE routes.
The receiver is on another spoke. Upon requesting a stream, source passes the traffic to the RP on the hub FortiGate,
and routes the traffic to the receiver over tunnel1. If a tunnel falls out of SLA, the multicast traffic fails over to the other
tunnel.
In this configuration, SD-WAN steers multicast traffic by using embedded SLA information in ICMP probes. See also
Embedded SD-WAN SLA information in ICMP probes. With this feature, the hub FortiGate can use the SLA information
of the spoke's health-check to control BGP and IKE routes over tunnels.

Fortinet Inc.
SD-WAN
Following is an overview of how to configure the topology:

1. Configure the hub FortiGate. The RP is configured on the hub FortiGate.
2. Configure the spoke FortiGate in front of the traffic receiver.
3. Configure the spoke FortiGate in front of the PIM source.
1. Configure loopbacks hub-lo1 172.31.0.1 for BGP and hub-lo100 172.31.100.100 for health-check:
config system interface
edit "hub-lo1"
set vdom "hub"
set ip 172.31.0.1 255.255.255.255
set allowaccess ping
set type loopback
set snmp-index 82
next
edit "hub-lo100"
set vdom "hub"
set ip 172.31.100.100 255.255.255.255
set type loopback
set snmp-index 81
next
end
2. Enable multicast routing with the following settings:

l Configure internal interface p25-v90 as RP.
l Enable interfaces for PIM sparse-mode.

Fortinet Inc.
SD-WAN
config rp-address
edit 1
next
end
end
config interface
edit "p11"
next
edit "p101"
next
edit "p25-v90"
next
end
end
3. Enable SD-WAN with the following settings:

l Add interfaces p11 and p101 as members.
l Configure embedded SLA health-checks to detect ICMP probes from each overlay tunnel. Prioritize based on
the health of each tunnel.
config system sdwan
set status enable
config zone
next
end
config members
edit 1
set interface "p11"
next
edit 2
set interface "p101"
next
end
config health-check
edit "1"
set detect-mode remote
set probe-timeout 60000
set recoverytime 1
set sla-id-redistribute 1
set members 1
config sla
edit 1
set link-cost-factor latency
set priority-in-sla 10
set priority-out-sla 20
next
end
next
edit "2"

Fortinet Inc.
SD-WAN
set detect-mode remote

set probe-timeout 60000
set recoverytime 1
set sla-id-redistribute 1
set members 2
config sla
edit 1
set priority-in-sla 15
set priority-out-sla 25
next
end
next
end
end
4. Configure BGP to peer with neighbors. Neighbor group is configured for tunnel interface IP addresses:
config router bgp

set as 65505
set router-id 172.31.0.1
set ibgp-multipath enable
set additional-path enable
set recursive-inherit-priority enable
config neighbor-group
edit "gr1"
set remote-as 65505
set update-source "hub-lo1"
set additional-path both
set route-reflector-client enable
next
end
config neighbor-range
edit 1
set prefix 10.10.0.0 255.255.0.0
set neighbor-group "gr1"
next
edit 66
set prefix 172.31.0.66 255.255.255.255
next
end
config network
....
edit 90
set prefix 192.90.0.0 255.255.0.0
next
end
end

Fortinet Inc.
SD-WAN
To configure the spoke (in front of the receiver):
1. Enable multicast routing to use SD-WAN. Configure the RP address. Enable interfaces for PIM sparse-mode.

set spt-threshold disable
config rp-address
edit 1
next
end
end
config interface
edit "p195"
next
edit "p196"
next
edit "internal4"
set static-group "225-1-1-122"
next
end
end
2. Configure SD-WAN with the following settings:

l Add overlay tunnel interfaces as members.
l Configure a performance SLA health-check to send ping probes to the hub.
l Configure a service rule for the PIM protocol. Use the lowest cost (SLA) strategy, and monitor with the ping
health-check.
l Disable ADVPN shortcuts.
config system sdwan
set status enable
config zone
next
end
config members
edit 6
next
edit 5
next
end

Fortinet Inc.
SD-WAN
config health-check
edit "ping"
set server "172.31.100.100"
set update-static-route disable
set members 0
config sla
edit 1
next
end
next
end
config service
edit 1
set mode sla
set protocol 103
set dst "all"
config sla
edit "ping"
set id 1
next
end
next
edit 2
set mode sla
set dst "all"
config sla
edit "ping"
set id 1
next
end
next
end
end
3. Configure BGP and set neighbors to the overlay gateway IP address on the hub:
config router bgp

set as 65505
config neighbor
edit "10.10.100.254"
set soft-reconfiguration enable

Fortinet Inc.
SD-WAN
set remote-as 65505

set connect-timer 10
next
edit "10.10.101.254"
set remote-as 65505
next
end
config network
edit 3
set prefix 192.84.0.0 255.255.0.0
next
end
end
4. Configure the default gateway to use the SD-WAN zone. Other routes are for the underlay to route traffic to the
hub's WAN interfaces:
edit 10
set distance 1
next
....
next
end
To configure the spoke (in front of the source):
1. Enable multicast routing to use SD-WAN. Configure the RP address. Enable interfaces for PIM sparse-mode:
config rp-address
edit 1
next
end
end
config interface
edit "p198"
next
edit "p200"
next
edit "npu0_vlink0"

Fortinet Inc.
SD-WAN
next
end
end
2. Configure loopback interface lo66 for BGP and sourcing SD-WAN traffic:
edit "lo66"
set vdom "root"
set ip 172.31.0.66 255.255.255.255
set type loopback
set snmp-index 21
next
end
3. Configure SD-WAN:
l Add overlay tunnel interfaces as members.
l Configure a performance SLA health-check to send ping probes to the hub.
l Configure a service rule for the PIM protocol. Use the lowest cost (SLA) strategy, and monitor with the ping
health-check.
l Disable the use of an ADVPN shortcut.
In the following example, 11.11.11.11 is the underlay address for one of the WAN links on the hub, and
172.31.100.100 is the loopback address on the server.
config system sdwan
set status enable
config zone
next
edit "overlay"
next
end
config members
edit 1
set zone "overlay"
set source 172.31.0.66
next
edit 2
set zone "overlay"
next
end
config health-check
edit "ping"
set server "11.11.11.11"
set members 0
config sla
edit 1
next
end
next

Fortinet Inc.
SD-WAN
edit "HUB"
set server "172.31.100.100"
set embed-measured-health enable
set members 0
config sla
edit 1
next
end
next
end
config service
edit 1
set mode sla
set protocol 103
set dst "all"
config sla
edit "ping"
set id 1
next
end
next
edit 2
set mode sla
set dst "all"
config sla
edit "ping"
set id 1
next
end
next
end
end
4. Configure BGP:
config router bgp
set as 65505
config neighbor
edit "172.31.0.1"
set next-hop-self enable
set remote-as 65505
set update-source "lo66"
next
end
config network
edit 3
set prefix 192.87.0.0 255.255.0.0

Fortinet Inc.
SD-WAN
next
end
end
5. Configure the default gateway to use the SD-WAN zone. Other routes are for the underlay to route to the hub's WAN
interfaces:
edit 10
set distance 1
set sdwan-zone "virtual-wan-link" "overlay"
next
...
next
end
Use SD-WAN rules for WAN link selection with load balancing
This example covers a use case where a user has multiple WAN links and wants to optimize the WAN link selection and
performance while limiting the use of more expensive and bandwidth intensive interfaces, such as 5G or LTE.
In this scenario, the user has three WAN links. The goal is to balance the load between wan1 and wan2; however, wan3,
which is quite costly to operate, should only be used if both wan1 and wan2 are unavailable.
This configuration involves the following steps:
1. Configuring the SD-WAN members

2. Configuring the manual SD-WAN rule
3. Configuring a static route
4. Configuring a firewall policy for SD-WAN
5. Verifying the configuration
Configuring the SD-WAN members
SD-WAN must be enabled first, and member interfaces must be selected and added to a zone. See Configuring the SD-
WAN interface on page 789 for more information.

Fortinet Inc.
SD-WAN
To configure the SD-WAN members in the GUI:
1. Configure the wan1, wan2, and wan3 interfaces (see Interface settings on page 164 for more details).
a. Set the wan1 interface IP/Netmask to 172.16.200.1 255.255.255.0.
b. Set the wan2 interface IP/Netmask to 10.1.100.1 255.255.255.0.
c. Set the wan3 interface IP/Netmask to 13.13.13.1 255.255.255.0.
2. Go to Network > SD-WAN, select the SD-WAN Zones tab, and click Create New > SD-WAN Member.
3. Configure the wan1 SD-WAN member:
a. Set the Interface to wan1.
b. Leave the SD-WAN Zone as virtual-wan-link.
c. Set the Gateway to 172.16.200.254.
d. Set the Status to Enable
e. Click OK.
4. Repeat step 3 for wan2 and wan3.
a. For wan2, set the Gateway to the ISP’s gateway, 10.1.100.254.
b. For wan3, set the Gateway to the ISP’s gateway, 13.13.13.254.
To configure the SD-WAN members in the CLI:
config system sdwan

set status enable
config zone
next
end
config members
edit 1
set interface "wan1"
set gateway 172.16.200.254
next
edit 2
next
edit 3
next
end
end
Configuring the manual SD-WAN rule
SD-WAN rules define specific routing options to route traffic to an SD-WAN member. See SD-WAN rules on page 854
and Manual strategy on page 867 for more information.
To configure a manual SD-WAN rule in the GUI:
1. Go to Network > SD-WAN, select the SD-WAN Rules tab, and click Create New.
2. Configure the following settings:

Fortinet Inc.
SD-WAN
Name test
Source > Address all
Destination > Address all
Interface selection strategy Manual
Interface preference wan1, wan2
Load balancing Enable this setting.
3. Configure the other settings as needed.

4. Click OK.
To configure a manual SD-WAN rule in the CLI:
config system sdwan

config service
edit 1
set name "test"
set dst "all"
set src "all"
next
end
end
Configuring a static route
A default route for SD-WAN must be configured. See Adding a static route on page 791 for more information.
To configure a static route for SD-WAN in the GUI:
1. Go to Network > Static Routes and click Create New. The New Static Route page opens.
2. Set the Destination to Subnet, and leave the IP address and subnet mask as 0.0.0.0/0.0.0.0.
3. Set the Interface to the SD-WAN zone, virtual-wan-link.
4. Set the Status to Enabled.
5. Click OK.
To configure a static route for SD-WAN in the CLI:

edit 1
set distance 1
next
end

Fortinet Inc.
SD-WAN
Configuring a firewall policy for SD-WAN
A firewall policy must be configured that allows traffic from the organization's internal network to the SD-WAN zone. See
Configuring firewall policies for SD-WAN on page 792 for more information.
To configure the firewall policy for SD-WAN in the GUI:
1. Go to Policy & Objects > Firewall Policy and click Create New.
2. Configure the following settings:
Name sd-wan
Incoming interface port1
Outgoing interface virtual-wan-link
Source all
Destination all
Schedule always
Service ALL
Action ACCEPT
NAT Enable and select NAT.
IP Pool Configuration Use Outgoing Interface Address
Enable this policy Enable this setting.
3. Configure the other settings as needed.

4. Click OK.
To configure the firewall policy for SD-WAN in the CLI:

edit 1
set name "sd-wan"
set srcintf "port1"
set action accept
set srcaddr "all"
set dstaddr "all"
set service "ALL"
set nat enable
next
end

Fortinet Inc.
SD-WAN
Verifying the configuration
To verify the SD-WAN member status:
# diagnose sys sdwan member

Member(1): interface: wan1, flags=0x0 , gateway: 172.16.200.254, priority: 1 1024, weight: 0
To verify the configuration when both wan1 and wan2 are up:
1. Verify the SD-WAN service rules status:

Tie break: cfg
Gen(1), TOS(0x0/0x0), Protocol(0): src(1->65535):dst(1->65535), Mode(manual hash-
mode=round-robin)
Members(2):
1: Seq_num(2 wan2 virtual-wan-link), alive, gid(1), selected
Src address(1):
0.0.0.0-255.255.255.255
Dst address(1):
0.0.0.0-255.255.255.255
This output indicates that both wan1 and wan2 are operational.
2. Verify the policy route list:
id=2130706433(0x7f000001) vwl_service=1(test) vwl_mbr_seq=1 2 dscp_tag=0xfc 0xfc

flags=0x10 load-balance hash-mode=round-robin tos=0x00 tos_mask=0x00 protocol=0
port=src(0->0):dst(0->0) iif=0(any)
path(2): oif=3(wan1) num_pass=0, oif=6(wan2) num_pass=0
source(1): 0.0.0.0-255.255.255.255
destination(1): 0.0.0.0-255.255.255.255
hit_count=154 last_used=2023-11-09 06:16:
This output indicates that both wan1 and wan2 are used to steer traffic.
To verify the configuration when wan2 is down and wan1 is up:


Tie break: cfg
mode=round-robin)
Members(2):
2: Seq_num(2 wan2 virtual-wan-link), dead, gid(1)

Fortinet Inc.
SD-WAN
Src address(1):
0.0.0.0-255.255.255.255
Dst address(1):
0.0.0.0-255.255.255.255
This output indicates that wan1 is operational, and wan2 is not.

id=2130706433(0x7f000001) vwl_service=1(test) vwl_mbr_seq=1 dscp_tag=0xfc 0xfc

flags=0x10 load-balance hash-mode=round-robin tos=0x00 tos_mask=0x00 protocol=0
port=src(0->0):dst(0->0) iif=0(any)
path(1): oif=3(wan1) num_pass=0
source(1): 0.0.0.0-255.255.255.255
destination(1): 0.0.0.0-255.255.255.255
This output indicates that wan1 is used to steer traffic.
To verify the configuration when wan1 is down and wan2 is up:


Tie break: cfg
mode=round-robin)
Members(2):
Src address(1):
0.0.0.0-255.255.255.255
Dst address(1):
0.0.0.0-255.255.255.255
This output indicates that wan2 is operational, and wan1 is not.


flags=0x10 load-balance has
h-mode=round-robin tos=0x00 tos_mask=0x00 protocol=0 port=src(0->0):dst(0->0) iif=0
(any)
source(1): 0.0.0.0-255.255.255.255
destination(1): 0.0.0.0-255.255.255.255

Fortinet Inc.
SD-WAN
To verify the configuration when both wan1 and wan2 down, and traffic is steered using wan3:
# diagnose sniffer packet wan3

Using Original Sniffing Mode
interfaces=[wan3]
filters=[none]
3.144417 13.13.13.1.52665 -> 204.79.197.239.443: 1610731732 ack 236747780
3.155250 204.79.197.239.443 -> 13.13.13.1.52665: ack 1610731733
5.047264 13.13.13.1.52613 -> 20.185.212.106.443: 1421254032 ack 3784884456
5.126008 20.185.212.106.443 -> 13.13.13.1.52613: ack 1421254033
To verify the configuration when either wan1 or wan2 is restored, and traffic ceases to be steered
through wan3:


Tie break: cfg
mode=round-robin)
Members(2):
Src address(1):
0.0.0.0-255.255.255.255
Dst address(1):
0.0.0.0-255.255.255.255
This output indicates that wan1 is operational.


flags=0x10 load-balance has
h-mode=round-robin tos=0x00 tos_mask=0x00 protocol=0 port=src(0->0):dst(0->0) iif=0
(any)
source(1): 0.0.0.0-255.255.255.255
destination(1): 0.0.0.0-255.255.255.255
Advanced routing
The following topics provide instructions on SD-WAN advanced routing:

Fortinet Inc.
SD-WAN
l Local out traffic on page 950

l Using BGP tags with SD-WAN rules on page 955
l BGP multiple path support on page 959
l Controlling traffic with BGP route mapping and service rules on page 961
l Applying BGP route-map to multiple BGP neighbors on page 968
l Using multiple members per SD-WAN neighbor configuration on page 974
Local out traffic
Local out, or self-originating, traffic is traffic that originates from the FortiGate going to external servers and services. The
traffic can be from Syslog, FortiAnalyzer logging, FortiGuard services, remote authentication, and others.
By default, local out traffic relies on routing table lookups to determine the egress interface that is used to initiate the
connection. However, many types of local out traffic support selecting the egress interface based on SD-WAN or
manually specified interfaces. When manually specifying the egress interface, the source IP address can also be
manually configured.
Go to Network > Local Out Routing to configure the available types of local out traffic. Some types of traffic can only be
configured in the CLI.
By default Local Out Routing is not visible in the GUI. Go to System > Feature Visibility to
enable it. See Feature visibility on page 3093 for more information.
When VDOMs are enabled, the following entries are available on the local out routing page:
Global view VDOM view
External Resources LDAP Servers
AWS_IP_Blacklist ldap
AWS_Malware_Hash Log
Log Log FortiAnalyzer Override Settings
Log FortiAnalyzer Setting Log Syslogd Override Settings
Log FortiAnalyzer Cloud Setting RADIUS Servers
FortiGate Cloud Log Settings fac_radius_server
Log Syslogd Setting TACACS+
System TACACS
System DNS
System FortiGuard
System FortiSandbox
If a service is disabled, it is grayed out. To enable it, select the service and click Enable Service. If a service is enabled,
there is a Local Out Setting button in the gutter of that service's edit page to directly configure the local-out settings.

Fortinet Inc.
SD-WAN
Examples
To configure DNS local-out routing:
1. Go to Network > Local Out Routing and double-click System DNS.

2. For Outgoing interface, select one of the following:
Auto Select the outgoing interface automatically based on the routing table.
SD-WAN Select the outgoing interface using the configured SD-WAN interfaces and
rules.
Specify Select the outgoing interface from the dropdown.
Use Interface IP Use the primary IP, which cannot be configured by the user.
Manually Selected an IP from the list, if the selected interface has multiple IPs
3. configured.
If Specify is selected, select a setting for Source IP:
4. Click OK.
To edit local-out settings from a RADIUS server entry:
1. Go to User & Authentication > RADIUS Servers and double-click an entry to edit it.
2. Click Local Out Setting.
The Edit Local Out Setting pane opens.

Fortinet Inc.
SD-WAN
3. Configure the settings for Outgoing interface and Source IP.
4. Click OK.
To edit multiple entries concurrently:
1. Go to Network > Local Out Routing.

2. If applicable, select IPv4 or IPv6. IPv4+IPv6 does not support multi-select.
3. Click Multi-Select Mode. All of the local out settings that can be edited concurrently are shown.
4. Select the specific entries, or click Select All to select all of the entries.
5. Click Edit and configure the local out settings as required.
6. Click OK.
7. Click Exit Multi-Select Mode to return to the normal view.
Configuring local out routing in the CLI
Some local out routing settings can only be configured using the CLI.
PING
IPv4 and IPv6 pings can be configured to use SD-WAN rules:

Fortinet Inc.
SD-WAN
execute ping-options use-sdwan {yes | no}

execute ping6-options use-sd-wan {yes | no}
Traceroute
IPv4 traceroute can be configured to use SD-WAN rules:

execute traceroute-options use-sdwan {yes | no}
Central management
Central management traffic can use SD-WAN rules or a specific interface:

config system central-management
set interface-select-method {auto | sdwan | specify}
set interface <interface>
end
NTP server
NTP server traffic can use SD-WAN rules or a specific interface:

config system ntp
config ntpserver
edit <id>
next
end
end
DHCP proxy
DHCP proxy traffic can use SD-WAN rules or a specific interface:

config system settings
set dhcp-proxy-interface-select-method {auto | sdwan | specify}
set dhcp-proxy-interface <interface>
end
dhcp-proxy-interface-select- Select the interface selection method:

method {auto | sdwan | l auto: Set the outgoing interface automatically (default).
specify} l sdwan: Set the interface by SD-WAN or policy routing rules.
l specify: Set the interface manually.
dhcp-proxy-interface Specify the outgoing interface. This option is only available and must be
<interface> configured when interface-select-method is specify.
DHCP relay
DHCP relay traffic can use SD-WAN rules or a specific interface:

edit <interface>
set dhcp-relay-interface-select-method {auto | sdwan | specify}

Fortinet Inc.
SD-WAN
set dhcp-relay-interface <interface>

next
end
dhcp-relay-interface-select- Select the interface selection method:

method {auto | sdwan | l auto: Set the outgoing interface automatically (default).
specify} l sdwan: Set the interface by SD-WAN or policy routing rules.
dhcp-relay-interface Specify the outgoing interface. This option is only available and must be
<interface> configured when interface-select-method is specify.
CA and local certificate renewal with SCEP
Certificate renewal with SCEP traffic can use SD-WAN rules or a specific interface:
config vpn certificate setting
end
IPS TLS protocol active probing
TLS active probing can use SD-WAN rules or a specific interface:

config ips global
config tls-active-probe
set interface-selection-method {auto | sdwan | specify}
set vdom <VDOM>
set source-ip <IPv4 address>
set source-ip6 <IPv6 address>
end
end
interface-select-method {auto | Select the interface selection method:

sdwan | specify} l auto: Set the outgoing interface automatically (default).
l sdwan: Set the interface by SD-WAN or policy routing rules.
interface <interface> Specify the outgoing interface. This option is only available and must be
configured when interface-select-method is specify.
vdom <VDOM> Specify the VDOM. This option is only available and must be configured when
interface-select-method is sdwan or specify.
source-ip <IPv4 address> Specify the source IPv4 address. This option is only available and must be
configured when interface-select-method is sdwan or specify.
source-ip6 <IPv6 address> Specify the source IPv6 address. This option is only available and must be
configured when interface-select-method is sdwan or specify.
NetFlow and sFlow
NetFlow and sFlow can use SD-WAN rules or a specific interface:

Fortinet Inc.
SD-WAN
config system {netflow | vdom-netflow}

end
config system {sflow | vdom-sflow}
config collectors
edit <id>
next
end
end
interface-select-method {auto | Select the interface selection method:

sdwan | specify} l auto: Set the outgoing interface automatically (default).
l sdwan: Set the interface by SD-WAN or policy routing rules.
interface <interface> Specify the outgoing interface. This option is only available and must be
configured when interface-select-method is specify.
FortiClient EMS
FortiClient EMS endpoint control traffic can use SD-WAN rules or a specific interface:
config endpoint-control fctems
edit fctems1
end
end
TACACS+
System log entries can be sent to external TACACS+ accounting servers. TACACS+ traffic can use SD-WAN rules or a
specific IP address:
config log tacacs+accounting setting
set source-ip <IP address>
end
Using BGP tags with SD-WAN rules
SD-WAN rules can use Border Gateway Protocol (BGP) learned routes as dynamic destinations.

Fortinet Inc.
SD-WAN
In this example, a customer has two ISP connections, wan1 and wan2. wan1 is used primarily for direct access to
internet applications, and wan2 is used primarily for traffic to the customer's data center.
The customer could create an SD-WAN rule using the data center's IP address range as the destination to force that
traffic to use wan2, but the data center's IP range is not static. Instead, a BGP tag can be used.
For this example, wan2's BGP neighbor advertises the data center's network range with a community number of 30:5.
This example assumes that SD-WAN is enabled on the FortiGate, wan1 and wan2 are added as SD-WAN members in
the virtual-wan-link SD-WAN zone, and a policy and static route have been created. See SD-WAN quick start on page
789 for details.
FortiOS supports IPv4 and IPv6 route tags.
To configure BGP tags with SD-WAN rules:
1. Configure the community list:

config router community-list
edit "30:5"
config rule
edit 1
set action permit
set match "30:5"
next
end
next
end
2. Configure the route map:

config router route-map
edit "comm1"
config rule
edit 1
set match-community "30:5"
set set-route-tag 15
next
end
next
end
3. Configure BGP:
config router bgp
set as xxxxx
set router-id xxxx
config neighbor
edit "10.100.20.2"
set remote-as xxxxx
set route-map-in "comm1"
next

Fortinet Inc.
SD-WAN
end
end
4. Configure the route tag address object:

config firewall address
edit "DataCenter_route_tag_15"
set type route-tag
set route-tag 15
next
end
5. Configure a firewall policy:

edit 1
set name "1"
set srcintf "dmz"
set srcaddr "all"
set dstaddr "DataCenter_route_tag_15"
set action accept
set service "ALL"
set nat enable
next
end
6. Edit the SD-WAN configuration:

config system sdwan
set status enable
config members
edit 1
next
edit 2
next
end
config service
edit 1
set name "DataCenter"
set mode manual
set dst "DataCenter_route_tag_15"
next
end
end
Troubleshooting BGP tags with SD-WAN rules
Check the network community
Use the get router info bgp network command to check the network community:

Fortinet Inc.
SD-WAN
# get router info bgp network

BGP table version is 5, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight RouteTag Path

*> 0.0.0.0/0 10.100.1.5 32768 0 ?
*> 1.1.1.1/32 0.0.0.0 32768 0 ?
*> 10.1.100.0/24 172.16.203.2 32768 0 ?
*> 10.100.1.0/30 0.0.0.0 32768 0 ?
*> 10.100.1.4/30 0.0.0.0 32768 0 ?
*> 10.100.1.248/29 0.0.0.0 32768 0 ?
*> 10.100.10.0/24 10.100.1.5 202 10000 15 20 e
*> 172.16.200.0/24 0.0.0.0 32768 0 ?
*> 172.16.200.200/32
0.0.0.0 32768 0 ?
*> 172.16.201.0/24 172.16.200.4 32768 0 ?
*> 172.16.203.0/24 0.0.0.0 32768 0 ?
*> 172.16.204.0/24 172.16.200.4 32768 0 ?
*> 172.16.205.0/24 0.0.0.0 32768 0 ?
*> 172.16.206.0/24 0.0.0.0 32768 0 ?
*> 172.16.207.1/32 0.0.0.0 32768 0 ?
*> 172.16.207.2/32 0.0.0.0 32768 0 ?
*> 172.16.212.1/32 0.0.0.0 32768 0 ?
*> 172.16.212.2/32 0.0.0.0 32768 0 ?
*> 172.17.200.200/32
0.0.0.0 32768 0 ?
*> 172.27.1.0/24 0.0.0.0 32768 0 ?
*> 172.27.2.0/24 0.0.0.0 32768 0 ?
*> 172.27.5.0/24 0.0.0.0 32768 0 ?
*> 172.27.6.0/24 0.0.0.0 32768 0 ?
*> 172.27.7.0/24 0.0.0.0 32768 0 ?
*> 172.27.8.0/24 0.0.0.0 32768 0 ?
*> 172.29.1.0/24 0.0.0.0 32768 0 ?
*> 172.29.2.0/24 0.0.0.0 32768 0 ?
*> 192.168.1.0 0.0.0.0 32768 0 ?
Total number of prefixes 28
# get router info bgp network 10.100.11.0

BGP routing table entry for 10.100.10.0/24
Paths: (2 available, best 1, table Default-IP-Routing-Table)
Advertised to non peer-group peers:
172.10.22.2
20
10.100.20.2 from 10.100.20.2 (6.6.6.6)
Origin EGP metric 200, localpref 100, weight 10000, valid, external, best
Community: 30:5 <<<<===========================
Last update: Wen Mar 20 18:45:17 2019
Check dynamic BGP addresses used in policy routes
Use the diagnose firewall proute list command to check dynamic BGP addresses used in policy routes:

Fortinet Inc.
SD-WAN

id=4278779905 vwl_service=1(DataCenter) flags=0x0 tos=0x00 tos_mask=0x00 protocol=0

sport=0:65535 iif=0 dport=1-65535 oif=16
BGP multiple path support
BGP supports multiple paths, allowing an ADVPN to advertise multiple paths. This allows BGP to extend and keep
additional network paths according to RFC 7911.
In this example, Spoke1 and Spoke2 each have four VPN tunnels that are connected to the Hub with ADVPN. The
Spoke-Hub has established four BGP neighbors on all four tunnels.
Spoke 1 and Spoke 2 can learn four different routes from each other.
config router bgp

set as 65505
set additional-path-select 4
edit "gr1"
set capability-default-originate enable
set remote-as 65505
set adv-additional-path 4
next
end

Fortinet Inc.
SD-WAN
edit 1
set prefix 10.10.0.0 255.255.0.0
next
end
config network
edit 12
set prefix 11.11.11.11 255.255.255.255
next
end
end
To configure a spoke:
config router bgp

set as 65505
config neighbor
edit "10.10.100.254"
set remote-as 65505
next
edit "10.10.200.254"
set remote-as 65505
next
edit "10.10.203.254"
set remote-as 65505
next
edit "10.10.204.254"
set remote-as 65505
next
end
config network
edit 3
set prefix 22.1.1.0 255.255.255.0
next
end
end

Fortinet Inc.
SD-WAN
To view the BGP routing table on a spoke:
Spoke1 # get router info routing-table bgp

B* 0.0.0.0/0 [200/0] via 10.10.200.254, vd2-2, 03:57:26
[200/0] via 10.10.203.254, vd2-3, 03:57:26
[200/0] via 10.10.204.254, vd2-4, 03:57:26
[200/0] via 10.10.100.254, vd2-1, 03:57:26
B 1.1.1.1/32 [200/0] via 11.1.1.1 (recursive via 12.1.1.1), 03:57:51
[200/0] via 11.1.1.1 (recursive via 12.1.1.1), 03:57:51
B 11.11.11.11/32 [200/0] via 10.10.200.254, vd2-2, 03:57:51
[200/0] via 10.10.203.254, vd2-3, 03:57:51
[200/0] via 10.10.204.254, vd2-4, 03:57:51
[200/0] via 10.10.100.254, vd2-1, 03:57:51
B 33.1.1.0/24 [200/0] via 10.10.204.3, vd2-4, 03:57:26
[200/0] via 10.10.203.3, vd2-3, 03:57:26
[200/0] via 10.10.200.3, vd2-2, 03:57:26
[200/0] via 10.10.100.3, vd2-1, 03:57:26
[200/0] via 10.10.204.3, vd2-4, 03:57:26
[200/0] via 10.10.203.3, vd2-3, 03:57:26
[200/0] via 10.10.200.3, vd2-2, 03:57:26
[200/0] via 10.10.100.3, vd2-1, 03:57:26
[200/0] via 10.10.204.3, vd2-4, 03:57:26
[200/0] via 10.10.203.3, vd2-3, 03:57:26
[200/0] via 10.10.200.3, vd2-2, 03:57:26
[200/0] via 10.10.100.3, vd2-1, 03:57:26
[200/0] via 10.10.204.3, vd2-4, 03:57:26
[200/0] via 10.10.203.3, vd2-3, 03:57:26
[200/0] via 10.10.200.3, vd2-2, 03:57:26
[200/0] via 10.10.100.3, vd2-1, 03:57:26
Controlling traffic with BGP route mapping and service rules
SD-WAN allows you to select different outbound WAN links based on performance SLAs. It is important that BGP
neighbors are aware of these settings, and changes to them.
BGP can adapt to changes in SD-WAN link SLAs in the following ways:
l Applying different route-maps based on the SD-WAN's health checks. For example, different BGP community
strings can be advertised to BGP neighbors when SLAs are not met.
l Traffic can be selectively forwarded based on the active BGP neighbor. If the SD-WAN service's role matches the
active SD-WAN neighbor, the service is enabled. If there is no match, then the service is disabled.
Example
In this topology, a branch FortiGate has two SD-WAN gateways serving as the primary and secondary gateways. The
gateways reside in different datacenters, but have a full mesh network between them.

Fortinet Inc.
SD-WAN
This example shows how route-maps and service rules are selected based on performance SLAs and the member that
is currently active. Traffic flows through the primary gateway unless the neighbor's health check is outside of its SLA. If
that happens, traffic routes to the secondary gateway.
BGP NBR1 is the primary neighbor and BGP NBR2 is the secondary neighbor.
The branch FortiGate's wan1 and wan2 interfaces are members of the SD-WAN. When the SD-WAN neighbor status is
primary, it will advertise community 20:1 to BGP NBR1 and 20:5 to BGP NBR2. When the SD-WAN neighbor status is
secondary, it will advertise 20:5 to BGP NBR1 and 20:2 to BGP NBR2.
Only one of the primary or secondary neighbors can be active at one time. The SD-WAN neighbor status is used to
decide which neighbor is selected:
l Primary: The primary neighbor takes precedence if its SLAs are met.
l Secondary: If the primary neighbor's SLAs are not met, the secondary neighbor becomes active if its SLAs are met.
l Standalone: If neither the primary or secondary neighbor's SLAs are met, the SD-WAN neighbor status becomes
standalone.
Route map
SD-WAN is configured to let BGP advertise different communities when the SLA status changes. When the SLA is
missed, it triggers BGP to advertise a different community to its BGP neighbor based on its route-map. The BGP
neighbors can use the received community string to select the best path to reach the branch.
To configure BGP route-maps and neighbors:
1. Configure an access for the routes to be matched:

config router access-list
edit "net192"
config rule
edit 1
set prefix 192.168.20.0 255.255.255.0
next
end

Fortinet Inc.
SD-WAN
next
end
2. Configure the primary neighbor's preferred route-map:

edit "comm1"
config rule
edit 1
set match-ip-address "net192"
set set-community "20:1"
next
end
next
end
3. Configure the secondary neighbor's preferred route-map:

edit "comm2"
config rule
edit 1
next
end
next
end
4. Configure the failed route-map:

edit "comm5"
config rule
edit 1
next
end
next
end
5. Configure BGP neighbors:

config router bgp
set as 65412
config neighbor
edit "10.100.1.1"
set remote-as 20
set route-map-out "comm5"
set route-map-out-preferable "comm1"
next
edit "10.100.1.5"
set remote-as 20
set route-map-out "comm5"

Fortinet Inc.
SD-WAN

next
end
end
When SLAs are met, route-map-out-preferable is used. When SLAs are missed, route-map-out is used.
To configure SD-WAN:

config system sdwan
set status enable
config members
edit 1
next
edit 2
next
end
end
2. Configure health checks for each member:

config system sdwan
config health-check
edit "ping"
set server "10.100.2.22"
set members 1
config sla
edit 1
set link-cost-factor packet-loss
set packetloss-threshold 1
next
end
next
edit "ping2"
set server "10.100.2.23"
set members 2
config sla
edit 1
next
end
next
end
end
3. Configure the SD-WAN neighbors and assign them a role and the health checks used to determine if the neighbor
meets the SLA:
SD-WAN neighbors can only be configured in the CLI.
config system sdwan
config neighbor
edit "10.100.1.1"
set member 1

Fortinet Inc.
SD-WAN
set role primary

set health-check "ping"
set sla-id 1
next
edit "10.100.1.5"
set member 2
set role secondary
set health-check "ping2"
set sla-id 1
next
end
end
Service rules
Create SD-WAN service rules to direct traffic to the primary neighbor when its SLAs are met, and to the secondary
neighbor when the primary neighbor's SLAs are missed.
To configure the SD-WAN service rules:
config system sdwan

config service
edit 1
set name "Primary-Out"
set role primary
set dst "all"
set src "all"
next
edit 2
set name "Secondary-Out"
set role secondary
set dst "all"
set src "all"
next
end
end
If neither the primary nor secondary neighbors are active, the SD-WAN neighbor status
becomes standalone. Only service rules with standalone-action enabled will continue to
pass traffic. This option is disabled by default.
Verification
To verify when the primary neighbor is passing traffic:
1. Verify the health check status:

FortiGate-Branch # diagnose sys sdwan health-check
Health Check(ping):
Seq(1 port1): state(alive), packet-loss(0.000%) latency(0.569), jitter(0.061) sla_
map=0x1

Fortinet Inc.
SD-WAN
Health Check(ping2):
map=0x1
2. Verify SD-WAN neighbor status:

FortiGate-Branch # diagnose sys sdwan neighbor
SD-WAN neighbor status: hold-down(disable), hold-down-time(0), hold_boot_time(0)
Selected role(primary) last_secondary_select_time/current_time in seconds 0/572
Neighbor(10.100.1.1): member(1) role(primary)
Health-check(ping:1) sla-pass selected alive
Neighbor(10.100.1.5): member(2) role(secondary)
Health-check(ping2:1) sla-pass alive
3. Verify service rules status:

FortiGate-Branch # diagnose sys sdwan service4

Gen(3), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(manual)
Service role: primary
Members:
Src address:
0.0.0.0-255.255.255.255
Dst address:
0.0.0.0-255.255.255.255

Service role: secondary, disabled by unselected.
Members:
Src address:
0.0.0.0-255.255.255.255
Dst address:
0.0.0.0-255.255.255.255
4. Verify neighbor routers:

a. Primary neighbor router:
FGT-NBR1 # get router info bgp network 192.168.20.0
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
64512
10.100.1.2 from 10.100.1.2 (192.168.122.98)
Origin IGP metric 0, localpref 100, valid, external, best
Community: 20:1
Last update: Thu Apr 30 13:41:40 2020
b. Secondary neighbor router:

VRF 0 BGP routing table entry for 192.168.20.0/24

Fortinet Inc.
SD-WAN
Original VRF 0
64512
10.100.1.6 from 10.100.1.6 (192.168.122.98)
Community: 20:5
To verify when the secondary neighbor is passing traffic:

Health Check(ping):
Seq(1 port1): state(dead), packet-loss(54.000%) sla_map=0x0
Health Check(ping2):
map=0x1

Selected role(secondary) last_secondary_select_time/current_time in seconds
936/936
Neighbor(10.100.1.1): member(1) role(primary)
Health-check(ping:1) sla-fail dead
Neighbor(10.100.1.5): member(2) role(secondary)
Health-check(ping2:1) sla-pass selected alive


Service role: primary, disabled by unselected.
Members:
Src address:
0.0.0.0-255.255.255.255
Dst address:
0.0.0.0-255.255.255.255

Service role: secondary
Members:
Src address:
0.0.0.0-255.255.255.255
Dst address:
0.0.0.0-255.255.255.255

Fortinet Inc.
SD-WAN

64512
10.100.1.2 from 10.100.1.2 (192.168.122.98)
Community: 20:5

Original VRF 0
64512
10.100.1.6 from 10.100.1.6 (192.168.122.98)
Community: 20:2
Applying BGP route-map to multiple BGP neighbors
Controlling traffic with BGP route mapping and service rules explained how BGP can apply different route-maps to the
primary and secondary SD-WAN neighbors based on SLA health checks.
In this example, SD-WAN neighbors that are not bound to primary and secondary roles are configured.
The FortiGate has multiple SD-WAN links and has formed BGP neighbors with both ISPs.
ISP1 is used primarily for outbound traffic, and has an SD-WAN service rule using the lowest cost algorithm applied to it.
When SLAs for ISP1 are not met, it will fail over to the MPLS line.
Inbound traffic is allowed by both WAN links, with each WAN advertising a community string when SLAs are met. When
SLAs are not met, the WAN links advertise a different community string.
This example uses two SD-WAN links. The topology can be expanded to include more links as needed.

Fortinet Inc.
SD-WAN
To configure BGP route-maps and neighbors:
1. Configure an access list for routes to be matched:

edit "net192"
config rule
edit 1
set prefix 192.168.20.0 255.255.255.0
next
end
next
end
2. Configure route-maps for neighbor ISP1:

edit "comm1"
config rule
edit 1
next
end
next
edit "comm-fail1"
config rule
edit 1
next
end
next
end
3. Configure route-maps for neighbor ISP2:

edit "comm2"
config rule
edit 1
next
end
next
edit "comm-fail2"
config rule
edit 1
next
end
next
end

Fortinet Inc.
SD-WAN
4. Configure the BGP neighbors:

config router bgp
set as 64512
set keepalive-timer 1
set holdtime-timer 3
config neighbor
edit "192.168.2.1"
set remote-as 64511
set route-map-out "comm-fail1"
next
edit "172.31.0.1"
set remote-as 64522
set route-map-out "comm-fail2"
next
end
config network
edit 1
set prefix 192.168.20.0 255.255.255.0
next
end
end
When SLAs are met, route-map-out-preferable is used. When SLAs are missed, route-map-out is used.

config system sdwan
set status enable
config members
edit 1
next
edit 2
set interface "MPLS"
set cost 20
next
end
end
2. Configure the health checks that must be met:

config system sdwan
config health-check
edit "pingserver"
set members 2 1
config sla
edit 1

Fortinet Inc.
SD-WAN
next
end
next
end
end
3. Configure the SD-WAN neighbors and assign them a role and the health checks used to determine if the neighbor
meets the SLA:
When no role is defined, the default role, standalone, is used.
config system sdwan
config neighbor
edit "192.168.2.1"
set member 1
set health-check "pingserver"
set sla-id 1
next
edit "172.31.0.1"
set member 2
set health-check "pingserver"
set sla-id 1
next
end
end
Service rules
Create SD-WAN service rules to direct traffic to the SD-WAN links based on the lowest cost algorithm The same SLA
health check and criteria that are used for the SD-WAN neighbor are used for this SD-WAN service rule.
When no roles are defined in the service rule, the default role, standalone, is used.
To configure the SD-WAN service rule:
config system sdwan

config service
edit 1
set name "OutboundAll"
set mode sla
set dst "all"
set src "all"
config sla
edit "pingserver"
set id 1
next
end
next
end
end

Fortinet Inc.
SD-WAN
Verification
To verify that when both SLAs are met, port1 is selected due to its lower cost:

Health Check(pingserver):
Seq(2 MPLS): state(alive), packet-loss(0.000%) latency(24.709), jitter(14.996) sla_
map=0x1
map=0x1

Neighbor(192.168.2.1): member(1) role(standalone)
Health-check(pingserver:1) sla-pass selected alive

Because the service role is standalone, it matches both neighbors. The mode (SLA) determines that port1 is
lower cost.

Service role: standalone
Members:
2: Seq_num(2 MPLS), alive, sla(0x1), cfg_order(1), cost(20), selected
Src address:
0.0.0.0-255.255.255.255
Dst address:
0.0.0.0-255.255.255.255

64512
192.168.2.5 from 192.168.2.5 (192.168.122.98)
Community: 64511:1

Original VRF 0

Fortinet Inc.
SD-WAN
64512
172.31.0.2 from 172.31.0.2 (192.168.122.98)
Community: 64522:1
Last update: Fri May 1 00:11:28 2020
To verify that when neighbor ISP1 misses SLAs, MPLS is selected and BGP advertises a different
community string for ISP1:

Health Check(pingserver):
Seq(2 MPLS): state(alive), packet-loss(0.000%) latency(25.637), jitter(17.820) sla_
map=0x1

Health-check(pingserver:1) sla-fail dead

As SLA failed for neighbor ISP1, MPLS is preferred.

Members:
1: Seq_num(2 MPLS), alive, sla(0x1), cfg_order(1), cost(20), selected
2: Seq_num(1 port1), dead, sla(0x0), cfg_order(0), cost(0)
Src address:
0.0.0.0-255.255.255.255
Dst address:
0.0.0.0-255.255.255.255

The community received on ISP1 is updated.
64512
192.168.2.5 from 192.168.2.5 (192.168.122.98)
Community: 64511:5

Fortinet Inc.
SD-WAN

Original VRF 0
64512
172.31.0.2 from 172.31.0.2 (192.168.122.98)
Community: 64522:1
Using multiple members per SD-WAN neighbor configuration
SD-WAN BGP neighbor configurations are used to define the SLA health check in which an SD-WAN member must
meet to qualify as being up. When the SD-WAN member meets the SLA threshold, the FortiGate will apply the route map
defined in the BGP neighbor's route-map-out-preferable option. If the SD-WAN member fails to meet the SLA,
the FortiGate will apply the route map defined in the BGP neighbor's route-map-out option instead. This allows the
FortiGate to advertise the health of the SD-WAN member to its BGP neighbor by advertising different community strings
based on its SLA status.
For more information, refer to the following BGP examples: Controlling traffic with BGP route
mapping and service rules on page 961 and Applying BGP route-map to multiple BGP
neighbors on page 968.
Selecting multiple SD-WAN members allows the SD-WAN neighbor feature to support topologies where there are
multiple SD-WAN overlays and/or underlays to a neighbor. The minimum-sla-meet-members option is used to
configure the minimum number of members that must be in an SLA per neighbor for the preferable route map to be used.
config system sdwan
config neighbor
edit <ip>
set member {<seq-num_1>} [<seq-num_2>] ... [<seq-num_n>]
set minimum-sla-meet-members <integer>
next
end
end
member {<seq-num_1>} Enter the member sequence number list. Multiple members can be defined.
[<seq-num_2>] ...
[<seq-num_n>]
minimum-sla-meet-members Set the minimum number of members that meet SLA when the neighbor is
<integer> preferred (1 - 255, default = 1).
l If the number of in SLA members is less than the minimum-sla-meet-
members value, the default route map will be used.

l If the number of in SLA members is equal or larger than the minimum-sla-
meet-members value, the preferable route map will be used.

Fortinet Inc.
SD-WAN
Example
In the following example, the spoke FortiGate has four tunnels: two tunnels to Hub_1 and two tunnels to Hub_2. The
spoke has two BGP neighbors: one to Hub_1 and one to Hub-2. BGP neighbors are established on loopback IPs.
The SD-WAN neighbor plus route-map-out-preferableconfiguration is deployed on the spoke to achieve the
following:
l If any tunnel to Hub_1 or Hub_2 is in SLA, the preferable route map will be applied on the BGP neighbor to Hub_1 or
Hub_2.
l If both tunnels to Hub_1 or Hub_2 are out of SLA, the default route map will be applied on the BGP neighbor to Hub_
1 or Hub_2.
The preferable route map and default route map are used to set different custom BGP communities as the spoke
advertises its LAN routes to the hub. Each hub can translate communities into different BGP MED or AS prepends and
signal them to the external peers to manipulate inbound traffic, thereby routing traffic to the spoke only when the SLAs
are met on at least one of two VPN overlays. In this example, community string 10:1 signals to the neighbor that SLAs
are met, and 10:2 signals that SLAs are not met.
To configure the BGP route maps and neighbors:
1. Configure an access list of prefixes to be matched:

edit "net10"
config rule
edit 1
set prefix 10.0.3.0 255.255.255.0
next
end
next
end

Fortinet Inc.
SD-WAN
2. Configure route maps for neighbors in SLA (preferable) and out of SLA (default):
edit "in_sla"
config rule
edit 1
next
end
next
edit "out_sla"
config rule
edit 1
next
end
next
end
3. Configure the BGP neighbors:

config router bgp
config neighbor
edit "172.31.0.1"
set route-map-out "out_sla"
set route-map-out-preferable "in_sla"
set update-source "Loopback0"
next
edit "172.31.0.2"
set update-source "Loopback0"
next
end
config network
edit 1
set prefix 10.0.3.0 255.255.255.0
next
end
end

config system sdwan
set status enable
config members
edit 1
set interface "H1_T11"
next
edit 4

Fortinet Inc.
SD-WAN

next
edit 6
next
edit 9
next
end
end
2. Configure the health check that must be met:

config system sdwan
config health-check
edit "HUB"
set server "172.31.100.100"
set members 0
config sla
edit 1
next
end
next
end
end
3. Configure the SD-WAN neighbors:

config system sdwan
config neighbor
edit "172.31.0.1"
set member 1 4
set health-check "HUB"
set sla-id 1
set minimum-sla-meet-members 1
next
edit "172.31.0.2"
set member 6 9
set sla-id 1
next
end
end
To verify that when two members to Hub_1/Hub_2 are in SLA, the preferable route map is be applied on
BGP neighbors to Hub_1/Hub_2:
Branch1_A_FGT (root) # diagnose sys sdwan health-check

Health Check(HUB):
Seq(1 H1_T11): state(alive), packet-loss(0.000%) latency(0.209), jitter(0.017), mos(4.404),

Fortinet Inc.
SD-WAN

# diagnose sys sdwan neighbor
Neighbor(172.31.0.1): member(1 4 )role(standalone)
Health-check(HUB:1) sla-pass selected alive
On Hub_1 and Hub_2, the expected communities have been attached into the spoke's LAN route:
Hub_1_FGT (root) # get router info bgp network 10.0.3.0/24
Original VRF 0
Local, (Received from a RR-client)
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Origin IGP metric 0, localpref 100, valid, internal, best
Community: 10:1
Last update: Wed Dec 29 22:38:29 2021
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:1
If one member for each neighbor becomes out of SLA, the preferable route map is still applied:
Health Check(HUB):
Seq(1 H1_T11): state(alive), packet-loss(0.000%) latency(120.207), jitter(0.018), mos

Fortinet Inc.
SD-WAN
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:1
Last update: Thu Dec 30 10:44:47 2021
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:1
If both members for Hub_1 become out of SLA, the default route map is applied:
Health Check(HUB):
Health-check(HUB:1) sla-fail alive
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:2
Last update: Thu Dec 30 10:57:33 2021
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:1

Fortinet Inc.
SD-WAN
VPN overlay
The following topics provide instructions on SD-WAN VPN overlays:

l ADVPN 2.0 edge discovery and path management on page 980
l ADVPN and shortcut paths on page 994
l Active dynamic BGP neighbor triggered by ADVPN shortcut on page 1007
l SD-WAN monitor on ADVPN shortcuts on page 1017
l Hold down time to support SD-WAN service strategies on page 1018
l Adaptive Forward Error Correction on page 1020
l Dual VPN tunnel wizard on page 1024
l Duplicate packets on other zone members on page 1025
l Duplicate packets based on SD-WAN rules on page 1028
l Interface based QoS on individual child tunnels based on speed test results on page 1030
l SD-WAN in large scale deployments on page 1033
l Keeping sessions in established ADVPN shortcuts while they remain in SLA on page 1044
l SD-WAN multi-PoP multi-hub large scale design and failover on page 1050
l Using a single IKE elector in ADVPN to match all SD-WAN control plane traffic on page 1069
l SD-WAN Overlay-as-a-Service on page 1076
ADVPN 2.0 edge discovery and path management
The SD-WAN with ADVPN solution has evolved to version 2.0 with major changes to ADVPN design and operation,
including the introduction of edge discovery and path management for ADVPN spokes.
ADVPN 2.0 incorporates intelligence into the spokes to ensure shortcut tunnels (also known as shortcuts) are
established using underlays available on both spokes and chosen based on matching certain link health criteria.
ADVPN 2.0 provides a more flexible SD-WAN solution than the original ADVPN to achieve resiliency against underlay
outages or degraded underlay performance because it no longer depends on specific BGP routing designs or
mechanisms, including route reflection, BGP next hop recursive resolution, BGP per overlay, and BGP on loopback.
Currently, ADVPN 2.0 only supports IPv4.
The following topics describe further details about ADVPN 2.0:

l SD-WAN with ADVPN 2.0 versus previous ADVPN on page 980
l SD-WAN CLI configuration on page 981
l Example SD-WAN configurations using ADVPN 2.0 on page 982
SD-WAN with ADVPN 2.0 versus previous ADVPN
With the previous version of ADVPN and SD-WAN, shortcut path selection relied entirely on the overlays between the
spokes. The hub and overlays were used to exchange IKE shortcut messages, and policy routes were configured on the

Fortinet Inc.
SD-WAN
hub to ensure shortcuts were established on the same overlay. In addition, user traffic was needed to trigger the process
of establishing shortcuts.
With the latest version of ADVPN and SD-WAN, shortcut path selection is achieved through edge discovery and path
management functionality on the ADVPN spokes.
1. Edge discovery:
l Expand IKE Shortcut-Reply message to allow the local spoke (spoke where user traffic is initiated) to obtain the
remote spoke (destination spoke for user traffic) WAN link information, which includes IP address, transport
group, link quality, link cost, and member configuration order.
l After shortcut establishment, WAN link information can be exchanged on the shortcut regularly every 5
seconds through UDP traffic. The path management function on the local spoke is regularly updated to pick up
changes to remote or local overlays and select the best shortcut path accordingly.
2. Path management:
The local spoke handles the remote spoke WAN link information, calculates the best shortcut path per SD-WAN
service or rule, and then advises IKE to establish a shortcut using the selected path.
SD-WAN CLI configuration
The following SD-WAN CLI configuration commands are used to configure ADVPN 2.0 on the spokes:
config system sdwan
config zone
edit <zone-name>
set advpn-select {enable | disable}
set advpn-health-check <health-check name>
next
end
config members
edit <integer>
set transport-group <integer>
next
end
config service
edit <integer>
set shortcut-priority {enable | disable | auto}
next
end
end
set advpn-select {enable Enable or disable SDWAN/ADVPN-2.0 (default=disabled).

| disable}
set advpn-health-check Specify the health check for the spoke whose info will be sent to the peer spoke.
<health-check name>
set transport-group Specify different group ID between (1 -255) to differentiate link-type, such as
<integer> Internet, MPLS, LTE, Satellite.

Fortinet Inc.
SD-WAN
set shortcut-priority Enable or disable making ADVPN shortcut a high priority over overlay parent
{enable | disable | interfaces, if SLA mode or link cost factor mode conditions are met:
auto}
l enable: enable a high priority of ADVPN shortcut for this service.
l disable: disable a high priority of ADVPN shortcut for this service.

l auto: automatically enable a high priority of ADVPN shortcut for this service
if ADVPN2.0 is enabled.
diagnose sys sdwan advpn- Diagnostic command run on local spoke to view remote spoke WAN link
session information and path manager shortcut path selection.
As with the previous version of ADVPN, on the hub, you must enable ADVPN and configure firewall policies between
spokes.
Example SD-WAN configurations using ADVPN 2.0
The configuration example illustrates the edge discovery and path management processes for a typical hub and spoke
topology. This example focuses on SD-WAN configuration for steering traffic and establishing shortcuts in the direction
from Spoke 1 to Spoke 2.
l Network Topology on page 982
l SD-WAN configuration and health check status on page 984
l Scenario 1: Traffic matching SD-WAN rule 1 on page 986
l Scenario 4: Spoke 2 H1_T22 overlay link out-of-SLA on page 991
Network Topology
In this example, BGP per overlay was used for dynamic routing to distribute the LAN routes behind each spoke to the
other spoke. However, this was a design choice. You can also use BGP on loopback for this example.

Fortinet Inc.
SD-WAN
Spokes 1 and 2 have the following VPN overlays between themselves and the hub:
VPN Overlays IP address on Spoke 1 IP address on Spoke 2
H1_T11 172.31.80.1/32 172.31.80.2/32
H1_T22 172.31.81.1/32 172.31.81.2/32
H1_T33 172.31.82.1/32 172.31.82.2/32
SD-WAN Rules/Services defined on Spoke 1:
SD-WAN Rule/Service 1 SD-WAN Rule/Service 2 SD-WAN Rule/Service 3
H1_T11 H1_T22 H1_T33
H1_T22 H1_T11 H1_T11
H1_T33 H1_T33 H1_T22
Strategy for choosing Lowest cost (SLA) Lowest cost (SLA) Best quality, link cost
outgoing interfaces factor: packet loss
Throughout this example, transport group 1 is used for VPN overlays over Internet links while transport group 2 is used
for the VPN overlay over an MPLS link.
In this example, user traffic is initiated behind Spoke 1 and destined to Spoke 2. Because of this, Spoke 1 is considered
the local spoke, and Spoke 2 is considered the remote spoke.

Fortinet Inc.
SD-WAN
SD-WAN configuration and health check status
SD-WAN configuration and health check status on Spoke 1:
config system sdwan

set status enable
config zone
next
edit "overlay"
set advpn-select enable
set advpn-health-check "HUB"
next
end
config members
edit 1
set zone "overlay"
set transport-group 1
next
edit 2
set zone "overlay"
next
edit 3
set zone "overlay"
next
end
config health-check
edit "HUB"
set server "172.31.100.100"
set members 1 2 3
config sla
edit 1
next
end
next
end
config service
edit 1
set name "1"
set mode sla
set shortcut-priority enable
set dst "spoke-2_LAN-1" "Tunnel_IPs"
set src "spoke-1_LAN-1" "Tunnel_IPs"
config sla
edit "HUB"
set id 1
next
end
set priority-members 1 2 3

Fortinet Inc.
SD-WAN
next
edit 2
set name "2"
set mode sla
set shortcut-priority enable
config sla
edit "HUB"
set id 1
next
end
next
edit 3
set name "3"
set mode priority
next
end
end
Health Check(HUB):
SD-WAN configuration and health check status on Spoke 2:
config system sdwan

set status enable
config zone
next
edit "overlay"
set advpn-select enable
set advpn-health-check "HUB"
next
end
config members
edit 1
set zone "overlay"
set cost 100
next
edit 2
set zone "overlay"

Fortinet Inc.
SD-WAN
next
edit 3
set zone "overlay"
next
end
config health-check
edit "HUB"
set server "172.31.100.100"
set members 3 1 2
config sla
edit 1
next
end
next
end
end
Health Check(HUB):
Scenario 1: Traffic matching SD-WAN rule 1
In this scenario, PC 1 connected to Spoke 1 initiates an ICMP ping destined for PC1 connected to Spoke 2. Therefore,
this user traffic matches SD-WAN rule 1 and triggers shortcut path selection and establishment.
The Path Manager of Spoke 1 will calculate the best shortcut path by comparing transport group, link quality (for SLA
mode), link cost, and member configuration order between Spoke 1 and Spoke 2.
For an SLA mode service, the following algorithm is followed for considering endpoints of the best shortcut path:
1. Overlays with the same transport group
2. In-SLA overlays
3. Lowest link cost overlays
4. Member configuration order as a final tiebreaker
Based on this algorithm, the Path Manager on Spoke 1 selects Spoke 1 H1_T11 because:
It is first in the priority-members order for SD-WAN rule 1, it has the lowest link cost, and it is within SLA. Likewise, the
Path Manager on Spoke 1 selects Spoke 2 H1_T22 since it has the lowest link cost compared to Spoke 2 H1_T11 (which
has a cost of 100), it is within SLA, and has the same transport group as Spoke 1 H1_T11. Therefore, the Path Manager
of Spoke 1 calculates the best shortcut path as Spoke 1 H1_T11 to Spoke 2 H1_T22.
The Path Manager will advise IKE to establish the best shortcut and add it to SD-WAN rule 1 as follows:
Branch1_FGT# diagnose sys sdwan service4

Fortinet Inc.
SD-WAN
Tie break: cfg

Shortcut priority: 1
Gen(11), TOS(0x0/0x0), Protocol(0): src(1->65535):dst(1->65535), Mode(sla), sla-compare-
order
2: seq_num(1), interface(H1_T11):
1: H1_T11_0(71)
Members(4):
1: Seq_num(1 H1_T11_0 overlay), alive, sla(0x1), gid(0), cfg_order(0), local cost(0),
selected
2: Seq_num(1 H1_T11 overlay), alive, sla(0x1), gid(0), cfg_order(0), local cost(0),
selected
selected
selected
Src address(2):
172.31.0.0-172.31.255.255
10.0.3.0-10.0.3.255
Dst address(2):
172.31.0.0-172.31.255.255
10.0.4.0-10.0.4.255
…
Since shortcut-priority is enabled, we observe that the shortcut is formed over the selected overlay path and prioritized
over the parent overlay.
From the diagnostic command on Spoke 1, we observe the selected shortcut path in bold. (Note that the remote IP
matches Spoke 2 H1_T22 in the corresponding table above.)
Branch1_FGT# diagnose sys sdwan advpn-session
Session head(Branch2_FGT-0-overlay:1)
(1) Service ID(1), last access(7809088), remote health check info(3)
Selected path: local(H1_T11, port1) gw: 172.31.3.1 remote IP: 172.31.3.105(172.31.81.2)
Remote information:
1: latency: 0.176267 jitter: 0.005733 pktloss: 0.000000 mos: 4.404302 sla: 0x1 cost: 0
transport_group: 1 bandwidth up: 999994 down: 999997 bidirection: 1999991
ipv4: 172.31.3.105(172.31.81.2) ipv6 2000:172:31:3::105(::)
ipv4: 172.31.4.101(172.31.82.2) ipv6 1410:4b02::f088:93ee:7f00:0
(c010:4b02::788a:93ee:7f00:0)
ipv4: 172.31.3.101(172.31.80.2) ipv6 2000:172:31:3::101(d88a:93ee:7f00:0:d88a:93ee:7f00:0)
From the diagnostic command on Spoke 2, we observe the selected shortcut in bold.
Branch2_FGT# diagnose sys sdwan health-check
Health Check(HUB):
Seq(2 H1_T22_0): state(alive), packet-loss(0.000%) latency(0.265), jitter(0.011), mos

Fortinet Inc.
SD-WAN
In this scenario, PC 1 connected to Spoke 1 initiates an ICMP ping destined for PC2 connected to Spoke 2. Therefore,
this user traffic matches SD-WAN rule 2, and traffic will go through shortcut H1_T11_0 of Spoke 1 previously established
in Scenario 1 above.
The local spoke generates local-out UDP packets and sends them to the hub to trigger an IKE shortcut message
exchange with updated remote spoke WAN link information. The local spoke will receive this updated remote spoke
WAN link information. Then the Path Manager of Spoke 1 will recalculate the best shortcut path by comparing transport
group, link quality (for SLA mode), link cost, and member configuration order between Spoke 1 and Spoke 2.
2. In-SLA overlays
Based on this algorithm, the Path Manager on Spoke 1 selects Spoke 1 H1_T22 because it is the first in the priority-
members order for SD-WAN rule 2, it has the lowest link cost, and it is within SLA. Likewise, the Path Manager on Spoke
1 selects Spoke 2 H1_T22 since it has the lowest link cost compared to Spoke 2 H1_T11 (which has a cost of 100), it is
within SLA, and has the same transport group as Spoke 1 H1_T11. Therefore, the Path Manager of Spoke 1 calculates
the best shortcut path as Spoke 1 H1_T22 to Spoke 2 H1_T22.
…
Tie break: cfg
order
1: H1_T22_0(72)
1: H1_T11_0(71)
Members(5):
selected
selected, last_used=2023-12-05 14:34:07
selected
selected
selected
Src address(2):
172.31.0.0-172.31.255.255
10.0.3.0-10.0.3.255
Dst address(2):
172.31.0.0-172.31.255.255
10.0.40.0-10.0.40.255
…

Fortinet Inc.
SD-WAN
The newly selected shortcut is prioritized over the previously selected shortcut as seen in the bolded output above.
Remote information:
ipv4: 172.31.4.101(172.31.82.2) ipv6 180:adfb::d88a:93ee:7f00:0
(d88a:93ee:7f00:0:d88a:93ee:7f00:0)
ipv4: 172.31.3.105(172.31.81.2) ipv6 2000:172:31:3::105(::)
ipv4: 172.31.3.101(172.31.80.2) ipv6 2000:172:31:3::101(c010:4b02::788a:93ee:7f00:0)
Remote information:
(d88a:93ee:7f00:0:d88a:93ee:7f00:0)
ipv4: 172.31.3.105(172.31.81.2) ipv6 2000:172:31:3::105(::)
ipv4: 172.31.3.101(172.31.80.2) ipv6 2000:172:31:3::101(c010:4b02::788a:93ee:7f00:0)
…
From the diagnostic command on Spoke 2, we observe the selected shortcut in bold:
Health Check(HUB):
In this scenario, PC 1 connected to Spoke 1 initiates an ICMP ping destined for PC 3 connected to Spoke 2. Therefore,
this user traffic matches SD-WAN rule 3, and traffic will go through shortcut H1_T11_0 of Spoke 1 previously established
in Scenario 1 above.

Fortinet Inc.
SD-WAN
The local spoke generates local-out UDP packets and sends them to the hub to trigger an IKE shortcut message
exchange with updated remote spoke WAN link information. The local spoke will receive this updated remote spoke
WAN link information. Then the Path Manager of Spoke 1 will recalculate the best shortcut path by comparing transport
group, best quality (based on link cost factor), and member configuration order between Spoke 1 and Spoke 2.
For a best quality mode service, the following algorithm is followed for considering endpoints of the best shortcut path:
2. Best quality overlays (link cost factor of packet loss, in this scenario)
Based on this algorithm, the Path Manager on Spoke 1 selects Spoke 1 H1_T33 because it is the first in the priority-
members order for SD-WAN rule 3, and it has the best quality link. Likewise, the Path Manager on Spoke 1 selects
Spoke 2 H1_T33 since it has the same transport group as Spoke 1 H1_T33. Therefore, the Path Manager of Spoke 1
calculates the best shortcut path as Spoke 1 H1_T33 to Spoke 2 H1_T33.
…
Tie break: cfg
Gen(13), TOS(0x0/0x0), Protocol(0): src(1->65535):dst(1->65535), Mode(priority), link-
cost-factor(packet-loss), link-cost-threshold(10), heath-check(HUB)
1: H1_T33_0(73)
1: H1_T11_0(71)
1: H1_T22_0(72)
Members(6):
1: Seq_num(3 H1_T33_0 overlay), alive, packet loss: 0.000%, selected
2: Seq_num(1 H1_T11_0 overlay), alive, packet loss: 0.000%, selected, last_used=2023-12-
05 14:38:02
3: Seq_num(2 H1_T22_0 overlay), alive, packet loss: 0.000%, selected
4: Seq_num(3 H1_T33 overlay), alive, packet loss: 0.000%, selected
Src address(2):
172.31.0.0-172.31.255.255
10.0.3.0-10.0.3.255
Dst address(2):
172.31.0.0-172.31.255.255
10.0.41.0-10.0.41.255
Remote information:

Fortinet Inc.
SD-WAN

(d88a:93ee:7f00:0:d88a:93ee:7f00:0)
ipv4: 172.31.3.105(172.31.81.2) ipv6 2000:172:31:3::105(c010:4b02::788a:93ee:7f00:0)
ipv4: 172.31.3.101(172.31.80.2) ipv6 2000:172:31:3::101(::)
From the diagnostic command on Spoke 2, we observe the selected shortcut in bold:
Health Check(HUB):
Scenario 4: Spoke 2 H1_T22 overlay link out-of-SLA
In this scenario, we place remote Spoke 2 H1_T22 out-of-SLA and observe that this link quality change is sensed by the
local spoke through regular WAN link information updates on shortcuts. Then the local Spoke 1 will generate local-out
UDP packets and send them to the hub to trigger an IKE shortcut message exchange. Once Spoke 1 receives a shortcut
reply, it will start to calculate new best shortcut paths for SD-WAN rules 1 and 2 because these are the only rules that
have new best shortcut paths when Spoke 2 H1_T22 is out-of-SLA.
2. In-SLA overlays
Based on this algorithm, the Path Manager on Spoke 1 still selects these Spoke 1 interfaces:
l SD-WAN Rule 1: H1_T11
These are the first in the priority-members order for SD-WAN rules 1 and 2, respectively.
Based on the updated WAN link information, the Path Manager on Spoke 1 selects these Spoke 2 interfaces because
they are the only remaining in-SLA VPN overlays over Internet links (transport group 1):
Therefore, the Path Manager of Spoke 1 calculates the best shortcut paths as follows:

Fortinet Inc.
SD-WAN
l SD-WAN Rule 1: Spoke 1 H1_T11 to Spoke 2 H1_T11

l SD-WAN Rule 2: Spoke 1 H1_T22 to Spoke 2 H1_T11
The Path Manager will advise IKE to establish the best shortcuts and add them to SD-WAN rules 1 and 2 as follows:
l For SD-WAN Rule 1, H1_T11_1 is the new best shortcut.
l For SD-WAN Rule 2, H1_T22_1 is the new best shortcut.
Tie break: cfg
order
1: H1_T11_0(74)
2: H1_T11_1(75)
1: H1_T22_0(72)
2: H1_T22_1(76)
1: H1_T33_0(73)
Members(8):
selected
selected
selected
selected
selected
selected
selected
selected
Src address(2):
172.31.0.0-172.31.255.255
10.0.3.0-10.0.3.255
Dst address(2):
172.31.0.0-172.31.255.255
10.0.4.0-10.0.4.255

Tie break: cfg
order
1: H1_T22_0(72)
2: H1_T22_1(76)

Fortinet Inc.
SD-WAN
1: H1_T11_0(74)
2: H1_T11_1(75)
1: H1_T33_0(73)
Members(8):
selected
selected
selected
selected
selected
selected
selected
selected
Src address(2):
172.31.0.0-172.31.255.255
10.0.3.0-10.0.3.255
Dst address(2):
172.31.0.0-172.31.255.255
10.0.40.0-10.0.40.255
…
From the diagnostic command on Spoke 1, we observe the newly selected shortcut paths in bold. (Note that the remote
IP 172.31.80.2 matches Spoke 2 H1_T11, which is the VPN overlay over the Internet link with cost 100 in the
corresponding table above.)
# diagnose sys sdwan advpn-session
Remote information:
(d88a:93ee:7f00:0:d88a:93ee:7f00:0)
ipv4: 172.31.3.105(172.31.81.2) ipv6 2000:172:31:3::105(c010:4b02::788a:93ee:7f00:0)
ipv4: 172.31.3.101(172.31.80.2) ipv6 2000:172:31:3::101(::)
Remote information:
(d88a:93ee:7f00:0:d88a:93ee:7f00:0)

Fortinet Inc.
SD-WAN
ipv4: 172.31.3.105(172.31.81.2) ipv6 2000:172:31:3::105(c010:4b02::788a:93ee:7f00:0)

ipv4: 172.31.3.101(172.31.80.2) ipv6 2000:172:31:3::101(::)
From the diagnostic command on Spoke 2, we observe the selected shortcuts in bold:
Health Check(HUB):
ADVPN and shortcut paths
This topic provides an example of how to use SD-WAN and ADVPN together.
ADVPN (Auto Discovery VPN) is an IPsec technology that allows a traditional hub-and-spoke VPN’s spokes to establish
dynamic, on-demand, direct tunnels between each other to avoid routing through the topology's hub device. The primary
advantage is that it provides full meshing capabilities to a standard hub-and-spoke topology. This greatly reduces the
provisioning effort for full spoke-to-spoke low delay reachability, and addresses the scalability issues associated with
very large fully meshed VPN networks.
If a customer's head office and branch offices all have two or more internet connections, they can build a dual-hub
ADVPN network. Combined with SD-WAN technology, the customer can load-balance traffic to other offices on multiple
dynamic tunnels, control specific traffic using specific connections, or choose better performance connections
dynamically.
SD-WAN load-balance mode rules (or services) do not support ADVPN members. Other
modes' rules, such as SLA and priority, support ADVPN members.
This topic covers three parts:

1. Configure dual-hub ADVPN with multiple branches.
2. Configure BGP to exchange routing information among hubs and spokes.
3. Configure SD-WAN on spoke to do load-balancing and control traffic.

Fortinet Inc.
SD-WAN
Configuration example
A typical ADVPN configuration with SD-WAN usually has two hubs, and each spoke connects to two ISPs and
establishes VPN tunnels with both hubs.
This example shows a hub-and-spoke configuration using two hubs and one spoke:
l Hub1 and Hub2 both use wan1 to connect to the ISPs and port10 to connect to internal network.
l Spoke1 uses wan1 to connect to ISP1 and wan2 to connect to ISP2.
l wan1 sets up VPN to hub1.
l wan2 sets up VPN to hub2.
The SD-WAN is configured on the spoke. It uses the two VPN interfaces as members and two rules to control traffic to
headquarters or other spokes using ADVPN VPN interfaces. You can create more rules if required.
For this example:
l Use SD-WAN member 1 (via ISP1) and its dynamic shortcuts for financial department traffic if member 1 meets SLA
requirements. If it doesn't meet SLA requirements, it will use SD-WAN member 2 (via ISP2).
l Use SD-WAN member 2 (via ISP2) and its dynamic shortcuts for engineering department traffic.
l Load balance other traffic going to hubs and other spokes between these two members.
l Set up all other traffic to go with their original ISP connection. All other traffic does not go through SD-WAN.
l Set up basic network configuration to let all hubs and spokes connect to their ISPs and the Internet.

Fortinet Inc.
SD-WAN
Hub internal network 172.16.101.0/24
Spoke1 internal network 10.1.100.0/24
ADVPN 1 network 10.10.100.0/24
ADVPN 2 network 10.10.200.0/24
Hub1 wan1 IP 11.1.1.11
Hub2 wan1 IP 11.1.2.11
Hub1 VPN IP 10.10.100.254
Hub2 VPN IP 10.10.200.254
Spoke1 to hub1 VPN IP 10.10.100.2
Spoke1 to hub2 VPN IP 10.10.200.2
Ping server in Headquarters 11.11.11.11
Internal subnet of spoke1 22.1.1.0/24
Internal subnet of spoke2 33.1.1.0/24
Firewall addresses Configure hub_subnets and spoke_subnets before using in policies. These can
be customized.
The GUI does not support some ADVPN related options, such as auto-discovery-sender, auto-discovery-receiver, auto-
discovery-forwarder, and IBGP neighbor-group setting, so this example only provides CLI configuration commands.
Hub1 sample configuration
To configure the IPsec phase1 and phase2 interface:
config vpn ipsec phase1-interface

edit "hub-phase1"
set type dynamic
set peertype any
set net-device disable
set proposal aes128-sha256 aes256-sha256 3des-sha256 aes128-sha1 aes256-sha1 3des-
sha1
set add-route disable
set dpd on-idle
set auto-discovery-sender enable
set psksecret sample
set dpd-retryinterval 5
next
end
edit "hub-phase2"
set phase1name "hub-phase1"
sha256
next
end

Fortinet Inc.
SD-WAN
When net-device is disabled, a tunnel ID is generated for each dynamic tunnel. This ID, in
the form of an IP address, is used as the gateway in the route entry to that tunnel. The
tunnel-search option is removed in FortiOS 7.0.0 and later.
To configure the VPN interface and BGP:

edit "hub-phase1"
set ip 10.10.100.254 255.255.255.255
set remote-ip 10.10.100.253 255.255.255.0
next
end
config router bgp
set as 65505
edit "advpn"
set link-down-failover enable
set remote-as 65505
next
end
edit 1
set prefix 10.10.100.0 255.255.255.0
set neighbor-group "advpn"
next
end
config network
edit 1
set prefix 172.16.101.0 255.255.255.0
next
edit 2
set prefix 11.11.11.0 255.255.255.0
next
end
end
To configure the firewall policy:

edit 1
set name "spoke2hub"
set srcintf "hub-phase1"
set dstintf "port10"
set srcaddr "spoke_subnets"
set dstaddr "hub_subnets"
set action accept
set service "ALL"
set comments "allow traffic from spokes to headquarter"
next
edit 2
set name "spoke2spoke"

Fortinet Inc.
SD-WAN
set dstintf "hub-phase1"

set dstaddr "spoke_subnets"
set action accept
set service "ALL"
set comments "allow traffic from spokes to spokes"
next
edit 3
set name "internal2spoke"
set srcintf "port10"
set srcaddr "hub_subnets"
set action accept
set service "ALL"
set comments "allow traffic from headquarter to spokes"
next
end
Hub2 sample configuration
Hub2 configuration is the same as hub1 except the wan1 IP address, VPN interface IP address, and BGP neighbor-
range prefix.

edit "hub-phase1"
set type dynamic
set peertype any
sha1
set dpd on-idle
next
end
edit "hub-phase2"
set phase1name "hub-phase1"
sha256
next
end

edit "hub-phase1"

Fortinet Inc.
SD-WAN
set ip 10.10.200.254 255.255.255.255

set remote-ip 10.10.200.253 255.255.255.0
next
end
config router bgp
set as 65505
edit "advpn"
set remote-as 65505
next
end
edit 1
set prefix 10.10.200.0 255.255.255.0
next
end
config network
edit 1
set prefix 172.16.101.0 255.255.255.0
next
edit 2
set prefix 11.11.11.0 255.255.255.0
next
end
end

edit 1
set name "spoke2hub"
set dstintf "port10"
set dstaddr "hub_subnets"
set action accept
set service "ALL"
set comments "allow traffic from spokes to headquarter"
next
edit 2
set name "spoke2spoke"
set action accept
set service "ALL"
set comments "allow traffic from spokes to spokes"
next
edit 3
set name "internal2spoke"

Fortinet Inc.
SD-WAN
set srcintf "port10"

set srcaddr "hub_subnets"
set action accept
set service "ALL"
set comments "allow traffic from headquarter to spokes"
next
end
Spoke1 sample configuration

edit "spoke1-phase1"
set peertype any
set net-device enable
set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1
set dpd on-idle
set auto-discovery-receiver enable
set remote-gw 11.1.1.11
next
edit "spoke1-2-phase1"
set peertype any
set proposal aes128-sha256 aes256-sha256 aes128-sha1 aes256-sha1
set dpd on-idle
next
end
set phase1name "spoke1-phase1"
set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm aes256gcm
chacha20poly1305
set auto-negotiate enable
next
set phase1name "spoke1-2-phase1"
set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm aes256gcm
chacha20poly1305
next
end

Fortinet Inc.
SD-WAN

set ip 10.10.100.2 255.255.255.255
set remote-ip 10.10.100.254 255.255.255.0
next
set ip 10.10.200.2 255.255.255.255
set remote-ip 10.10.200.254 255.255.255.0
next
end
config router bgp
set as 65505
config neighbor
edit "10.10.100.254"
set advertisement-interval 1
set remote-as 65505
next
edit "10.10.200.254"
set remote-as 65505
next
end
config network
edit 1
set prefix 10.1.100.0 255.255.255.0
next
end
end
config system sdwan

set status enable
config members
edit 1
set interface "spoke1-phase1"
next
edit 2
set interface "spoke1-2-phase1"
next
end
config health-check
edit "ping"
set server "11.11.11.11"
set members 1 2
config sla
edit 1
next
end

Fortinet Inc.
SD-WAN
next
end
config service
edit 1
set mode sla
set dst "financial-department"
config sla
edit "ping"
set id 1
next
end
next
edit 2
set dst "engineering-department"
next
end
end
If no SD-WAN zone is specified, members are added to the default virtual-wan-link zone.

edit 1
set name "outbound_advpn"
set srcintf "internal"
set dstaddr "spoke_subnets" "hub_subnets"
set action accept
set service "ALL"
set comments "allow internal traffic going out to headquarter and other spokes"
next
edit 2
set name "inbound_advpn"
set srcintf "virtual-wan-link"
set dstintf "internal"
set srcaddr "spoke_subnets" "hub_subnets"
set action accept
set service "ALL"
set comments "allow headquarter and other spokes traffic coming in"
next
end

Fortinet Inc.
SD-WAN
Troubleshooting ADVPN and shortcut paths
Before spoke vs spoke shortcut VPN is established
Use the following CLI commands to check status before spoke vs spoke shortcut VPN is established.
# get router info bgp summary
BGP router identifier 2.2.2.2, local AS number 65505
BGP table version is 13
3 BGP AS-PATH entries
0 BGP community entries
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd

10.10.100.254 4 65505 3286 3270 11 0 0 00:02:15 5
10.10.200.254 4 65505 3365 3319 12 0 0 00:02:14 5
Total number of neighbors 2


B* 0.0.0.0/0 [200/0] via 10.10.200.254, spoke1-2-phase1, 00:00:58
[200/0] via 10.10.100.254, spoke1-phase1, 00:00:58
B 11.11.11.0/24 [200/0] via 10.10.200.254, spoke1-2-phase1, 00:01:29
[200/0] via 10.10.100.254, spoke1-phase1, 00:01:29
B 33.1.1.0/24 [200/0] via 10.10.200.3, spoke1-2-phase1, 00:00:58
[200/0] via 10.10.100.3, spoke1-phase1, 00:00:58
[200/0] via 10.10.200.3, spoke1-2-phase1, 00:00:58
[200/0] via 10.10.100.3, spoke1-phase1, 00:00:58
# diagnose vpn tunnel list
list all ipsec tunnel in vd 3
------------------------------------------------------
name=spoke1-phase1 ver=1 serial=5 12.1.1.2:0->11.1.1.11:0 tun_id=11.1.1.11 dst_mtu=15324
bound_if=48 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/536 options[0218]=npu create_dev
frag-rfc accept_traffic=1
proxyid_num=1 child_num=0 refcnt=22 ilast=0 olast=0 ad=r/2

stat: rxp=1 txp=185 rxb=16428 txb=11111
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=4
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=spoke1 proto=0 sa=1 ref=4 serial=1 auto-negotiate adr
src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
SA: ref=6 options=1a227 type=00 soft=0 mtu=15262 expire=42820/0B replaywin=2048
seqno=ba esn=0 replaywin_lastseq=00000002 itn=0 qat=0
life: type=01 bytes=0/0 timeout=42903/43200
dec: spi=03e01a2a esp=aes key=16 56e673f0df05186aa657f55cbb631c13
ah=sha1 key=20 b0d50597d9bed763c42469461b03da8041f87e88
enc: spi=2ead61bc esp=aes key=16 fe0ccd4a3ec19fe6d520c437eb6b8897
ah=sha1 key=20 e3e669bd6df41b88eadaacba66463706f26fb53a
dec:pkts/bytes=1/16368, enc:pkts/bytes=185/22360
npu_flag=03 npu_rgwy=11.1.1.11 npu_lgwy=12.1.1.2 npu_selid=0 dec_npuid=1 enc_npuid=1
------------------------------------------------------
name=spoke1-2-phase1 ver=1 serial=6 112.1.1.2:0->11.1.2.11:0 tun_id=11.1.2.11 dst_mtu=15324

Fortinet Inc.
SD-WAN


proxyid=spoke1-2 proto=0 sa=1 ref=4 serial=1 auto-negotiate adr
src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
seqno=bb esn=0 replaywin_lastseq=00000002 itn=0 qat=0
dec: spi=03e01a2b esp=aes key=16 fe49f5042a5ad236250bf53312db1346
ah=sha1 key=20 5dbb15c8cbc046c284bb1c6425dac2b3e15bec85
enc: spi=2ead61bd esp=aes key=16 d6d97be52c3cccb9e88f28a9db64ac46
ah=sha1 key=20 e20916ae6ea2295c2fbd5cbc8b8f5dd8b17f52f1

TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla)
Member sub interface:
Members:
1: Seq_num(1), alive, sla(0x1), cfg_order(0), cost(0), selected
Dst address: 33.1.1.1-33.1.1.100

TOS(0x0/0x0), Protocol(0: 1->65535), Mode(manual)
Members:
1: Seq_num(2), alive, selected
Dst address: 33.1.1.101-33.1.1.200
id=2132869121 vwl_service=1 vwl_mbr_seq=1 dscp_tag=0xff 0xff flags=0x0 tos=0x00 tos_

mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=70 oif=71
destination(1): 33.1.1.1-33.1.1.100

mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=71
destination(1): 33.1.1.101-33.1.1.200
After spoke vs spoke shortcut VPN is established
Use the following CLI commands to check status after spoke vs spoke shortcut VPN is established.

Fortinet Inc.
SD-WAN
B* 0.0.0.0/0 [200/0] via 10.10.200.254, spoke1-2-phase1, 00:01:33

[200/0] via 10.10.100.254, spoke1-phase1, 00:01:33
B 11.11.11.0/24 [200/0] via 10.10.200.254, spoke1-2-phase1, 00:02:04
[200/0] via 10.10.100.254, spoke1-phase1, 00:02:04
B 33.1.1.0/24 [200/0] via 10.10.200.3, spoke1-2-phase1_0, 00:01:33
[200/0] via 10.10.100.3, spoke1-phase1_0, 00:01:33
[200/0] via 10.10.200.3, spoke1-2-phase1_0, 00:01:33
[200/0] via 10.10.100.3, spoke1-phase1_0, 00:01:33

TOS(0x0/0x0), Protocol(0: 1->65535), Mode(sla)
1: seq_num(1), interface(spoke1-phase1):
1: spoke1-phase1_0(111)
2: seq_num(2), interface(spoke1-2-phase1):
1: spoke1-2-phase1_0(113)
Members:
Dst address: 33.1.1.1-33.1.1.100

TOS(0x0/0x0), Protocol(0: 1->65535), Mode(manual)
1: seq_num(2), interface(spoke1-2-phase1):
1: spoke1-2-phase1_0(113)
Members:
1: Seq_num(2), alive, selected
Dst address: 33.1.1.101-33.1.1.200
------------------------------------------------------
name=spoke1-phase1 ver=1 serial=5 12.1.1.2:0->11.1.1.11:0 tun_id=11.1.1.11 dst_mtu=15324

proxyid=vd2-1 proto=0 sa=1 ref=5 serial=1 auto-negotiate adr
src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
seqno=2f8 esn=0 replaywin_lastseq=00000002 itn=0 qat=0
dec: spi=03e01a42 esp=aes key=16 1f131bda108d33909d49fc2778bd08bb
ah=sha1 key=20 14131d3f0da9b741a2fd13d530b0553aa1f58983
enc: spi=2ead61d8 esp=aes key=16 81ed24d5cd7bb59f4a80dceb5a560e1f
ah=sha1 key=20 d2ccc2f3223ce16514e75f672cd88c4b4f48b681

Fortinet Inc.
SD-WAN
------------------------------------------------------
name=spoke1-2-phase1 ver=1 serial=6 112.1.1.2:0->11.1.2.11:0 tun_id=11.1.2.11 dst_mtu=15324

src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
seqno=2f5 esn=0 replaywin_lastseq=00000002 itn=0 qat=0
dec: spi=03e01a43 esp=aes key=16 7fc87561369f88b56d08bfda769eb45b
ah=sha1 key=20 0ed554ef231c5ac16dc2e71d1907d7347dda33d6
enc: spi=2ead61d9 esp=aes key=16 00286687aa1762e7d8216881d6720ef3
ah=sha1 key=20 59d5eec6299ebcf038c190860774e2833074d7c3
------------------------------------------------------
name=spoke1-phase1_0 ver=1 serial=55 12.1.1.2:0->13.1.1.3:0 tun_id=13.1.1.3 dst_mtu=15324
bound_if=48 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/728 options[02d8]=npu
create_dev no-sysctl rgwy-chg frag-rfc accept_traffic=1
parent=vd2-1 index=0
src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
dec: spi=03e01a44 esp=aes key=16 c3b77a98e3002220e2373b73af14df6e
ah=sha1 key=20 d18d107c248564933874f60999d6082fd7a78948
enc: spi=864f6dba esp=aes key=16 eb6181806ccb9bac37931f9eadd4d5eb
ah=sha1 key=20 ab788f7a372877a5603c4ede1be89a592fc21873
------------------------------------------------------
name=spoke1-2-phase1_0 ver=1 serial=57 112.1.1.2:0->113.1.1.3:0 tun_id=113.1.1.3 dst_
mtu=15324
bound_if=90 lgwy=static/1 tun=intf/0 mode=dial_inst/3 encap=none/728 options[02d8]=npu
create_dev no-sysctl rgwy-chg frag-rfc accept_traffic=1
parent=vd2-2 index=0
src: 0:0.0.0.0/0.0.0.0:0

Fortinet Inc.
SD-WAN
dst: 0:0.0.0.0/0.0.0.0:0
seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0
dec: spi=03e01a45 esp=aes key=16 0beb519ed9f800e8b4c0aa4e1df7da35
ah=sha1 key=20 bc9f38db5296cce4208a69f1cc8a9f7ef4803c37
enc: spi=864f6dbb esp=aes key=16 1d26e3556afcdb9f8e3e33b563b44228
ah=sha1 key=20 564d05ef6f7437e1fd0a88d5fee7b6567f9d387e

mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=111 oif=70 oif=113 oif=71
destination(1): 33.1.1.1-33.1.1.100

mask=0x00 protocol=0 sport=0:65535 iif=0 dport=1-65535 oif=113 oif=71
destination(1): 33.1.1.101-33.1.1.200
Active dynamic BGP neighbor triggered by ADVPN shortcut
When a customer using SD-WAN with ADVPN has numerous IPv4 and IPv6 routes per spoke and there are many
spokes in the topology, using ADVPN with a route reflector-based design poses the following challenges:
l The hub FortiGate will experience high CPU usage due to the amount of processing required to reflect the routes to
the spoke FortiGates.
l Spoke FortiGates will learn many unnecessary routes.
For such cases, it is more suitable to deploy an IPv4- and IPv6-supported solution without a route-reflector that involves
an active dynamic BGP neighbor triggered by an ADVPN shortcut. This solution allows a spoke FortiGate to form a BGP
neighbor with another spoke FortiGate only after the shortcut tunnel between them has been established. As a result, the
spoke only learns routes from its BGP neighbors.
How this solution differs from typical SD-WAN with ADVPN
In a topology where the Spoke 1 and Spoke 2 FortiGates are connected directly to the Hub FortiGate, route reflection will
not be enabled. The Hub FortiGate is only configured with each spoke's summary route. An ADVPN shortcut tunnel is
established between the Spoke 1 and Spoke 2 FortiGates. The valid routing between the Spoke 1 and Spoke 2
FortiGate is still through the Hub FortiGate at this point.
When a host behind Spoke 1 tries to connect to a host behind Spoke 2, Spoke 1 first reaches the Hub based on the valid
routing table. The Hub determines that the destination is reachable, and the ADVPN shortcut tunnel between the spokes
is established. Then, Spoke 1 and Spoke 2 will actively initiate a BGP connection to each other over the shortcut. Once
established, they will exchange their routing information using BGP. On both spokes, BGP will resolve those routes on
the shortcut and update the routing table accordingly.
For this solution, the following IPv4/IPv6 BGP configuration settings are required:

Fortinet Inc.
SD-WAN
l The hub FortiGate should be configured with neighbor-group and neighbor-range/neighbor-range6.

l Each spoke FortiGate should be configured with neighbor-group and neighbor-range/neighbor-range6
like the hub. More importantly, each spoke should be configured with set passive disable to ensure spokes
are able to initiate dynamic BGP connections between each other.
l The hub FortiGate should have route reflection disabled (by default) where each neighbor-group setting should
have set route-reflector-client disable.
In the configuration, each of the spokes will form a BGP neighbor relationship with the hub. This is unchanged from the
typical SD-WAN with ADVPN configuration.
Example
This example configuration contains the following structure:

l Use SD-WAN member 1 (via ISP1) and its dynamic shortcuts for Financial Department traffic.
l Use SD-WAN member 2 (via ISP2) and its dynamic shortcuts for Engineering Department traffic.
l Internal subnets of Spoke 1:
l IPv4: 22.1.1.0/24
l IPv6: 2001:22::/64
l Internal subnets of Spoke 2:
l IPv4: 33.1.1.0/24
l Financial Department: 33.1.1.1 to 33.1.1.100
l Engineering Department: 33.1.1.101 to 33.1.1.200
l IPv6: 2001:33::/64
l Financial Department: 2001:33::1 to 2001:33::100
l Engineering Department: 2001:33::101 to 2001:33::200

Fortinet Inc.
SD-WAN
To configure the Hub FortiGate:
1. Configure the BGP settings (neighbor group and ranges):

config router bgp
set as 65100
edit "EDGE"
set activate6 disable
set remote-as 65100
set route-reflector-client disable
next
edit "EDGEv6"
set activate disable
set remote-as 65100
set route-reflector-client disable
next
end
edit 2
set prefix 10.200.1.0 255.255.255.0
set neighbor-group "EDGE"
next
end
config neighbor-range6
edit 2
set prefix6 2001::10:200:1:0/112
set neighbor-group "EDGEv6"
next
end
config network
edit 2
set prefix 10.200.1.0 255.255.255.0
next
edit 4
set prefix 33.0.0.0 255.0.0.0
next
edit 5
set prefix 22.0.0.0 255.0.0.0
next
end
config network6
edit 4
set prefix6 2001:33::/32
next
edit 2
set prefix6 2001:22::/32
next
end
end
2. Configure the static routes.

Fortinet Inc.
SD-WAN
a. For IPv4:
edit 33
set dst 33.0.0.0 255.0.0.0
set blackhole enable
set vrf 0
next
edit 22
set dst 22.0.0.0 255.0.0.0
set vrf 0
next
end
b. For IPv6:
config router static6
edit 33
set dst 2001:33::/32
set vrf 0
next
edit 22
set dst 2001:22::/32
set vrf 0
next
end
The following IPv4 summary routes are advertised:

l 33.0.0.0/8
l 22.0.0.0/8
The following IPv6 summary routes are advertised:
l 2001:33::/32
l 2001:22::/32
Because route reflection has been disabled in this example, initially, Spoke 1 will not know the local subnet of Spoke 2,
and Spoke 2 will not know the local subnet of Spoke 1. Therefore, for traffic routing, summary routes are configured on
the hub as blackhole routes and then advertised to the spokes using BGP.
For example, for traffic from the local subnet of Spoke 2 destined for the local subnet of Spoke 1:
l For the IPv4 case, the summary route 22.0.0.0/8, which includes the local subnet of Spoke 1 (22.1.1.0/24), is
advertised to Spoke 2. When Spoke 2 sends traffic destined for 22.1.1.0/24 to the Hub, the Hub forwards this traffic
to Spoke 1 since they are BGP neighbors.
l For the IPv6 case, the summary route 2001:22::/32, which includes the local subnet of Spoke 1 (2001:22::/64), is
advertised to Spoke 2. When Spoke 2 sends traffic destined for 2001:22::/64 to the Hub, the Hub forwards this traffic
to Spoke 1 since they are BGP neighbors.
Although traffic from spoke-to-spoke goes through the hub first, it is expected that the spoke will eventually go through
the shortcut tunnel.

Fortinet Inc.
SD-WAN
To configure the Spoke 1 FortiGate:

config system sdwan
set status enable
config zone
next
end
config members
edit 1
set interface "spoke1-1"
set cost 10
next
edit 2
set interface "spoke-2"
set cost 20
next
end
config health-check
edit "ping"
set server "11.11.11.11"
set source 10.200.1.100
set members 1 2
config sla
edit 1
next
end
next
end
config service
edit 1
set dst "financial-department"
next
edit 2
set dst "engineering-department"
next
edit 61
set addr-mode ipv6
set dst6 "financial-department-IPv6"
next
edit 62
set addr-mode ipv6
set dst6 "engineering-department-IPv6"
next
end
end
2. Configure the BGP settings (neighbor group and ranges):

Fortinet Inc.
SD-WAN
config router bgp

set as 65100
config neighbor
edit "10.200.1.1"
set remote-as 65100
next
edit "2001::10:200:1:1"
set remote-as 65100
next
end
edit "spokes"
set passive disable
set remote-as 65100
next
edit "spokesv6"
set passive disable
set remote-as 65100
next
end
edit 1
set prefix 10.200.1.0 255.255.255.0
set neighbor-group "spokes"
next
end
edit 1
set prefix6 2001::10:200:1:0/112
set neighbor-group "spokesv6"
next
end
config network
edit 3
set prefix 22.1.1.0 255.255.255.0
next
end
config network6
edit 1
set prefix6 2001:22::/64
next
end
end

Fortinet Inc.
SD-WAN
Verifying the configuration before a spoke-to-spoke shortcut VPN is established
IPv4 use case
To verify the status on Spoke 1:
1. Verify the BGP status:

VRF 0 BGP router identifier 10.200.1.100, local AS number 65100
10.200.1.1 4 65100 222 225 3 0 0 00:15:14 3
2. Verify the BGP routing table:

B 11.11.11.11/32 [200/0] via 10.200.1.1 (recursive via spoke1-1 tunnel 11.1.1.11),
00:15:19
(recursive via spoke1-2 tunnel
111.1.1.11), 00:15:19, [1/0]
00:15:19
(recursive via spoke1-2 tunnel 111.1.1.11),
00:15:19, [1/0]
00:15:19
00:15:19, [1/0]
IPv6 use case

# get router info6 bgp summary
2001::10:200:1:1 4 65100 223 224 4 0 0 00:15:21 3

# get router info6 routing-table bgp
B 2001::11:11:11:11/128 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1
tunnel ::11.1.1.11), 00:15:29
(recursive via spoke1-2

Fortinet Inc.
SD-WAN
tunnel ::111.1.1.11), 00:15:29, [1024/0]

B 2001:22::/32 [200/0] via 2001::10:200:1:1 (recursive via spoke1-1 tunnel
::11.1.1.11), 00:15:29
::111.1.1.11), 00:15:29, [1024/0]
::11.1.1.11), 00:15:29
::111.1.1.11), 00:15:29, [1024/0]
Verifying the configuration after a single spoke-to-spoke shortcut VPN is established
IPv4 use case
To trigger a single spoke-to-spoke shortcut VPN, on host 22.1.1.22, ping the host 33.1.1.33 in the Financial Department.
Because of the SD-WAN rule, use SD-WAN member 1 (via ISP1) and its dynamic shortcuts to reach hosts in the
Financial Department.

10.200.1.1 4 65100 252 254 3 0 0 00:17:22 3
10.200.1.101 4 65100 6 6 5 0 0 00:00:14 1
Spoke 1 has as its BGP neighbors:

l Hub FortiGate at 10.200.1.1
l Spoke 2 FortiGate at 10.200.1.101
00:17:26
111.1.1.11), 00:17:26, [1/0]
00:17:26
00:17:26, [1/0]
00:17:26
00:17:26, [1/0]
B 33.1.1.0/24 [200/0] via 10.200.1.101 (recursive via spoke1-1_0 tunnel 13.1.1.3),
00:00:18, [1/0]
The remote route learned from Spoke 2 through the spoke1_1_0 tunnel and using BGP is 33.1.1.0/24.

Fortinet Inc.
SD-WAN
IPv6 use case
To trigger a single spoke-to-spoke shortcut VPN over IPv6, on host 2001:22::22/64, ping the host 2001:33::33/64 in the
Financial Department. Because of the SD-WAN rule, use SD-WAN member 1 (via ISP1) and its dynamic shortcuts to
reach hosts in the Financial Department.

2001::10:200:1:1 4 65100 253 254 4 0 0 00:17:28 3
2001::10:200:1:101 4 65100 7 7 6 0 0 00:00:21 1
Spoke 1 has as its BGP neighbors:

l Hub FortiGate at 2001::10:200:1:1
l Spoke 2 FortiGate at 2001::10:200:1:101
tunnel ::11.1.1.11), 00:17:30
tunnel ::111.1.1.11), 00:17:30, [1024/0]
::11.1.1.11), 00:17:30
::111.1.1.11), 00:17:30, [1024/0]
::11.1.1.11), 00:17:30
::111.1.1.11), 00:17:30, [1024/0]
B 2001:33::/64 [200/0] via 2001::10:200:1:101 (recursive via spoke1-1_0 tunnel
::13.1.1.3), 00:00:24, [1024/0]
The remote route learned from Spoke 2 through the spoke1-1_0 tunnel and using BGP is 2001:33::/64.
Verifying the configuration after a second spoke-to-spoke shortcut VPN is established
IPv4 use case
To trigger a second spoke-to-spoke shortcut VPN, on host 22.1.1.22, ping the host 33.1.1.133 in the Engineering
Department. Because of the SD-WAN rule, use SD-WAN member 2 (via ISP2) and its dynamic shortcuts to reach hosts
in the Engineering Department.

Fortinet Inc.
SD-WAN

10.200.1.1 4 65100 263 265 3 0 0 00:18:12 3
10.200.1.101 4 65100 17 17 5 0 0 00:01:04 1
Total number of neighbors
Spoke 1 continues to have its BGP neighbors:

l Hub FortiGate at 10.200.1.1
l Spoke 2 FortiGate at 10.200.1.101
00:18:17
111.1.1.11), 00:18:17, [1/0]
00:18:17
00:18:17, [1/0]
00:18:17
00:18:17, [1/0]
B 33.1.1.0/24 [200/0] via 10.200.1.101 (recursive via spoke1-1_0 tunnel 13.1.1.3),
00:01:09
(recursive via spoke1-2_0 tunnel
113.1.1.3), 00:01:09, [1/0]
The remote route learned from Spoke 2 through the spoke1-2_0 tunnel and using BGP is 33.1.1.0/24.
IPv6 use case
To trigger a second spoke-to-spoke shortcut VPN over IPv6, on host 2001:22::22/64, ping the host 2001:33::133/64 in
the Engineering Department. Because of the SD-WAN rule, use SD-WAN member 2 (via ISP2) and its dynamic
shortcuts to reach hosts in the Engineering Department.


Fortinet Inc.
SD-WAN
2001::10:200:1:1 4 65100 264 265 4 0 0 00:18:18 3

2001::10:200:1:101 4 65100 19 19 6 0 0 00:01:11 1
Spoke 1 continues to have its BGP neighbors:

l Hub FortiGate at 2001::10:200:1:1
l Spoke 2 FortiGate at 2001::10:200:1:101
tunnel ::11.1.1.11), 00:18:20
tunnel ::111.1.1.11), 00:18:20, [1024/0]
::11.1.1.11), 00:18:20
::111.1.1.11), 00:18:20, [1024/0]
::11.1.1.11), 00:18:20
::111.1.1.11), 00:18:20, [1024/0]
B 2001:33::/64 [200/0] via 2001::10:200:1:101 (recursive via spoke1-1_0 tunnel
::13.1.1.3), 00:01:14
(recursive via spoke1-2_0 tunnel
::113.1.1.3), 00:01:14, [1024/0]
The remote route learned from Spoke 2 through the spoke1-2_0 tunnel and using BGP is 2001:33::/64.
SD-WAN monitor on ADVPN shortcuts
SD-WAN monitors ADVPN shortcut link quality by dynamically creating link monitors for each ADVPN link. The dynamic
link monitor on the spoke will use ICMP probes and the IP address of the gateway as the monitored server. These ICMP
probes will not be counted as actual user traffic that keeps the spoke-to-spoke tunnel alive.

Fortinet Inc.
SD-WAN
l When no shortcut is established:

Health Check(ping):
Seq(1 tunnel-1): state(alive), packet-loss(0.000%) latency(0.038), jitter(0.006) sla_
map=0x3
map=0x3
l When one shortcut is established:

Health Check(ping):
map=0x3
Seq(1 tunnel-1_0): state(alive), packet-loss(0.000%) latency(0.060), jitter(0.023) sla_
map=0x3
map=0x3
l When more than one shortcut is established:

Health Check(ping):
map=0x3
map=0x3
map=0x3
map=0x3
Hold down time to support SD-WAN service strategies
In a hub and spoke SD-WAN topology with shortcuts created over ADVPN, a downed or recovered shortcut can affect
which member is selected by an SD-WAN service strategy. When a downed shortcut tunnel recovers and the shortcut is
added back into the service strategy, the shortcut is held at a low priority until the hold down time has elapsed.
By default, the hold down time is zero seconds. It can be set to 0 - 10000000 seconds.
To configure the hold down time:
config system sdwan

config service
edit 1
set hold-down-time <integer>
next
end
end
Example
In this example, the hold down time is set to 15 seconds, and then the SD-WAN service is looked at before and after the
hold down elapses after a downed shortcut recovers.

Fortinet Inc.
SD-WAN
To configure the hold down time:
config system sdwan

config service
edit 1
set hold-down-time 15
next
end
end
To view which SD-WAN member is selected before and after the hold down time elapses:
Before the hold down time has elapsed:

Gen(34), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(priority), link-cost-factor(packet-
loss), link-cost-threshold(0), heath-check(ping)
Hold down time(15) seconds, Hold start at 2003 second, now 2010
1: vd2-1_0(86)
1: vd2-2_0(88)
Members(4):
1: Seq_num(1 vd2-1), alive, packet loss: 27.000%, selected
2: Seq_num(2 vd2-2_0), alive, packet loss: 0.000%, selected
Dst address(1):
33.1.1.101-33.1.1.200
After the hold down time has elapsed:

Gen(35), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(priority), link-cost-factor(packet-
loss), link-cost-threshold(0), heath-check(ping)

Fortinet Inc.
SD-WAN
1: vd2-2_0(88)
1: vd2-1_0(86)
Members(4):
Dst address(1):
33.1.1.101-33.1.1.200\
Adaptive Forward Error Correction
Forward Error Correction (FEC) is used to control and correct errors in data transmission by sending redundant data
across the VPN in anticipation of dropped packets occurring during transit. The mechanism sends out x number of
redundant packets for every y number of base packets.
Adaptive FEC considers link conditions and dynamically adjusts the FEC packet ratio:
l The FEC base and redundant packet relationship is dynamically adjusted based on changes to the network SLA
metrics defined in the SD-WAN SLA health checks. For example, when there is no or low packet loss in the network,
FEC can work on a low redundant level sending only one redundant packet for every 10 base packets. As packet
loss increases, the number of redundant packets sent can rise accordingly.
l FEC can be applied only to streams that are sensitive to packet loss. For Example, policies that allow the UDP
based VoIP protocol can enable FEC, while TCP based traffic policies do not. This reduces unnecessary bandwidth
consumption by FEC.
l Because FEC does not support NPU offloading, the ability to specify streams and policies that do not require FEC
allows those traffic to be offloaded. This means that not all traffic suffers a performance impact.
In this example, an IPsec tunnel is configured between two FortiGates that have FEC enabled and supporting
configuration to protect traffic that egresses FortiGate A and ingresses FortiGate B. The tunnel is an SD-WAN zone, and
an SLA health-check is used to monitor the quality of the VPN overlay. The intention is to apply FEC to UDP traffic that is
passing through the VPN overlay, while allowing all other traffic to pass through without FEC. An FEC profile is
configured to adaptively increase redundant levels if the link quality exceeds a 10% packet loss threshold, or the
bandwidth exceeds 950 Mbps.
The DMZ interface and IPsec tunnel vd1-p1 are SD-WAN members. FEC is enabled on vd1-p1, and health-check works
on vd1-p1.
To configure the FortiGates:
1. On FortiGate A, enable FEC for egress traffic and NPU offloading on the IPsec tunnel vd1-p1:
edit "vd1-p1"

Fortinet Inc.
SD-WAN
set npu-offload enable

set fec-egress enable
next
end
2. On FortiGate B, enable FEC for ingress traffic and NPU offloading on the IPsec tunnel vd1-p1:
edit "vd1-p1"
set npu-offload enable
set fec-ingress enable
next
end
3. On FortiGate A, configure SD-WAN:

The VPN overlay member (vd1-p1) must be included in the health-check and configured as the higher priority
member in the SD-WAN rule.
config system sdwan
set status enable
config zone
next
end
config members
edit 1
set interface "dmz"
next
edit 2
set interface "vd1-p1"
next
end
config health-check
edit "1"
set members 2
config sla
edit 1
next
end
next
end
config service
edit 1
set name "1"
set dst "all"
set src "172.16.205.0"
next
end
end
4. On FortiGate A, create a policy to specify performing FEC on UDP traffic, and a policy for other traffic:
edit 1

Fortinet Inc.
SD-WAN
set srcintf "port5"

set action accept
set srcaddr "172.16.205.0"
set dstaddr "all"
set service "ALL_UDP"
set fec enable
next
edit 2
set srcintf "any"
set dstintf "any"
set action accept
set srcaddr "all"
set dstaddr "all"
set service "ALL"
next
end
5. On FortiGate A, configure FEC mapping to bind network SLA metrics and FEC base and redundant packets:
config vpn ipsec fec
edit "m1"
config mappings
edit 1
set base 8
set redundant 2
set packet-loss-threshold 10
next
edit 2
set base 9
set redundant 3
set bandwidth-up-threshold 950000
next
end
next
end
The mappings are matched from top to bottom: packet loss greater than 10% with eight base and two redundant
packets, and then uploading bandwidth greater than 950 Mbps with nine base and three redundant packets.
6. On FortiGate A, apply the FEC mappings on vd1-p1:
edit "vd1-p1"
set fec-health-check "1"
set fec-mapping-profile "m1"
set fec-base 10
set fec-redundant 1
next
end
The FEC base and redundant values are used when the link quality has not exceeded the limits specified in the FEC
profile mapping. If fec-codec is set to xor the base and redundant packet values will not be updated.

Fortinet Inc.
SD-WAN
To verify the results:
1. Send TCP and UDP traffic from PC1 to PC2, then check the sessions on FortiGate A:
# diagnose sys session list
session info: proto=6 proto_state=01 duration=12 expire=3587 timeout=3600 flags=00000000

socktype=0 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/255
state=may_dirty npu
statistic(bytes/packets/allow_err): org=112/2/1 reply=112/2/1 tuples=2
tx speed(Bps/kbps): 0/0 rx speed(Bps/kbps): 0/0
orgin->sink: org pre->post, reply pre->post dev=15->102/102->15
gwy=172.16.209.2/172.16.205.11
hook=pre dir=org act=noop 172.16.205.11:39176->10.1.100.22:5001(0.0.0.0:0)
hook=post dir=reply act=noop 10.1.100.22:5001->172.16.205.11:39176(0.0.0.0:0)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 policy_id=2 pol_uuid_idx=719 auth_info=0 chk_client_info=0 vd=0
serial=00020f7a tos=ff/ff app_list=0 app=0 url_cat=0
sdwan_mbr_seq=2 sdwan_service_id=1
rpdb_link_id=ff000001 rpdb_svc_id=0 ngfwid=n/a
npu_state=0x5000c00
npu info: flag=0x82/0x81, offload=8/8, ips_offload=0/0, epid=249/74, ipid=74/86,
vlan=0x0000/0x0000
vlifid=74/249, vtag_in=0x0000/0x0001 in_npu=1/1, out_npu=1/1, fwd_en=0/0, qid=5/5

origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty fec
orgin->sink: org pre->post, reply pre->post dev=15->102/102->15 gwy=172.16.209.2/0.0.0.0
serial=000210fa tos=ff/ff app_list=0 app=0 url_cat=0
rpdb_link_id=ff000001 rpdb_svc_id=0 ngfwid=n/a
npu_state=0x5040000
no_ofld_reason: non-npu-intf
Non-FEC protected TCP traffic is offloaded, while FEC protected UDP traffic is not offloaded
2. On FortiGate A, check the health-check result and the corresponding FEC base and redundant packets:
Health Check(1):
Seq(2 vd1-p1): state(alive), packet-loss(0.000%) latency(0.168), jitter(0.021),
Because bandwidth-up is more than 950000kbps, base and redundant are set to 9 and 3:

Fortinet Inc.
SD-WAN
# diagnose vpn tunnel fec vd1-p1

egress:
enabled=1 base=9 redundant=3 codec=0 timeout=10(ms)
encode=6621 encode_timeout=6621 encode_fail=0
tx_data=6880 tx_parity=18601
ingress:
enabled=0 timeout=0(ms)
fasm_cnt=0 fasm_full=0
ipsec_fec_chk_fail=0 complete=0
rx_data=0 rx_parity=0
recover=0 recover_timeout=0 recover_fail=0
rx=0 rx_fail=0
3. Make packet loss more than 10%, then check the health-check result and the corresponding FEC base and
redundant packets again on FortiGate A:
Health Check(1):
Seq(2 vd1-p1): state(alive), packet-loss(15.000%) latency(0.168), jitter(0.017),
Because packet loss is more than 10%, entry one in FEC mapping is first matched, and base and redundant are set
to 8 and 2:
# diagnose vpn tunnel fec vd1-p1
egress:
enabled=1 base=8 redundant=2 codec=0 timeout=10(ms)
encode=6670 encode_timeout=6670 encode_fail=0
tx_data=6976 tx_parity=18748
ingress:
enabled=0 timeout=0(ms)
fasm_cnt=0 fasm_full=0
ipsec_fec_chk_fail=0 complete=0
rx_data=0 rx_parity=0
recover=0 recover_timeout=0 recover_fail=0
rx=0 rx_fail=0
Dual VPN tunnel wizard
This wizard is used to automatically set up multiple VPN tunnels to the same destination over multiple outgoing
interfaces. This includes automatically configuring IPsec, routing, and firewall settings, avoiding cumbersome and error-
prone configuration steps.

Fortinet Inc.
SD-WAN
To create a new SD-WAN VPN interface using the tunnel wizard:
2. In the Interface drop-down, click +VPN. The Create IPsec VPN for SD-WAN members pane opens.
3. Enter the required information, then click Next.

4. Review the settings then click Create.
5. Click Close to return to the SD-WAN page.
The newly created VPN interface will be highlighted in the Interface drop-down list.
6. Select the VPN interface to add it as an SD-WAN member, then click OK.
Duplicate packets on other zone members
When duplication rules are used, packets are duplicated on other good links within the SD-WAN zone and de-duplicated
on the destination FortiGate. Use force mode to force duplication on other links within the SD-WAN zone, or use on-
demand mode to trigger duplication only when SLA fails on the selected member.
The duplication rule is configured in the CLI by using the config duplication command. The following options can
be configured:

Fortinet Inc.
SD-WAN
Parameter Description
srcaddr Source address or address group names.
dstaddr Destination address or address group names.
srcaddr6 Source IPv6 address or IPv6 address group names.
dstaddr6 Destination IPv6 address or IPv6 address group names.
srcintf Incoming (ingress) interfaces or zones.
dstintf Outgoing (egress) interfaces or zones.
service Service and service group names.
packet-duplication Configure packet duplication method.

l disable: Disable packet duplication (default).
l force: Duplicate packets across all interface members of the SD-WAN zone.
l on-demand: Duplicate packets across all interface members of the SD-WAN
zone based on the link quality.
packet-de-duplication Enable/disable discarding of packets that have been duplicated (default =

disable).
The duplication-max-num <integer> option under config system sdwan is the maximum number of
interface members that a packet is duplicated on in the SD-WAN zone (2 - 4, default = 2). If this value is set to 3, the
original packet plus two more copies are created. If there are three member interfaces in the SD-WAN zone and the
duplication-max-num is set to 2, the packet duplication follows the configuration order, so the packets are
duplicated on the second member.
Example
The packet duplication feature works best in a spoke-spoke or hub-and-spoke topology. In this example, a hub-and-
spoke ADVPN topology is used. Before shortcuts are established, Hub 1 forwards the duplicate packets from Spoke 1 to
Spoke 2. Once shortcuts are established, Hub 1 is transparent, and duplicate packets are exchanged directly between
the spokes.

Fortinet Inc.
SD-WAN
To configure packet duplication between Spoke 1 and Spoke 2:
1. Configure Spoke 1:
config system sdwan
set status enable
config zone
next
edit "sdwanzone_v4"
next
end
config members
edit 1
set interface "t1"
set zone "sdwanzone_v4"
next
edit 4
set interface "t21"
next
edit 2
set interface "t2"
next
end
config health-check
edit "h1"
set interval 1000
set failtime 10
set members 1 2
config sla
edit 1

Fortinet Inc.
SD-WAN
next
end
next
end
config duplication
edit 1
set srcaddr "all"
set dstaddr "all"
set srcintf "port1"
set dstintf "sdwanzone_v4"
set service "ALL"
set packet-duplication force
set packet-de-duplication enable
next
end
end
2. Configure Spoke 2 with similar settings.
Duplicate packets based on SD-WAN rules
SD-WAN duplication rules can specify SD-WAN service rules to trigger packet duplication. This allows the duplication to
occur based on an SD-WAN rule instead of the source, destination, and service parameters in the duplication rule.
1. Packets can be forced to duplicate to all members of the same SD-WAN zone. See Duplicate packets on other zone
members on page 1025 for details.
For example, in Spoke 1 set packet-duplication to force so that when a client sends a packet to the server, it
is duplicated to all members of the same zone as long as its health check is alive. If a members health check is
dead, then the member is removed from the SD-WAN duplication zone.
2. Packets can be duplicated to other members of the SD-WAN zone on-demand only when the condition of the link is
not good enough.
Set packet-duplication to on-demand. If sla-match-service is disabled, when all the SLAs of the
member exceed threshold (sla_map=0), the packet is duplicated. But when the SLAs are within threshold (sla_
map!=0), the packet is not duplicated.
If sla-match-service is enabled, then only the SLA health checks and targets used in the service rule need to
exceed threshold in order to trigger packet duplication.
3. Packets can be duplicated to all members of the same SD-WAN zone when the traffic matches one or more regular
SD-WAN service rules.
The following example shows the third type of packet duplication.

Fortinet Inc.
SD-WAN
In this example, SD-WAN is configured with three members: vpn1, vpn2, and vpn3. Service rule 1 controls all traffic from
10.100.20.0/24 to 172.16.100.0/24 using member 1.
To send a duplicate of the traffic that matches service rule 1 using member 2, members 1 and 2 are added to the same
SD-WAN zone, and a duplication rule is configured with service-id set to 1.
To send a duplicate of the traffic that matches service rule 1 using member 2:
config system sdwan

set status enable
config zone
next
edit "zone2"
next
end
config members
edit 1
set interface "vpn1"
next
edit 2
next
edit 3
set zone "zone2"
next
end
config service
edit 1
set dst "172.16.100.0"
set src "10.100.20.0"
next

Fortinet Inc.
SD-WAN
end
config duplication
edit 1
set service-id 1
set packet-duplication force
next
end
end
Interface based QoS on individual child tunnels based on speed test results
In a hub and spoke SD-WAN topology that uses dial-up VPN overlays, QoS can be applied on individual tunnels based
on the measured bandwidth between the hub and spokes. The FortiGate can use the built in speed test to dynamically
populate the egress bandwidth to individual dial-up tunnels from the hub.
A bandwidth limit, derived from the speed test, and a traffic shaping profile can be applied on the dial-up IPsec tunnel
interface on the hub. A class ID and percentage based QoS settings can be applied to individual child tunnels using a
traffic shaping policy and profile.
CLI commands
If the interface is an IPsec dial-up server, then egress shaping profile type can only be set to policing; it cannot be set
to queuing:
config firewall shaping-profile
edit <profile-name>
set type policing
next
end
The outbandwidth value is dynamically obtained from the speed test results for each individual child tunnel, and should
not be set manually:
edit <dialup-server-phase1-name>
set egress-shaping-profile <profile-name>
set outbandwidth <bandwidth>
next
end

Fortinet Inc.
SD-WAN
Example
In this example, the hub is configured as a VPN dial-up server and both of the spokes are connected to the hub. It is
assumed that the VPN configuration is already done, with a dynamic gateway type and kernel device creation (net-
device) disabled. Only one SD-WAN interface is used, so there is only one VPN overlay member in the SD-WAN zone.
Multiple WAN interfaces and VPN overlays could be used.
The VPN interfaces and IP addresses are:
FortiGate Interface IP Address
FGT_A (Hub) hub-phase1 10.10.100.254
FGT_B (Spoke) spoke11-p1 10.10.100.2
FGT_D (Spoke) spoke21-p1 10.10.100.3
The hub VPN has two child tunnels, one to each spoke.
The speed test configuration is shown in Running speed tests from the hub to the spokes in dial-up IPsec tunnels on
page 1156. This example shows applying a shaping profile to the hub's tunnel interface in order to apply interface based
traffic shaping to the child tunnels.
A traffic shaping policy is used to match and assign traffic to the classes in the shaping profile.
To configure the hub FortiGate (FGT_A) and check the results:
1. Configure the hub FortiGate (FGT_A) as in Running speed tests from the hub to the spokes in dial-up IPsec tunnels
on page 1156.
2. Configure the shaping profile:
edit "profile_1"
config shaping-entries
edit 1

Fortinet Inc.
SD-WAN
set class-id 2
set priority low
set guaranteed-bandwidth-percentage 10
set maximum-bandwidth-percentage 10
next
edit 2
set class-id 3
set priority medium
next
edit 3
set class-id 4
set priority high
next
end
set default-class-id 2
next
end
3. Configure a traffic shaping policy:

config firewall shaping-policy
edit 2
set service "ALL"
set class-id 3
set srcaddr "all"
set dstaddr "all"
next
end
In this example, all traffic through the hub-phase1 interface is put into class ID 3. Class IDs an be assigned based on
your traffic requirements.
4. At the schedules time, the speed test will start for the hub-phase1 interface from the hub to the spokes. The speed
test results can then be dynamically applied on individual child tunnels as egress traffic shaping, and the class ID
percentage based QoS settings is applicable on them as templates.
------------------------------------------------------
name=hub-phase1_0 ver=2 serial=c 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 dst_
mtu=1500 dpd-link=on remote_location=0.0.0.0 weight=1
...
egress traffic control:
bandwidth=737210(kbps) lock_hit=0 default_class=2 n_active_class=3
class-id=2 allocated-bandwidth=73720(kbps) guaranteed-
bandwidth=73720(kbps)
max-bandwidth=73720(kbps) current-bandwidth=0(kbps)
priority=low forwarded_bytes=52
dropped_packets=0 dropped_bytes=0

Fortinet Inc.
SD-WAN
priority=medium forwarded_bytes=0
priority=high forwarded_bytes=0
------------------------------------------------------
name=hub-phase1_1 ver=2 serial=d 172.16.200.1:0->172.16.200.2:0 tun_id=172.16.200.2 dst_
...
The guaranteed and maximum bandwidths equal 10% of the speed test result, as expected.
SD-WAN in large scale deployments
Phase 2 selectors can be used to inject IKE routes on the ADVPN shortcut tunnel. When configuration method (mode-
cfg) is enabled in IPsec phase 1 configuration, enabling mode-cfg-allow-client-selector allows custom phase
2 selectors to be configured. By also enabling the addition of a route to the peer destination selector (add-route) in the
phase 1 configuration, IKE routes based on the phase 2 selectors can be injected. This means that routes do not need to
be reflected on the hub to propagate them between spokes, avoiding possible BGP daemon process load issues and
improving network scalability in a large-scale ADVPN network.
Route map rules can apply priorities to BGP routes. On the hub, priorities can be set in a route map's rules, and the route
map can be applied on BGP routes. This allows the hub to mark the preferred path learned from the spokes with a
priority value (lower priority is preferred), instead of using multiple SD-WAN policy routes on the hub. When a preferred
outbound route map (route-map-out-preferable) is also configured in an SD-WAN neighbor on the spoke,
deploying SD-WAN rules on the hub to steer traffic from the hub to a spoke is unnecessary.
SD-WAN members' local cost can be exchanged on the ADVPN shortcut tunnel so that spokes can use the remote cost
as tiebreak to select a preferred shortcut. If multiple shortcuts originate from the same member to different members on
the same remote spoke, then the remote cost on the shortcuts is used as the tiebreak to decide which shortcut is
preferred.

Fortinet Inc.
SD-WAN
In this example, SD-WAN is configured on an ADVPN network with a BGP neighbor per overlay.
Instead of reflecting BGP routes with the route-reflector on the hub, when the shortcuts are triggered, IKE routes on the
shortcuts are directly injected based on the configured phase 2 selectors to allow routes to be exchanged between
spokes.
Routes between the hub and the spokes are exchanged by BGP, and the spokes use the default route to send spoke-to-
spoke traffic to the hub and trigger the shortcuts.
Instead of configuring SD-WAN rules on the hub, different priorities are configured on the BGP routes by matching
different BGP communities to steer traffic from the hub to the spokes.
To configure Spoke 1:
1. Configure phase 1:
edit "spoke11-p1"
...
set ike-version 2
set add-route enable
set mode-cfg enable
set mode-cfg-allow-client-selector enable
set link-cost 11
...
next
edit "spoke12-p1"
...
set ike-version 2
set mode-cfg enable
set link-cost 21
next
end
edit "spoke11-p2"

Fortinet Inc.
SD-WAN
...
set src-name "LAN_Net"
set dst-name "all"
next
edit "spoke12-p2"
...
set dst-name "all"
next
end
3. Configure an address group:

Spoke 1 uses LAN subnet 10.1-3.100.0/24.
config firewall addrgrp
edit "LAN_Net"
set member "10.1.100.0" "10.2.100.0" "10.3.100.0"
next
end
4. Configure route maps:

l If overlay 1 to the hub is in SLA, attach "65000:1" to the BGP routes advertised to the hub over overlay 1.
l If any overlay to the hub is out of SLA, attach "65000:9999" to the BGP routes advertised to the hub over any
overlay.
edit "HUB_CARRIER1"
config rule
edit 1
...
next
end
...
next
edit "HUB_CARRIER2"
config rule
edit 1
...
next
end
...
next
edit "HUB_BAD"
config rule
edit 1
...
next
end
...
next
end
5. Configure BGP and SD-WAN members and neighbors:

Fortinet Inc.
SD-WAN
config router bgp

set as 65412
config neighbor
edit "10.10.15.253"
set remote-as 65412
set route-map-out "HUB_BAD"
set route-map-out-preferable "HUB_CARRIER1"
...
next
edit "10.10.16.253"
set remote-as 65412
...
next
end
end
config system sdwan
config members
edit 1
set interface "spoke11-p1"
set cost 10
next
edit 2
set cost 20
next
end
config neighbor
edit "10.10.15.253"
set member 1
set health-check "1"
set sla-id 1
next
edit "10.10.16.253"
set member 2
set sla-id 1
next
end
end
To configure Spoke 2:
edit "spoke21-p1"
...
set ike-version 2
set mode-cfg enable
set link-cost 101

Fortinet Inc.
SD-WAN
...
next
edit "spoke22-p1"
...
set ike-version 2
set mode-cfg enable
set link-cost 201
next
end
edit "spoke21-p2"
...
set dst-name "all"
next
edit "spoke22-p2"
...
set dst-name "all"
next
end
3. Configure an address group:

Spoke 2 uses LAN subnet 192.168.5-7.0/24.
edit "LAN_Net"
set member "192.168.5.0" "192.168.6.0" "192.168.7.0"
next
end
4. Configure route maps:

l If any overlay to the hub is out of SLA, attach "65000:9999" to the BGP routes advertised to the hub over any
overlay.
edit "HUB_CARRIER1"
config rule
edit 1
...
next
end
...
next
edit "HUB_CARRIER2"
config rule
edit 1

Fortinet Inc.
SD-WAN

...
next
end
...
next
edit "HUB_BAD"
config rule
edit 1
...
next
end
...
next
end
5. Configure BGP and SD-WAN members and neighbors:

config router bgp
set as 65412
config neighbor
edit "10.10.15.253"
set remote-as 65412
...
next
edit "10.10.16.253"
set remote-as 65412
...
next
end
end
config system sdwan
config members
edit 1
set cost 10
next
edit 2
set cost 20
next
end
config neighbor
edit "10.10.15.253"
set member 1
set sla-id 1
next
edit "10.10.16.253"
set member 2

Fortinet Inc.
SD-WAN
set sla-id 1
next
end
end
1. Configure the route maps:

l Set the priority to 100 for routes with community 65000:1, indicating that they are in SLA for overlay 1.
l Set the priority to 200 for routes with community 65000:2, indicating that they are in SLA for overlay 2.
l Set the priority to 9999 for routes with community 65000:9999, indicating that they are out of SLA for any
overlay.
edit "Set_Pri"
config rule
edit 1
set match-community "comm_65000:1"
set set-priority 100
next
edit 2
next
edit 3
next
end
next
end
2. Configure BGP:
config router bgp
set as 65412
edit "advpn"
set remote-as 65412
set route-map-in "Set_Pri"
...
next
edit "advpn2"
set remote-as 65412
set route-map-in "Set_Pri"
...
next
end
edit 1
set prefix 10.10.15.0 255.255.255.0
next
edit 2
set prefix 10.10.16.0 255.255.255.0
set neighbor-group "advpn2"

Fortinet Inc.
SD-WAN
next
end
end
1. Check the routing tables on the spokes:

Spoke 1:
spoke-1 (root) # get router info routing-table all
B* 0.0.0.0/0 [200/0] via 10.10.15.253 (recursive is directly connected, spoke11-
p1), 00:01:17, [1/0] // default route to hub
[200/0] via 10.10.16.253 (recursive is directly connected,
spoke12-p1), 00:01:17, [1/0]
B 9.0.0.0/24 [200/0] via 10.10.15.253 (recursive is directly connected, spoke11-
p1), 00:01:17, [1/0] // route to the server behind hub
spoke12-p1), 00:01:17, [1/0]
C 10.1.100.0/24 is directly connected, port2 // route to PC 1
C 10.10.15.0/24 is directly connected, spoke11-p1 // overlay 1
C 10.10.15.1/32 is directly connected, spoke11-p1
Spoke 2:
spoke-2 (root) # get router info routing-table all
B* 0.0.0.0/0 [200/0] via 10.10.15.253 (recursive is directly connected, spoke21-
p1), 00:46:14, [1/0] // default route to hub
spoke22-p1), 00:46:14, [1/0]
B 9.0.0.0/24 [200/0] via 10.10.15.253 (recursive is directly connected, spoke21-
p1), 00:46:18, [1/0] // route to the server behind hub
spoke22-p1), 00:46:18, [1/0]
C 192.168.5.0/24 is directly connected, port2 // route to PC 2
2. Send traffic from PC 1 to PC 2 and trigger the shortcut:

The IKE routes on the shortcut are directly injected based on the phase 2 selectors, and spoke-to-spoke traffic then
goes directly through the shortcut instead of going through the hub.
Spoke 1:
spoke-1 (root) # get router info routing-table static
S 192.168.5.0/24 [15/0] via spoke11-p1_0 tunnel 172.16.200.4 vrf 0, [1/0]
spoke-1 (root) # diagnose sniffer packet any 'host 192.168.5.44' 4
interfaces=[any]
1.446327 spoke11-p1_0 out 10.1.100.22 -> 192.168.5.44: icmp: echo request

Fortinet Inc.
SD-WAN
1.446521 spoke11-p1_0 in 192.168.5.44 -> 10.1.100.22: icmp: echo reply

1.446536 port2 out 192.168.5.44 -> 10.1.100.22: icmp: echo reply
Spoke 2:
spoke-2 (root) # get router info routing-table static
3. Confirm that the overlays are in SLA on the spokes:

Spoke 1:
spoke-1 (root) # diagnose sys sdwan neighbor
Neighbor(10.10.15.253): member(1)role(standalone)
Health-check(1:1) sla-pass selected alive
Spoke 2:
4. On the hub, check that the routes received from the spokes have the expected priorities:
hub (root) # diagnose ip route list | grep proto=11
tab=254 vf=0 scope=0 type=1 proto=11 prio=100 0.0.0.0/0.0.0.0/0->10.1.100.0/24
pref=0.0.0.0 gwy=10.10.15.1 dev=101(hub-phase1)
pref=0.0.0.0 gwy=10.10.16.1 dev=102(hub2-phase1)
The priority set by the hub's route map is based on the community string received from the spoke. The route with a
lower priority value is selected, so traffic to Spoke 1 goes out on the hub-phase1 tunnel:
hub (root) # diagnose sniffer packet any 'host 9.0.0.2' 4
interfaces=[any]
2.735456 R190 in 9.0.0.2 -> 10.1.100.22: icmp: echo request
2.735508 hub-phase1 out 9.0.0.2 -> 10.1.100.22: icmp: echo request
2.735813 hub-phase1 in 10.1.100.22 -> 9.0.0.2: icmp: echo reply
2.735854 R190 out 10.1.100.22 -> 9.0.0.2: icmp: echo reply
5. If overlay 1 goes out of SLA, the priorities of the routes on the hub are updated and traffic from the hub to Spoke 1
goes through overlay 2:
Spoke 1:
Health-check(1:1) sla-fail alive

Fortinet Inc.
SD-WAN
Spoke 2:
Health-check(1:1) sla-fail alive
Hub:
hub (root) # diagnose ip route list | grep proto=11
hub (root) # diagnose sniffer packet any 'host 9.0.0.2' 4
interfaces=[any]
3.550181 R190 in 9.0.0.2 -> 10.1.100.22: icmp: echo request
3.550234 hub2-phase1 out 9.0.0.2 -> 10.1.100.22: icmp: echo request
3.550713 hub2-phase1 in 10.1.100.22 -> 9.0.0.2: icmp: echo reply
3.550735 R190 out 10.1.100.22 -> 9.0.0.2: icmp: echo reply
6. Verify the service diagnostics on Spoke 1:


Tie break: cfg
Members(2):
1: Seq_num(1 spoke11-p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(10),
selected
selected
Src address(1):
10.1.100.0-10.1.100.255
Dst address(1):
0.0.0.0-255.255.255.255
7. Verify the service diagnostics on Spoke 2:


Tie break: cfg
Members(2):
selected
selected
Src address(1):

Fortinet Inc.
SD-WAN
192.168.5.0-192.168.5.255
Dst address(1):
0.0.0.0-255.255.255.255
8. Trigger shortcuts between Spoke 1 and Spoke 2:

l Shortcuts spoke11-p1_1 and spoke11-p1_0 originate from spoke11-p1.
l spoke11-p1_1 corresponds to spoke21-p1_0 on Spoke 2.
l spoke11-p1_0 corresponds to spoke22-p1_0 on Spoke 2.
Spoke 1:

Tie break: cfg
3: seq_num(1), interface(spoke11-p1):
1: spoke11-p1_0(80)
2: spoke11-p1_1(81)
Members(4):
1: Seq_num(1 spoke11-p1_1), alive, sla(0x1), gid(0), remote cost(101), cfg_order(0),
local cost(10), selected
selected
selected
Src address(1):
10.1.100.0-10.1.100.255
Dst address(1):
0.0.0.0-255.255.255.255
Spoke 2:

Tie break: cfg
1: spoke21-p1_0(75)
1: spoke22-p1_0(74)
Members(4):
selected
selected

Fortinet Inc.
SD-WAN
Src address(1):
192.168.5.0-192.168.5.255
Dst address(1):
0.0.0.0-255.255.255.255
The spoke11-p1_1 shortcut on Spoke 1 is preferred over spoke11-p1_0 due to the lower remote link cost of 101
when they have the same local SD-WAN member cost of 10.
9. Verify the policy route list on Spoke 1:
id=2131755009(0x7f100001) vwl_service=1(1) vwl_mbr_seq=1 1 1 2 dscp_tag=0xfc 0xfc

flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0(any) dport=1-65535 path
(4) oif=81(spoke11-p1_1) oif=80(spoke11-p1_0) oif=54(spoke11-p1) oif=55(spoke12-p1)
source(1): 10.1.100.0-10.1.100.255
destination(1): 0.0.0.0-255.255.255.255
Keeping sessions in established ADVPN shortcuts while they remain in SLA
In an SD-WAN hub and spoke configuration where ADVPN is used, when a primary shortcut goes out of SLA, traffic
switches to the backup shortcut. During idle timeout, sessions will prefer using the primary parent tunnel and try to
establish a new primary shortcut. However, because it is out of SLA, traffic switches back to the backup shortcut, which
causes unnecessary traffic interruption.
The sla-stickiness option keeps existing sessions on the established ADVPN shortcuts while they remain in SLA
instead of switching to a new link every idle timeout. New sessions will be routed through the primary shortcut if it is in
SLA.
config system sdwan
config service
edit <id>
set mode sla
set sla-stickiness {enable | disable}
next
end
end
The sla-stickiness option can be applied in the following use cases.
Use case 1:
1. The sessions will switch over to the backup shortcut due to the primary shortcut being out of SLA.
2. After an idle timeout, the primary shortcut is torn down, and the routes will be reinstalled on the primary parent
tunnel.
3. When sla-stickiness is enabled, even though the primary parent tunnel is preferred, established ADVPN
sessions will remain on the backup shortcut (stickiness) instead of switching to the primary parent tunnel.
4. New sessions will be routed to the primary parent tunnel and trigger the primary shortcut, then traffic switches to the
primary shortcut if it is in SLA.

Fortinet Inc.
SD-WAN
Use case 2:
1. The sessions will switch over to the backup shortcut due to the primary shortcut being out of SLA.
2. After some time, the primary shortcut becomes in SLA.
3. When sla-stickiness is enabled, even though primary shortcut is preferred, established ADVPN sessions will
remain on the backup shortcut (stickiness) instead of switching to the primary shortcut.
4. New sessions will be routed through the primary shortcut.
Example configuration
The following example demonstrates using the sla-stickiness option in use case 1.
After an idle timeout occurs, existing sessions remain on the spoke12-p1_0 backup shortcut tunnel. New sessions will try
to create a shortcut over spoke11-p1, but will fall back to spoke12-p1_0 when it detects spoke11-p1 is out of SLA.
To configure shortcut stickiness for ADVPN shortcuts:
1. Configure SD-WAN on the Spoke_1 FortiGate:

config system sdwan
set status enable
config zone
next
end
config members
edit 1
next
edit 2
next
end
config health-check
edit "1"

Fortinet Inc.
SD-WAN
set members 1 2
config sla
edit 1
next
end
next
end
config service
edit 1
set name "1"
set mode sla
set sla-stickiness enable
set dst "all"
set src "10.1.100.0"
config sla
edit "1"
set id 1
next
end
next
end
end
2. Verify the SD-WAN configuration.

a. Verify the health check status:
Health Check(1):
Seq(1 spoke11-p1): state(alive), packet-loss(0.000%) latency(0.368), jitter(0.051),
mos(4.404), bandwidth-up(999999), bandwidth-dw(1000000), bandwidth-bi(1999999) sla_
map=0x1
Seq(2 spoke12-p1): state(alive), packet-loss(0.000%) latency(0.211), jitter(0.019),
mos(4.404), bandwidth-up(999999), bandwidth-dw(999979), bandwidth-bi(1999978) sla_
map=0x1
b. Verify the service status:

Service(1): Address Mode(IPV4) flags=0x2200 use-shortcut-sla sla-stickiness

Tie break: cfg
Members(2):
selected
selected
Src address(1):
10.1.100.0-10.1.100.255
Dst address(1):
0.0.0.0-255.255.255.255
The SD-WAN service rule prefers the primary parent tunnel (spoke11-p1) over the backup parent tunnel
(spoke12-p1) before shortcuts are established.
3. Send traffic from PC-1 to PC-2 to trigger the primary shortcut. Verify the diagnostics.

Fortinet Inc.
SD-WAN
a. Run a sniffer trace:

interfaces=[any]
14.878905 spoke11-p1 out 10.1.100.22 -> 192.168.5.44: icmp: echo request
14.879842 spoke11-p1 in 192.168.5.44 -> 10.1.100.22: icmp: echo reply
The SD-WAN service rule sends traffic to the parent tunnel (spoke11-p1) initially, and then switches to the
primary shortcut tunnel (spoke11-p1_0) once it is established.

Tie break: cfg
1: spoke11-p1_0(57)
Members(3):
1: Seq_num(1 spoke11-p1_0), alive, sla(0x1), gid(0), cfg_order(0), local cost(0),
selected
selected
selected
Src address(1):
10.1.100.0-10.1.100.255
Dst address(1):
0.0.0.0-255.255.255.255
The SD-WAN service rule prefers the primary shortcut tunnel (spoke11-p1_0) over other tunnels.
4. Make the primary shortcut be out of SLA. The traffic will switch to the backup parent tunnel and trigger the backup
shortcut. Verify the diagnostics.
interfaces=[any]

Fortinet Inc.
SD-WAN
When the primary shortcut tunnel goes out of SLA (spoke11-p1_0, alive, sla(0x0)), traffic reroutes to the
backup parent tunnel (spoke12-p1) and then to the backup shortcut tunnel (spoke12-p1_0) once established.

Tie break: cfg
1: spoke11-p1_0(62)
1: spoke12-p1_0(63)
Members(4):
selected
selected
selected
selected
Src address(1):
10.1.100.0-10.1.100.255
Dst address(1):
0.0.0.0-255.255.255.255
The backup shortcut tunnel (spoke12-p1_0) is now preferred.

5. After an idle timeout, the primary shortcut is torn down. The primary parent tunnel is now preferred, but traffic is still
kept on the backup shortcut due to sla-stickiness being enabled. Verify the diagnostics.
a. Verify the service status:

Tie break: cfg
1: spoke12-p1_0(63)
Members(3):
selected
selected
selected
Src address(1):
10.1.100.0-10.1.100.255
Dst address(1):
0.0.0.0-255.255.255.255
b. Run a sniffer trace:

Fortinet Inc.
SD-WAN

interfaces=[any]
6. Send new traffic from PC1 to PC2. The traffic is routed to the primary parent tunnel and triggers the primary
shortcut, then traffic will switch to the primary shortcut if it is in SLA. Verify the connection.
interfaces=[any]
...
At first, traffic tries to go to the primary parent tunnel so that it can trigger the primary shortcut to establish. The
primary shortcut (spoke11-p1_0) is in SLA and new traffic flows through it.
...
After the primary shortcut goes out of SLA, the traffic switches to the backup shortcut (spoke12-p1_0).

Tie break: cfg
1: spoke11-p1_0(67)

Fortinet Inc.
SD-WAN
1: spoke12-p1_0(66)
Members(4):
selected
selected
selected
selected
Src address(1):
10.1.100.0-10.1.100.255
Dst address(1):
0.0.0.0-255.255.255.255
New traffic switches back to the backup shortcut while the primary shortcut is still out of SLA.
SD-WAN multi-PoP multi-hub large scale design and failover
FortiOS 7.2.0 introduced a feature to define the minimum number of SD-WAN interface members that must meet SLA in
order for the spoke to select a hub to process its SD-WAN traffic. This design is suitable for a single-PoP multi-hub
architecture in order to achieve hub-to-hub failover. See Using multiple members per SD-WAN neighbor configuration
on page 974.
In FortiOS 7.4.1 and later, the design is enhanced to support a multi-PoP multi-hub architecture in which incoming and
outgoing traffic failover between PoPs is supported.

Fortinet Inc.
SD-WAN
Based on the preceding diagram, incoming and outgoing traffic to the spoke is preferred over PoP1. If a single hub within
PoP1 goes out of SLA, traffic will continue to flow through the PoP. If the minimum number of members to meet SLA in
the PoP cannot be met, then traffic will fail over to PoP2.
The following enhancements have been made to support the multi-PoP failover scenario.
l Add minimum-sla-meet-members setting in the SD-WAN zone configurations and zone-mode setting in the
SD-WAN service configurations:
config system sdwan
config zone
edit <name>
set minimum-sla-meet-members <integer>
next
end
config service
edit <id>
set mode sla
set zone-mode {enable | disable}
next
end
end
When zone-mode is enabled on a SD-WAN service rule, the traffic is steered based on the status of the zone.

Fortinet Inc.
SD-WAN
The state of the health check referenced in the SD-WAN service can be defined as follows:
l If the number of in SLA members in a zone is less than the minimum-sla-meet-members, then the zone's
state is out of SLA; otherwise, it is in SLA.
l If a zone's state is out of SLA, then all members in the zone are out of SLA.
l If a zone's state is in SLA, then the health check's state of individual members in the zone is determined by its
own state.
l Add service-id setting in the SD-WAN neighbor configurations:
config system sdwan
config neighbor
edit <bgp_neighbor_ip>
set member <member_id>
set service-id <id>
next
end
end
The SD-WAN neighbor’s behavior can be determined by SD-WAN service and naturally synchronizes with SD-
WAN service.
l The SD-WAN service defines priority zones, whose SLA state determines the advertised community preferable
string.
l The SD-WAN service defines the hold-down-time, which determines how long an advertised community
preferable string can be kept when it is expected to be changed.
l Add sla-stickness setting in the SD-WAN service configurations:
config system sdwan
config service
edit <id>
set mode sla
set sla-stickiness {enable | disable}
next
end
end
The switch-over of an existing session is determined as follows:

l If the outgoing interface of the session is in SLA, then the session can keep its outgoing interface.
l Otherwise, the session switches to a preferable path if one exists.
l Allow the neighbor group to be configured in the SD-WAN neighbor configurations:
config system sdwan
config neighbor
edit <bgp_neighbor_group>
set member <member_id>
set health-check <name>
set sla-id <id>
next
end
end
Outgoing path control
The outgoing path from spoke to hub operates as follows:

Fortinet Inc.
SD-WAN
1. Overlays to the primary and secondary PoP are assigned separately into an SD-WAN primary and secondary zone
on the spoke.
2. One SD-WAN service rule is defined to include these zones as SD-WAN members.
3. When the primary zone is in SLA (minimum-sla-meet-members is met), the SD-WAN service rule steers traffic
to the in SLA overlay members.
4. When the primary zone is out of SLA (minimum-sla-meet-members is not met), the SD-WAN service rule steers
traffic to the in SLA overlay members in the secondary zone.
5. When the primary zone SLA is recovered:
a. If sla-stickness is disabled on the SD-WAN service rule, then traffic will wait the duration of the hold-
down-time before switching back to in SLA overlays in the primary zone.
b. If sla-stickness is enabled on the SD-WAN service rule, then existing traffic will be kept on the in SLA
overlays on the secondary zone, but new traffic will be steered to in SLA overlays in the primary zone.
Incoming path control
The incoming traffic from the core/external peers, to PoP, to spoke operates as follows:
1. When the primary zone is in SLA, the spoke uses the preferable route map to advertise local routes with the in SLA
community to hubs in the primary and secondary PoPs.
a. Hubs in the primary PoP translate the in SLA community into a short AS path and advertise it to external peers
to attract incoming traffic.
b. Hubs in the secondary PoP translate the in SLA community into a longer AS path and advertise it to external
peers to deflect incoming traffic.
2. If the number of in SLA overlays in the primary zone is less than the minimum-sla-meet-members, then the
spoke will use the default route map to advertise routes instead of with an out of SLA community to hubs in the
primary PoP.
a. Hubs in the primary PoP translate the out of SLA community into a longest AS path, and advertise it to external
peers to deflect incoming traffic.
b. As a result, inbound traffic is routed to hubs in the secondary PoP.
3. When the primary zone SLA is recovered:
a. The spoke will wait the duration of the predefined hold-down-time in the SD-WAN service rule to use the
preferable route map again to advertise routes with the in SLA community to hubs in the primary PoP.
b. As a result, inbound traffic will be routed back to hubs in the primary PoP.
Neighbor group configuration
By configuring the neighbor group for spokes under the hub's SD-WAN neighbor configuration, if all paths from the hub
to external peers are detected as out of SLA, then the hub will use the default route map to deny external routes to
spokes that belong to this neighbor group defined on the hub. As a result, spokes will skip that specific hub and connect
to external peers from other hubs.
This allows spokes to only measure overlay quality to each hub, and hubs to manage health checks to services by
external peers. This significantly decreases the number of health check probes directly from the spoke to services and
decreases the overall complexity. The complexity is further simplified by using multiple VRFs or segmentation where
each spoke needs to send health check probes.

Fortinet Inc.
SD-WAN
Example
This example configuration contains the following components:

l Two PoPs:
l The primary PoP has two hubs (Hub-1 and Hub-2).
l The secondary PoP has one hub (Hub-3).
l Spoke-1 has six overlays, with two overlay connections to each hub.
l Spoke-1 has three BGP neighbors, with one BGP neighbor for each hub.
l All BGP neighbors are established on loopback IPs.
l Each hub has two paths to external peers.
Normally, outbound and inbound traffic go through hubs in the primary PoP. If the number of in SLA overlays to the
primary PoP is less than the minimum-sla-meet-members (set to 2 in this example), bi-directional traffic needs to be
switched to hubs in the secondary PoP. But when the primary PoP recovers and the minimum-sla-meet-members is
met again, bi-directional traffic is forced back to hubs in the primary PoP after the predefined hold-down-time
duration.
The hubs do not require SD-WAN configurations to the spokes. However, they use SD-WAN for connections to external
peer routers.
Configuring the FortiGates
The following configurations highlight important routing and SD-WAN settings that must be configured on the spoke and
the hubs. It is assumed that other configurations such as underlays, IPsec VPN overlays, loopbacks, static routes, and
so on are already configured.

Fortinet Inc.
SD-WAN
To configure Spoke-1:
1. Create the primary (PoP1) and secondary (PoP2) zones, and set the minimum-sla-meet-members to 2 on
PoP1:
config system sdwan
set status enable
config zone
next
edit "PoP1"
next
edit "PoP2"
next
end
end
2. Add the overlay members to each zone. Four overlays are defined for PoP1, and two overlays are defined for PoP2:
config system sdwan
config members
edit 1
set zone "PoP1"
next
edit 2
set zone "PoP1"
next
edit 3
set zone "PoP1"
next
edit 4
set zone "PoP1"
next
edit 5
set zone "PoP2"
next
edit 6
set zone "PoP2"
next
end
end
3. Configure a performance SLA health check to a probe server behind the three hubs:
config system sdwan
config health-check
edit "Hubs"
set server "172.31.100.100"
set members 0
config sla

Fortinet Inc.
SD-WAN
edit 1
next
end
next
end
end
4. Configure the service rule with the following settings: use SLA mode, enable zone mode to steer traffic based on the
zone statuses, enable sla-stickiness, and use a 30-second hold down so that upon a recovery, existing
sessions will remain on the secondary PoP while new sessions will switch back to the primary PoP once the 30-
second duration ends:
config system sdwan
config service
edit 1
set mode sla
set zone-mode enable
set dst "all"
set src "CORP_LAN"
set hold-down-time 30
set sla-stickiness enable
config sla
edit "Hubs"
set id 1
next
end
set priority-zone "PoP1" "PoP2"
next
end
end
Since the PoP1 zone is specified before PoP2, PoP1 is regarded as the primary and preferred over the PoP2 zone.
5. Configure the in_sla and out_sla route maps that define the communities that are advertised to the hub when the
zones are in and out of SLA.
a. Configure the access list:
edit "net10"
config rule
edit 1
set prefix 10.0.3.0 255.255.255.0
next
end
next
end
b. Configure the route maps:

edit "in_sla"
config rule
edit 1
next

Fortinet Inc.
SD-WAN
end
next
edit "out_sla"
config rule
edit 1
next
end
next
end
6. Configure the default route map for out of SLA scenarios, preferable route map for in SLA scenarios, and the local
network to be advertised:
config router bgp
config neighbor
edit "172.31.0.1"
...
...
next
edit "172.31.0.2"
...
...
next
edit "172.31.0.129"
...
...
next
end
config network
edit 1
set prefix 10.0.3.0 255.255.255.0
next
end
...
end
7. Define SD-WAN neighbors for each hub. The minimum-sla-meet-members is configured for the Hub-1 neighbor
so that bi-directional traffic goes through Hub-1 as long as the in SLA overlays to Hub-1 are no less than 1.
Associate the previously defined service rule to each SD-WAN neighbor:
config system sdwan
config neighbor
edit "172.31.0.1"
set member 1 2
set service-id 1
next
edit "172.31.0.2"
set member 3 4

Fortinet Inc.
SD-WAN
set service-id 1
next
edit "172.31.0.129"
set member 5 6
set service-id 1
next
end
end
To configure the hubs:
1. Configure the SD-WAN zone, members, and health check for the external connections to peer routers.
Performance SLA health checks are sent to external servers in order to measure the health of the external
connections:
config system sdwan
set status enable
config zone
next
end
config members
edit 1
next
edit 2
next
end
config health-check
edit "external_peers"
set members 1 2
config sla
edit 1
next
end
next
end
end
2. Configure the route maps for in and out of SLA scenarios. When out of SLA (one of the external connections is
down), external routes are denied to be advertised to the spokes that are part of the neighbor group.
a. Configure the access list:
edit "net_Lo"
config rule
edit 1
set prefix 172.31.200.200 255.255.255.255
next
end
next
end

Fortinet Inc.
SD-WAN
b. Configure the route maps:

edit "in_sla"
config rule
edit 1
set match-ip-address "net_Lo"
next
end
next
edit "out_sla"
config rule
edit 1
set action deny routes
set match-ip-address "net_Lo"
next
end
next
end
3. In the BGP settings, configure the external network prefix to advertise. Then configure the neighbor group and
neighbor range for the spokes. Configure the preferable and default route maps to define the behavior when the
external connections are in and out of SLA:
config router bgp
...
config network
edit 1
set prefix 172.31.200.200 255.255.255.255
next
end
edit "EDGE"
...
...
next
end
edit 1
set prefix 172.31.0.64 255.255.255.192
set neighbor-group "EDGE"
next
end
...
end
4. Configure the SD-WAN neighbor to match the neighbor group that includes spokes as members. Specify that at
least one of the external peer connections needs to be up to be considered in SLA:
config system sdwan
config neighbor
edit "EDGE"
set member 1 2
set health-check "external_peers"

Fortinet Inc.
SD-WAN
set sla-id 1
next
end
end
Testing and verification
The following tests use diagnostic commands on various FortiGates to verify the connections in the SD-WAN
configuration.
Test case 1: the primary PoP and Hub-1 are in SLA
1. Verify the SD-WAN service rules status on Spoke-1. When all six overlays are in SLA on Spoke-1, the primary PoP
and primary zone PoP1 are preferred. In particular, the overlay H1_T11 over PoP1 is preferred:
Spoke-1 (root) # diagnose sys sdwan service4
Service(1): Address Mode(IPV4) flags=0x1c200 use-shortcut-sla use-shortcut sla-
stickiness
Tie break: cfg
Gen(1), TOS(0x0/0x0), Protocol(0): src(1->65535):dst(1->65535), Mode(sla), sla-
compare-order
Members(6):
1: Seq_num(1 H1_T11 PoP1), alive, sla(0x1), gid(0), cfg_order(0), local cost(0),
selected
selected
selected
selected
selected
selected
Src address(1):
10.0.0.0-10.255.255.255
Dst address(1):
0.0.0.0-255.255.255.255
2. Verify the BGP learned routes on Hub-1. The local route with in SLA community 10:1 is advertised to all hubs.
Though, the AS paths on Hub-1 and Hub-2 are shorter than Hub-3:
PoP1-Hub1 (root) # get router info bgp network 10.0.3.0/24
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:1
Last update: Mon Jul 17 15:16:57 2023

Fortinet Inc.
SD-WAN
3. Send traffic from a host behind Spoke-1 to 172.31.200.200.

4. Run a sniffer trace on Spoke-1. Traffic leaves and returns on the H1_T11 overlay :
Spoke-1 (root) # diagnose sniffer packet any 'host 172.31.200.200' 4
interfaces=[any]
filters=[host 172.31.200.200]
5.098339 H1_T11 out 10.0.3.2 -> 172.31.200.200: icmp: echo request
5.098618 H1_T11 in 172.31.200.200 -> 10.0.3.2: icmp: echo reply
Test case 2: a single SD-WAN member on Hub-1 is out of SLA
Hub-1 and PoP1 are still preferred in this scenario.
1. Verify the health check status on Spoke-1. The H1_T11 overlay on Hub-1/PoP1 is out of SLA:
Spoke-1 (root) # diagnose sys sdwan health-check
Health Check(Hubs):
…
2. Verify the SD-WAN neighbor status. The SD-WAN neighbor still displays Hub-1’s zone status as pass/alive:
Spoke-1 (root) # diagnose sys sdwan neighbor
Selected role(standalone) last_secondary_select_time/current_time in seconds
0/436439
Neighbor(172.31.0.1): member(1 2)role(standalone)
Health-check(:0) sla-pass selected alive
3. Verify the SD-WAN service rules status. Spoke-1 steers traffic to the H1_T22 overlay through Hub-1:
stickiness
Tie break: cfg
compare-order
Members(6):
selected
selected

Fortinet Inc.
SD-WAN

selected
selected
selected
selected
Src address(1):
10.0.0.0-10.255.255.255
Dst address(1):
0.0.0.0-255.255.255.255
4. Verify the BGP learned routes on Hub-1. The hubs continue to receive community 10:1 from the spoke and continue
to route incoming traffic through Hub-1:
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:1

6. Run a sniffer trace on Spoke-1. Traffic leaves and returns on the H1_T22 overlay:
interfaces=[any]
filters=[host 172.31.200.200]
Test case 3: both SD-WAN members on Hub-1 are out of SLA
Other in SLA overlays in zone PoP1 though Hub-2 are still preferred over PoP2 in this scenario.
1. Verify the health check status on Spoke-1. Both H1_T11 and H1_T22 overlays on Hub-1/PoP1 are out of SLA:
Health Check(Hubs):

Fortinet Inc.
SD-WAN

2. Verify the SD-WAN neighbor status. The SD-WAN neighbor displays Hub-1’s zone status as failed. However, SD-
WAN Hub-2 is pass/alive:
0/436535
Health-check(:0) sla-fail alive
3. Verify the SD-WAN service rules status. Spoke-1 steers traffic to the H2_T11 overlay through Hub-2:
stickiness
Tie break: cfg
compare-order
Members(6):
selected
selected
selected
selected
selected
selected
Src address(1):
10.0.0.0-10.255.255.255
Dst address(1):
0.0.0.0-255.255.255.255
4. Verify the BGP learned routes on Hub-1 and Hub-2. Hub-2 and Hub-3 continue to receive community 10:1 from
Spoke-1, but Hub-1 receives the out of SLA community of 10:2.
a. On Hub-1:
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)

Fortinet Inc.
SD-WAN

Community: 10:2
b. On Hub-2:
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:1

interfaces=[any]
filters=[host 172.31.200.200]

Test case 4: three SD-WAN members on PoP1 are out of SLA
The number of in SLA overlays in zone PoP1 is less than the minimum-sla-meet-members in zone PoP1. The SD-
WAN service rule for Hub-2 is forcibly marked as sla(0x0) or out of SLA.
1. Verify the health check status on Spoke-1. All three H1_T11, H1_T22, and H2_T11 overlays on PoP1 are out of
SLA:
Health Check(Hubs):
2. Verify the SD-WAN neighbor status. The SD-WAN neighbor displays Hub-1 and Hub-2’s zone status as failed:

Fortinet Inc.
SD-WAN

0/436605
3. Verify the SD-WAN service rules status. Since the minimum SLA members is not met for the primary zone (PoP1),
the remaining overlay in PoP1 associated with the SD-WAN service rule is forcibly set to out of SLA. Spoke-1 steers
traffic to the H3_T11 overlay through Hub-3:
stickiness
Tie break: cfg
compare-order
Members(6):
selected
selected
selected
selected
selected
selected
Src address(1):
10.0.0.0-10.255.255.255
Dst address(1):
0.0.0.0-255.255.255.255
4. Verify the BGP learned routes on each hub. Hub-3 continues to receive community 10:1 from Spoke-1, but Hub-1
and Hub-2 receive the out of SLA community of 10:2.
a. On Hub-1:
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:2
b. On Hub-2:

Fortinet Inc.
SD-WAN

Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:2
c. On Hub-3:
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:1

interfaces=[any]
filters=[host 172.31.200.200]
Test case 5: an SD-WAN member on PoP1 recovers
SD-WAN member H2_T11 recovers and brings the number of overlays in SLA back to being above the minimum-sla-
meet-members threshold in PoP1. After the hold down time duration (30 seconds), in SLA overlays in zone PoP1 are
preferred over PoP2 again. With sla-stickiness enabled, existing traffic is kept on H3_T11, but new traffic is steered
to H2_T11.
1. Verify the SD-WAN service rules status on Spoke-1. The hold down timer has not yet passed, so H2_T11 is not yet
preferred—even though the SLA status is pass/alive:

stickiness
Tie break: cfg
compare-order

Fortinet Inc.
SD-WAN
Members(6):
selected
selected
selected
selected
selected
selected
2. Verify the SD-WAN service rules status again after the hold down timer passes. H2_T11 and H2_T22 from PoP1
are now preferred:
stickiness
Tie break: cfg
compare-order
Members(6):
selected
selected
selected
selected
selected
selected
3. Verify the BGP learned routes on Hub-2, which now receives community 10:1 from Spoke-1:
Original VRF 0
172.31.0.65 from 172.31.0.65 (172.31.0.65)
Community: 10:1
Last update: Tue Jul 18 14:41:32 2023

5. Run a sniffer trace on Spoke-1. Because of sla-stickiness, the existing traffic is kept on H3_T11:
interfaces=[any]

Fortinet Inc.
SD-WAN
filters=[host 172.31.200.200]

Test case 6: Hub-1 has an in SLA path to external peers
Since Hub-1 has an in SLA path to external peers, it will advertise the external route with destination 172.31.200.200/32
to Spoke-1.
1. Verify the health check status on Hub-1. Note that port4 meets SLA, but port5 does not:
PoP1-Hub1 (root) # diagnose sys sdwan health-check
Health Check(external_peers):
Seq(1 port4): state(alive), packet-loss(0.000%) latency(0.161), jitter(0.009), mos
2. Verify the SD-WAN neighbor status. The minimum-sla-meet-members threshold of 1 is still met:
PoP1-Hub1 (root) # diagnose sys sdwan neighbor
Neighbor(EDGE): member(1 2)role(standalone)
Health-check(external_peers:1) sla-pass selected alive
3. Verify the BGP learned routes. Hub-1 still advertises the external route to the Spoke-1 BGP neighbor:
PoP1-Hub1 (root) # get router info bgp neighbors 172.31.0.65 advertised-routes
VRF 0 BGP table version is 13, local router ID is 172.31.0.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight RouteTag Path
*>i172.31.200.200/32 172.31.0.1 100 32768 0 i <-/->
Total number of prefixes 1
Test case 7: all external peers on Hub-1 are out of SLA
In this case, Hub-1 will now advertise the default route map, which denies the advertisement of the external route.
Spoke-1 will now route traffic to the next hub.
1. Verify the health check status on Hub-1. Note that port4 and port5 do not meet SLA:
PoP1-Hub1 (root) # diagnose sys sdwan health-check
Health Check(external_peers):
2. Verify the SD-WAN neighbor status. The minimum-sla-meet-members threshold of 1 is not met:
PoP1-Hub1 (root) # diagnose sys sdwan neighbor
Neighbor(EDGE): member(1 2)role(standalone)
Health-check(external_peers:1) sla-fail dead

Fortinet Inc.
SD-WAN
3. Verify the BGP learned routes. Hub-1 does not advertise any external routes to the Spoke-1 BGP neighbor:
PoP1-Hub1 (root) # get router info bgp neighbors 172.31.0.65 advertised-routes
% No prefix for neighbor 172.31.0.65
Using a single IKE elector in ADVPN to match all SD-WAN control plane traffic
In the SD-WAN with ADVPN use case, two spokes can communicate with each other on the control plane by an ADVPN
shortcut. In order to separate the control traffic from data traffic, the IKE creates a dynamic selector for health check
packets sent between the spokes. BGP traffic is also matched by this dynamic IKE selector. Therefore, when spokes
establish BGP peering with other spokes, the BGP traffic does not count towards the data traffic and will not impact
IPsec idle timeout and shortcut tunnel tear down.
Example
In this example, SD-WAN with ADVPN is configured. The IPsec ADVPN shortcut tunnel is required to tear down when it
is idle. SD-WAN health checks are configured, and BGP neighbors established between the spokes is required.
To configure the Hub FortiGate:
1. Configure the phase 1 interface:

edit "Hub"
set type dynamic
set ike-version 2
set peertype any
set proposal aes128-sha256 aes256-sha256 aes128gcm-prfsha256 aes256gcm-prfsha384
chacha20poly1305-prfsha256
set dpd on-idle
set psksecret ************

Fortinet Inc.
SD-WAN
next
end

edit "Hub"
set phase1name "Hub"
set proposal aes128-sha1 aes256-sha1 aes128-sha256 aes256-sha256 aes128gcm
aes256gcm chacha20poly1305
next
end
3. Configure the VPN interface:

edit "Hub"
set vdom "root"
set ip 10.10.1.250 255.255.255.255
set type tunnel
set remote-ip 10.10.1.254 255.255.255.0
set snmp-index 50
next
end
4. Configure the BGP settings:

config router bgp
set as 65412
config neighbor
edit "10.10.1.1"
set remote-as 65412
next
edit "10.10.1.2"
set remote-as 65412
next
end
config network
edit 1
set prefix 174.16.101.0 255.255.255.0
next
end
end
To configure the Spoke1 FortiGate:

edit "Spoke1"
set ike-version 2

Fortinet Inc.
SD-WAN
set peertype any

set npu-offload disable
set idle-timeout enable
set idle-timeoutinterval 5
next
end

edit "Spoke1"
set phase1name "Spoke1"
next
end

edit "Spoke1"
set vdom "root"
set ip 10.10.1.1 255.255.255.255
set type tunnel
set remote-ip 10.10.1.254 255.255.255.0
set snmp-index 28
next
end

config router bgp
set as 65412
config neighbor
edit "10.10.1.250"
set remote-as 65412
next
edit "10.10.1.2"
set remote-as 65412
next
end
config network
edit 1
set prefix 10.1.100.0 255.255.255.0
next
end
end

Fortinet Inc.
SD-WAN
config system sdwan

set status enable
config zone
next
end
config members
edit 1
set interface "Spoke1"
next
end
config health-check
edit "1"
set server "174.16.101.44"
set members 0
next
end
end
To configure the Spoke2 FortiGate:

edit "Spoke2"
set ike-version 2
set peertype any
next
end

edit "Spoke2"
set phase1name "Spoke2"
next
end

edit "Spoke2"
set vdom "root"
set ip 10.10.1.2 255.255.255.255

Fortinet Inc.
SD-WAN
set type tunnel

set remote-ip 10.10.1.254 255.255.255.0
set snmp-index 15
next
end

config router bgp
set as 65412
config neighbor
edit "10.10.1.250"
set remote-as 65412
next
edit "10.10.1.1"
set remote-as 65412
next
end
config network
edit 1
set prefix 192.168.4.0 255.255.255.0
next
end
end

config system sdwan
set status enable
config zone
next
end
config members
edit 1
set interface "Spoke2"
next
end
config health-check
edit "1"
set server "174.16.101.44"
set members 0
next
end
end
1. Send traffic between the spokes to establish the ADVPN shortcut.

2. Verify the IPsec tunnel state on the Spoke1 FortiGate:
Spoke1 # diagnose vpn tunnel list
------------------------------------------------------
name=Spoke1_0 ver=2 serial=7 172.16.200.1:0->172.16.200.3:0 tun_id=10.10.1.2 tun_

Fortinet Inc.
SD-WAN
id6=::10.0.0.3 dst_mtu=1500 dpd-link=on weight=1

bound_if=19 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/66224 options
[102b0]=create_dev rgwy-chg frag-rfc role=primary accept_traffic=1 overlay_id=0
parent=Spoke1 index=0
fec: egress=0 ingress=0
proxyid=Spoke1 proto=0 sa=1 ref=5 serial=2 adr health-check
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:10.10.1.2-10.10.1.2:0
SA: ref=3 options=92626 type=00 soft=0 mtu=1438 expire=43055/0B replaywin=2048
seqno=214 esn=0 replaywin_lastseq=00000213 qat=0 rekey=0 hash_search_len=1
dec: spi=17a473be esp=aes key=16 40dfada9532cefe5563de71ac5908aa1
ah=sha1 key=20 36e967d9b6fce8807132c3923d0edfae6cb6c115
enc: spi=75cde30a esp=aes key=16 9bf08196d6830455a75bc676e04c816f
ah=sha1 key=20 638db13dc4db0a6e5f523047805d18413eea4d4d
npu_flag=00 npu_rgwy=172.16.200.3 npu_lgwy=172.16.200.1 npu_selid=c dec_npuid=0 enc_
npuid=0
proxyid=Spoke1 proto=0 sa=1 ref=2 serial=1 adr
src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
seqno=2 esn=0 replaywin_lastseq=00000000 qat=0 rekey=0 hash_search_len=1
dec: spi=17a473bd esp=aes key=16 c78e5085857d0c5842e394fc44b38822
ah=sha1 key=20 0bb885a85f77aa491a1209e4d36b7cddd7caf152
enc: spi=75cde309 esp=aes key=16 6717935721e4a25428d6a7a633da75a9
ah=sha1 key=20 eaf092280cf5b9f9db09ac95258786ffbfacead0
npu_flag=00 npu_rgwy=172.16.200.3 npu_lgwy=172.16.200.1 npu_selid=b dec_npuid=0 enc_
npuid=0
------------------------------------------------------
name=Spoke1 ver=2 serial=1 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 tun_
bound_if=19 lgwy=static/1 tun=intf mode=auto/1 encap=none/560 options[0230]=create_dev
frag-rfc role=primary accept_traffic=1 overlay_id=0

src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
seqno=22a esn=0 replaywin_lastseq=0000021f qat=0 rekey=0 hash_search_len=1
dec: spi=17a473bc esp=aes key=16 eff2dc03b48968bb55b9e3950ebde431
ah=sha1 key=20 5db42a32aec15bc8a5fe392c256d1ae8ab3b4ef8

Fortinet Inc.
SD-WAN
enc: spi=bdc3bd80 esp=aes key=16 d0ec06b61ad572cc8813b599edde8c68

ah=sha1 key=20 0306850f0184d957e9475da33d7971653a95c233
npu_flag=00 npu_rgwy=172.16.200.4 npu_lgwy=172.16.200.1 npu_selid=0 dec_npuid=0 enc_
npuid=0
The dynamic selector is created (highlighted) for SD-WAN control traffic, SD-WAN health checks, and BGP
between spokes traffic.
3. Verify the BGP neighbors and check the routing table:
Spoke1 # get router info bgp summary


10.10.1.2 4 65412 52 76 7 0 0 00:06:27 1
10.10.1.250 4 65412 70 69 1 0 0 00:58:44 2
4. Stop sending traffic between the spokes, and wait for a few minutes (idle timeout).
5. Verify the IPsec tunnel state on the Spoke1 FortiGate:
Spoke1 # diagnose vpn tunnel list
------------------------------------------------------
name=Spoke1 ver=2 serial=1 172.16.200.1:0->172.16.200.4:0 tun_id=172.16.200.4 tun_
bound_if=19 lgwy=static/1 tun=intf mode=auto/1 encap=none/560 options[0230]=create_dev
frag-rfc role=primary accept_traffic=1 overlay_id=0

src: 0:0.0.0.0-255.255.255.255:0
dst: 0:0.0.0.0-255.255.255.255:0
seqno=5be esn=0 replaywin_lastseq=000005bc qat=0 rekey=0 hash_search_len=1
dec: spi=76fdf7d1 esp=aes key=16 b26fd2dae76665f580d255b67f79df1e
ah=sha1 key=20 14b0acc3c8c92a0af8ab43ff0437d2141b6d3f65
enc: spi=bdc3bd85 esp=aes key=16 3eae3ad42aa32d7cdd972dfca286acd1
ah=sha1 key=20 3655f67ee135f38e3f0790f1c7e3bd19c4a9285c
npu_flag=00 npu_rgwy=172.16.200.4 npu_lgwy=172.16.200.1 npu_selid=0 dec_npuid=0 enc_
npuid=0
The shortcut tunnel between the spokes has been torn down. When data traffic is idle, the BGP traffic does not get
sent on the data traffic selector, so the tunnel is not kept alive. This behavior is the expected, which consequently
allows the shortcut tunnel to be torn down when idle.
6. Verify the IKE debugs messages to confirm the ADVPN shortcut was torn down:

Fortinet Inc.
SD-WAN
Spoke1 # diagnose debug enable

Spoke1 # diagnose debug application ike -1
...
ike 0:Spoke1_0: connection idle time-out
ike 0:Spoke1_0: deleting
ike 0:Spoke1_0: flushing
ike 0:Spoke1_0: deleting IPsec SA with SPI 75cde338
ike 0:Spoke1_0:Spoke1: deleted IPsec SA with SPI 75cde338, SA count: 0
ike 0:Spoke1_0: sending SNMP tunnel DOWN trap for Spoke1
ike 0:Spoke1_0: tunnel down event 0.0.0.0
ike 0:Spoke1_0:Spoke1: delete
ike 0:Spoke1_0: deleting IPsec SA with SPI 75cde337
ike 0:Spoke1_0:Spoke1: deleted IPsec SA with SPI 75cde337, SA count: 0
ike 0:Spoke1_0: sending SNMP tunnel DOWN trap for Spoke1
ike 0:Spoke1_0: tunnel down event 0.0.0.0
ike 0:Spoke1_0:Spoke1: delete
ike 0:Spoke1_0: flushed
ike 0:Spoke1_0:23:86: send informational
ike 0:Spoke1_0:23: sent IKE msg (INFORMATIONAL): 172.16.200.1:500->172.16.200.3:500,
len=80, vrf=0, id=0304e1284a432105/fa7d3fd75e7f481e:00000004
ike 0:Spoke1_0: delete connected route 10.10.1.1 -> 10.10.1.2
ike 0:Spoke1_0: delete dynamic
ike 0:Spoke1_0: deleted
ike 0:Spoke1: schedule auto-negotiate
ike 0: comes 172.16.200.3:500->172.16.200.1:500,ifindex=19,vrf=0....
ike 0: IKEv2 exchange=INFORMATIONAL_RESPONSE
id=0304e1284a432105/fa7d3fd75e7f481e:00000004 len=80
SD-WAN Overlay-as-a-Service
SD-WAN Overlay-as-a-Service (OaaS) is supported through a license displayed as SD-WAN Overlay as a Service on
the System > FortiGuard page. Each FortiGate used by the FortiCloud Overlay-as-a-Service portal must have this
license applied to it.
To view the status of the OaaS license in the GUI:
1. Go to System > FortiGuard.

2. Expand License Information. The SD-WAN Overlay as a Service license status is listed as:
l Licensed: OaaS is currently licensed and will expire on the provided date.

Fortinet Inc.
SD-WAN
l Expires Soon: OaaS is currently licensed but will expire soon on the provided date.
l Expired: The OaaS license has already expired on the provided date.
l Not Licensed: OaaS has not been licensed.

Fortinet Inc.
SD-WAN
To view the status of the OaaS license in the CLI:
1. Verify that the entitlement can be updated:
The SD-WAN Overlay-as-a-Service license is listed as SWOS in the CLI.
# diagnose test update info
System contracts:
FMWR,Wed Dec 20 16:00:00 2023
SPAM,Wed Dec 20 16:00:00 2023
SBCL,Wed Dec 20 16:00:00 2023
SWNO,Wed Dec 20 16:00:00 2023
SWNM,Wed Sep 27 17:00:00 2023
SWOS,Mon Aug 14 17:00:00 2023
SPRT,Wed Dec 20 16:00:00 2023
SDWN,Sun Dec 10 16:00:00 2023
SBCL,Wed Dec 20 16:00:00 2023
SBEN,Wed Dec 20 16:00:00 2023
2. Verify that the expiration date log can be generated:

# execute log display
1: date=2023-08-10 time=00:00:01 eventtime=1691650800645347120 tz="-0700"

logid="0100020138" type="event" subtype="system" level="warning" vd="root"
logdesc="FortiGuard SD-WAN Overlay as a Service license expiring" msg="FortiGuard SD-WAN
Overlay Service license will expire in 4 day(s)"
To ensure FortiGate spoke traffic remains uninterrupted when configuration is orchestrated from the SD-WAN Overlay-
as-a-Service (OaaS), support for an OaaS agent on the FortiGate is available. The OaaS agent communicates with the
OaaS controller in FortiCloud, validates and compares the FortiOS configuration, and applies the FortiOS configuration
to the FortiGate as a transaction when it has been orchestrated from the OaaS portal. Secure communication between
the OaaS agent and the OaaS controller is achieved using the FGFM management tunnel.
If any configuration change fails to be applied, then the OaaS agent rolls back all configuration changes that were
orchestrated. The OaaS status can be acquired using get oaas status.
To determine the status of OaaS:
# get oaas status

Account ID: 78992
Account: admin@domain.com
Site: site1
Configuration version: 4
Configuration sync status: SUCCESS
Target version: 4
Task ID: xxxxxxxxx
Error:

Fortinet Inc.
SD-WAN
Advanced configuration
The following topics provide instructions on SD-WAN advanced configuration:

l SD-WAN with FGCP HA on page 1079
l Configuring SD-WAN in an HA cluster using internal hardware switches on page 1086
l SD-WAN configuration portability on page 1089
l SD-WAN segmentation over a single overlay on page 1095
l SD-WAN segmentation over a single overlay using IPv6 on page 1110
l Matching BGP extended community route targets in route maps on page 1117
l Copying the DSCP value from the session original direction to its reply direction on page 1122
See also Packet distribution for aggregate static IPsec tunnels in SD-WAN on page 2212.
SD-WAN with FGCP HA
This example shows how to convert a standalone FortiGate SD-WAN solution to a FGCP HA cluster with full-mesh WAN
set up. This configuration allows you to load balance your internet traffic between multiple ISP links. It also provides
redundancy for your internet connection if your primary ISP in unavailable, or if one of the FortiGates in the HA cluster
fails.
This example assumes that a standalone FortiGate has already been configured for SD-WAN by following the SD-WAN
quick start on page 789.
Standalone FortiGate:

Fortinet Inc.
SD-WAN
FGCP HA cluster:
The following devices are required to convert the topology to HA:

l A second FortiGate that is the same model running the same firmware version.
l Two switches for connecting each FortiGate's WAN interface to the corresponding ISP modem.
Before you begin:
l Ensure that the licenses and subscriptions on both HA members match.
l Ensure that there are one or more ports reserved for HA heartbeat.
l Ensure you have physical access to both HA members.
Enabling HA and re-cabling the WAN interfaces will cause network interruptions.
This procedure should be performed during a maintenance window.
Configuring the standalone FortiGate for HA
After running the following commands, the FortiGate negotiates to establish an HA cluster. You might temporarily lose
connectivity with the FortiGate as FGCP negotiations take place and the MAC addresses of the FortiGate interfaces are
changed to HA virtual MAC addresses.
This configurations sets the HA mode to active-passive.
The ha1 and ha2 interfaces are configured as the heartbeat interfaces, with priorities set to 200 and 100 respectively.
Setting different priorities for the heartbeat interfaces is a best practice, but is not required.
If you have more than one cluster on the same network, each cluster should have a different group ID. Changing the
group ID changes the cluster interface's virtual MAC addresses. If the group IP causes a MAC address conflict on your
network, select a different group ID.
Enabling override and increasing the device priority means that this FortiGate always becomes the primary unit.

Fortinet Inc.
SD-WAN
To configure the standalone FortiGate for HA in the GUI:
1. Go to System > Settings and change the Host name so that the FortiGate can be easily identified as the primary
unit.
2. Go to System > HA and configure the following options:
Mode Active-Passive
Device priority 250
Group name My-cluster
Password <password>
Heartbeat interfaces ha1 and ha2
Heartbeat Interface Priority port2 (ha1): 200

port3 (ha2): 100
Override and the group ID can only be configured from the CLI.
3. Click OK.
Connectivity with the FortiGate will temporarily be lost.
To configure the standalone FortiGate for HA in the CLI:
1. Change the host name so that the FortiGate can be easily identified:
set hostname primary_FG
end
2. Configure HA:
config system ha
set mode a-p
set group-id 100

Fortinet Inc.
SD-WAN
set group-name My-cluster

set password <password>
set priority 250
set override enable
set hbdev ha1 200 ha2 100
end
If HA mode does not start after running the above steps, ensure that none of the FortiGate's
interfaces use DHCP or PPPoE addressing.
Configuring the secondary FortiGate for HA
The secondary FortiGate must be the same model and running the same firmware version as the primary FortiGate. The
HA settings are the same as the for the primary unit, except the secondary device has a lower priority and override is not
enabled.
It is best practice to reset the FortiGate to factory default settings prior to configuring HA. This
reduces the chance of synchronization problems.
# execute factoryreset
This operation will reset the system to factory default!
Do you want to continue? (y/n) y
This is unnecessary if the device is new from the factory.
To configure the secondary FortiGate for HA in the GUI:
1. Go to System > Settings and change the Host name so that the FortiGate can be easily identified as the backup unit.
2. Go to System > HA and configure the options the same as for the primary FortiGate, except with a lower priority:
Mode Active-Passive
Device priority 128
Group name My-cluster
Password <password>
Heartbeat interfaces ha1 and ha2
Heartbeat Interface Priority port2 (ha1): 200

port3 (ha2): 100
3. Click OK.
To configure the secondary FortiGate for HA in the CLI:
1. Change the host name so that the secondary FortiGate can be easily identified:
set hostname secondary_FG
end

Fortinet Inc.
SD-WAN
2. Configure HA:
config system ha
set mode a-p
set group-id 100
set group-name My-cluster
set password <password>
set priority 128
set hbdev ha1 200 ha2 100
end
Connecting the heartbeat interfaces between the FortiGates
To connect and check the heartbeat interfaces:
1. Connect the heartbeat interfaces ha1 and ha2 between the primary and secondary FortiGate.
a. An HA primary device is selected. Because the primary FortiGate has a higher priority and override enabled, it
assumes the role of HA primary.
b. The secondary FortiGate synchronizes its configuration from the primary device.
2. Verify that the checksums match between the primary and secondary FortiGates:
# diagnose sys ha checksum cluster
================== FG5H0XXXXXXXXXX0 ==================
is_manage_primary()=1, is_root_primary()=1
debugzone
global: 2b e9 81 38 c2 9d 4f db b7 0e 1f 49 42 c6 1e fb
root: af a6 48 c5 c2 9a 8b 81 a5 53 fb 27 e9 ae 01 6a
all: 89 1f 63 77 48 8a 30 ee 57 06 ca eb 71 e6 8e ad
checksum
================== FG5H0XXXXXXXXXX1 ==================
is_manage_primary()=0, is_root_primary()=0
debugzone
checksum
If all of the cluster members have identical checksums, then their configurations are synchronized. If the checksums
are not the same, wait for a few minutes, then repeat the command. Some parts of the configuration might take a
significant amount of time to synchronize (tens of minutes).

Fortinet Inc.
SD-WAN
Connecting other traffic interfaces
After the device configurations are synchronized, you can connect the rest of the traffic interfaces. Making these
connections will disrupt traffic as cables are disconnected and reconnected.
Switches must be used between the cluster and the ISPs, and between the cluster and the internal network, as shown in
the topology diagram.
Checking cluster operations
The HA Status dashboard widget shows the synchronization status. Hover over the host names of each FortiGate in the
widget to verify that they are synchronized and have the same checksum.
To view more information about the cluster status, including the number of sessions passing through the cluster
members, go to System > HA.
See Check HA synchronization status on page 2905 for more information.
Results
1. Browse the internet on a computer in the internal network.

2. Go to Network > SD-WAN and select the SD-WAN Zones tab to see the bandwidth, volume, and sessions for traffic
on the SD-WAN interfaces. See Results on page 794 for details.
3. Go to Dashboard > Network, and expand the SD-WAN widget to see information about each interface, such as the
number of sessions and the bit rate.
Testing HA failover
All traffic should currently be flowing through the primary FortiGate. If it becomes unavailable, traffic fails over to the
secondary FortiGate. When the primary FortiGate rejoins the cluster, the secondary FortiGate continues to operate as
the primary FortiGate.
To test this, ping a reliable IP address from a computer in the internal network, and then power off the primary FortiGate.
There will be a momentary pause in the ping results until traffic diverts to the backup FortiGate, allowing the ping traffic to
continue:
64 bytes from 184.25.76.114: icmp_seq=69 ttl=52 time=8.719 ms\
Request timeout for icmp_seq 75\
64 bytes from 184.25.76.114: icmp_seq=83 ttl=52 time=8.639 ms}

Fortinet Inc.
SD-WAN
If you are using port monitoring, you can also unplug the primary FortiGate's internet facing
interface to test failover.
After the secondary FortiGate becomes the primary, you can log into the cluster using the same IP address as before the
fail over. If the primary FortiGate is powered off, you will be logged into the backup FortiGate. Check the host name to
verify what device you have logged into. The FortiGate continues to operate in HA mode, and if you restart the primary
FortiGate, it will rejoin the cluster and act as the backup FortiGate. Traffic is not disrupted when the restarted FortiGate
rejoins the cluster.
You can also use the CLI to force an HA failover. See Force HA failover for testing and demonstrations on page 2933 for
information.
Testing ISP failover
To test a failover of the redundant internet configuration, you need to simulate a failed internet connection to one of the
ports. You can do this by disconnecting power from the wan1 switch, or by disconnecting the wan1 interfaces of both
FortiGates from ISP1.
After disconnecting, verify that users still have internet access
l Go to Dashboard > Network, and expand the SD-WAN widget. The Upload and Download columns for wan1 show
that traffic is not going through that interface.
l Go to Network > SD-WAN and select the SD-WAN Zones tab. The Bandwidth, Volume, and Sessions tabs show
that traffic is entirely diverted to wan2.
Users on the network should not notice the wan1 failure. If you are using the wan1 gateway IP address to connect to the
administrator dashboard, it will appear as though you are still connecting through wan1.
After verifying a successful failover, reestablish the connection to ISP1.

Fortinet Inc.
SD-WAN
Configuring SD-WAN in an HA cluster using internal hardware switches
In this SD-WAN configuration, two FortiGates in an active-passive (A-P) HA pair are used to provide hardware
redundancy. Instead of using external switches to provide a mesh network connection to the ISP routers, the FortiGates
use their built-in hardware switches to connect to the ISP routers.
Only FortiGate models that have hardware switches can be used for this solution. Ports in a
software switch are not in a forwarding state when a FortiGate is acting as a secondary device
in a A-P cluster.
In this topology:
l Two hardware switches are created, HD_SW1 and HD_SW2.
l HD_SW1 is used to connect to ISP 1 Router and includes the internal1 and internal2 ports.
l HD_SW2 is used to connect to ISP 2 Router and includes the internal3 and internal4 ports.
l Another interface on each device is used as the HA heartbeat interface, connecting the two FortiGates in HA.
The FortiGates create two hardware switches to connect to ISP 1 and ISP2. When FGT_A is the primary device, it
reaches ISP 1 on internal1 in HD_SW1 and ISP 2 on internal4 in HD_SW2. When FGT_B is the primary device, it
reaches ISP 1 on internal2 in HD_SW1 and ISP 2 on internal3 on HD_SW2.
HA failover
This is not a standard HA configuration with external switches. In the case of a device failure, one of the ISPs will no
longer be available because the switch that is connected to it will be down.
For example, If FGT_A loses power, HA failover will occur and FGT_B will become the primary unit. Its connection to
internal2 on HD_SW1 will also be down, so it will be unable to connect to ISP 1. Its SD-WAN SLAs will be broken, and
traffic will only be routed through ISP 2.
A link on a hardware switch cannot be monitored in HA monitor, so it is impossible to perform

link failure when a port in either of the hardware switches fails. Performing a link failure is
unnecessary in this configuration though, because any link failure on the hardware switch will
be experienced by both cluster members. SD-WAN SLA health checks should be used to
monitor the health of each ISP.

Fortinet Inc.
SD-WAN
Failure on a hardware switch or ISP router
If a hardware switch or switch interface is down, or the ISP router is down, the SD-WAN can detect the broken SLA and
continue routing to the other ISP.
For example, if FGT_A is the primary unit, and ISP 2 Router becomes unreachable, the SLA health checks on SD-WAN
will detect the broken SLA and cause traffic to stop routing to ISP 2.
Configuration
To configure the HA A-P cluster with internal hardware switches:
1. Configure two FortiGates with internal switches in an A-P HA cluster (follow the steps in HA active-passive cluster
setup on page 2887), starting by connecting the heartbeat interface.
2. When the HA cluster is up, connect to the primary FortiGate's GUI.
3. Remove the existing interface members from the default hardware switch:
a. Go to Network > Interfaces.
b. In the LAN section, double-click the internal interface to edit it.
c. In Interface Members, remove all of the interfaces
d. Click OK.
4. Configure the hardware switch interfaces for the two ISPs:
a. Go to Network > Interfaces and click Create New > Interface.
b. Enter a name (HD_SW1).
c. Set Type to Hardware Switch.
d. In Interface Members, add two interfaces (internal1 and internal2).
e. Set IP/Netmask to 192.168.1.2/24.
f. Configure the remaining settings as needed.

Fortinet Inc.
SD-WAN
g. Click OK.
h. Repeat these steps to create a second hardware switch interface (HD_SW2) with two interface members
(internal3 and internal4) and IP/Netmask set to 192.168.3.2/24.
To connect the devices as shown in the topology:
1. Connect the incoming interface to the internal switch on both FortiGates.

2. On FGT_A, connect internal1 of HD_SW1 to ISP 1 Router.
3. On FGT_B, connect internal3 of HD_SW2 to ISP 2 Router.
4. For HD_SW1, connect FGT_A internal2 directly to FGT_B internal2.
5. For HD_SW2, connect FGT_A internal4 directly to FGT_B internal4.
The primary FortiGate makes all the SD-WAN decisions.

Fortinet Inc.
SD-WAN
1. On the primary FortiGate, go to Network > SD-WAN, select the SD-WAN Zones tab, and click Create New > SD-
WAN Member.
2. In the Interface dropdown, select HD_SW1.
3. Leave SD-WAN Zone set to virtual-wan-link.
4. Enter the Gateway address 192.168.1.1.
5. Click OK.
6. Repeat these steps to add the second interface (HD_SW2) with the gateway 192.168.3.1.
7. Click Apply.
8. Create a health check:

a. Go to Network > SD-WAN, select the Performance SLA tab, and click Create New.
b. Set Name to GW_HC.
c. Set Protocol to Ping and Servers to 8.8.8.8.
d. Set Participants to All SD-WAN Members.
e. Enable SLA Target and leave the default values.
f. Click OK.
9. Create SD-WAN rules as needed. The SLA health check can be used to determine when the ISP connections are in
or out of SLA, and to failover accordingly.
SD-WAN configuration portability
When configuring SD-WAN, adding interfaces to members is optional.

This allows the SD-WAN to be configured without associating any interfaces to SD-WAN members. It also allows a
configuration to be copied directly from one device to another, without requiring the devices to have interfaces with the
same names.
After the configuration is created, add interfaces to the members make it functional.

Fortinet Inc.
SD-WAN
Example 1
In this example, we create a template with two SD-WAN members configured without assigned interfaces that are used
in a performance SLA and SD-WAN rule. The template can be used to configure new devices, as in Example 2 on page
1094. Interfaces are then assigned to the members, and the configuration becomes active.
To create the SD-WAN members in the GUI:
2. Leave all the settings set to their default values and click OK.
3. Repeat the above steps to create a second member.

The empty members are listed on the SD-WAN Zones tab.

Fortinet Inc.
SD-WAN
The members are disabled until interfaces are configured, but can still be used in performance SLAs and SD-WAN
rules.
To create a performance SLA in the GUI:
1. Go to Network > SD-WAN and select the Performance SLAs tab.

2. Click Create New.
3. Configure the performance SLA, specifying the empty members as participants.
4. Click OK.

Fortinet Inc.
SD-WAN
To create an SD-WAN rule in the GUI:
2. Configure the rule, adding both members to the Interface preference field:
3. Click OK.
To assign interfaces to the SD-WAN members in the GUI:
1. Go to Network > SD-WAN and select the SD-WAN Zones tab.

2. Edit the first member
3. Set Interface to an actual interface.
4. Click OK.
5. Repeat the above steps to assign an interface to the second member.

Fortinet Inc.
SD-WAN
To configure the SD-WAN in the CLI:
1. Create SD-WAN members:

config system sdwan
set status enable
config members
edit 1
next
edit 2
next
end
end
2. Create a health check (performance SLA):

config system sdwan
config health-check
edit "office"
set server "office365.com"
set protocol http
set sla-fail-log-period 300
set sla-pass-log-period 300
set members 2 1
config sla
edit 1
next
edit 2
next
end
next
end
end
3. Create a service (rule):

config system sdwan
config service
edit 3
set name "Office365"
set mode sla
set internet-service-app-ctrl 33182
config sla
edit "office"
set id 2
next
end
next
end
end
The SD-WAN configuration can now be used in as a template for new spokes, as in Example 2 on page 1094.

Fortinet Inc.
SD-WAN
Example 2
In this example, the configuration from Example 1 is copied onto a new FortiGate.
Using the CLI console and the GUI
To copy the SD-WAN configuration from the original FortiGate:
1. Optionally, change the console screen paging setting. See Screen paging on page 61 for details.
2. Open the CLI console.
3. If necessary, click Clear console to empty the console.
4. Enter the following command:
show system sdwan
5. Either click Download and open the file in a text editor, or click Copy to clipboard and paste the content into a text
editor.
6. Edit the CLI configuration as necessary. For example, the first line that shows the show command should be
deleted, and the default health checks can be removed.
7. If required, save the CLI configuration as a text file.
To paste the SD-WAN configuration onto a new FortiGate:
1. Copy the SD-WAN configuration from the text editor.

2. On the new FortiGate, open the CLI console.
3. Press Ctrl + v to paste the CLI commands.
4. In necessary, press Enter to apply the last end command.
The SD-WAN configuration is copied to the new FortiGate.
If the interfaces do not exist, the SD-WAN members are created without interfaces, and are disabled until interfaces
are configured.
To assign interfaces to the SD-WAN members:
1. Go to Network > SD-WAN and select the SD-WAN Zones tab.

2. Edit the first member
3. Set Interface to an actual interface.

Fortinet Inc.
SD-WAN
4. Click OK.
5. Repeat the above steps to assign an interface to the second member.
Using a terminal emulator
The following instructions use PuTTy. The steps may vary in other terminal emulators.
To copy the SD-WAN configuration from the original FortiGate:
1. Connect to the FortiGate. See Connecting to the CLI on page 53 for details.
2. Enter the following command:
show system sdwan
3. Select the output, press Ctrl + c to copy it, and then paste it into a text editor.
4. Edit the CLI configuration as necessary. For example, the default health checks can be removed.
5. If required, save the CLI configuration as a text file.
To paste the SD-WAN configuration onto a new FortiGate:
1. Connect to the new FortiGate. See Connecting to the CLI on page 53 for details.
2. Copy the SD-WAN configuration from the text editor.
3. Right-click to paste the SD-WAN configuration.
4. In necessary, press Enter to apply the last end command.
The SD-WAN configuration is copied to the new FortiGate.
If the interfaces do not exist, the SD-WAN members are created without interfaces, and are disabled until interfaces
are configured.
SD-WAN segmentation over a single overlay
SD-WAN, VPN, and BGP configurations support L3 VPN segmentation over a single overlay. In these configurations, a
hub and spoke SD-WAN deployment requires that branch sites, or spokes, are able to accommodate multiple
companies or departments, and each company's subnet is separated by a different VRF. A subnet on one VRF cannot
communicate with a subnet on another VRF between different branches, but can communicate with the same VRF.
SD-WAN options
VRF-aware SD-WAN health checks
SD-WAN on the originating spoke can tag the health check probes with the correct VRF when transmitting to a multi-
VRF tunnel. The hub can then forward the probes to the correct health check server in the same VRF as the hub. The IP
version (addr-mode) can be either IPv4 or IPv6.
config system sdwan
config health-check
edit <name>
set vrf <vrf id>
set source <address>
next
end
end

Fortinet Inc.
SD-WAN
vrf <vrf id> Virtual Routing Forwarding ID.

source <address> Source IP address used in the health-check packet to the server.
Overlay stickiness
When a hub has multiple overlays, traffic received on one overlay should egress on the same overlay when possible.
The service-sla-tie-break option ensures overlay stickiness. In SD-WAN service rules, options are available to
ensure that traffic received in a zone stays in that zone.
config system sdwan
config zone
edit <name>
set service-sla-tie-break input-device
next
end
config service
edit <id>
set input-zone <zone>
set tie-break input-device
next
end
end
service-sla-tie-break Members that meet the SLA are selected by matching the input device.
input-device
input-zone <zone> Source input-zone name.
tie-break input-device Members that meet the SLA are selected by matching the input device.
IPsec options
Configurable rate limit for shortcut offers sent by the hub
By default, the hub sends a shortcut offer to a spoke every five seconds. If the hub continues to send offers that keep
failing, and there are a large number of spokes, this can cause a high load on the hub. This setting makes the interval
between shortcut offers configurable.
edit <name>
set auto-discovery-offer-interval <interval>
next
end
auto-discovery-offer- Interval between shortcut offer messages, in seconds (1 - 300, default = 5).
interval <interval>
Segmentation over a single overlay
Segmentation requires that VRF info is encapsulated within the IPsec VPN tunnel. This setting enables multi-VRF
IPSEC tunnels. It is only applicable for forwarding traffic, and cannot be used for local-out traffic.

Fortinet Inc.
SD-WAN

edit <name>
set encapsulation vpn-id-ipip
next
end
encapsulation vpn-id-ipip VPN ID with IPIP encapsulation.
VPN configuration for BGP
The role of a VRF can be specified, along with other VRF details. Up to 252 VRFs can be configured per VDOM for any
device.
config router bgp
config vrf
edit <vrf>
set role {standalone | ce | pe}
set rd <string>
set export-rt <route_target>
set import-rt <route_target>
set import-route-map <route_map>
config leak-target
edit <vrf>
set route-map <route-map>
next
end
next
end
end
role {standalone | ce | VRF role: standalone, customer edge (CE), or provider edge (PE).
pe}
rd <string> Route Distinguisher: AA|AA:NN. This option is only available when the role is CE.
export-rt <route_target> List of export route target. This option is only available when the role is CE.
import-rt <route_target> List of import route target. This option is only available when the role is CE.
import-route-map <route_ Import route map. This option is only available when the role is CE.
map>
route-map <route-map> Route map of VRF leaking.
interface <interface> Interface that is used to leak routes to the target VRF.
In FortiOS 7.0, config vrf was config vrf-leak, and config leak-target was
config target.

Fortinet Inc.
SD-WAN
Display BGP routes by VRF and neighbor
# diagnose ip router bgp set-filter vrf <vrf>

# diagnose ip router bgp set-filter neighbor <neighbor address>
# diagnose ip router bgp set-filter reset
# execute router clear bgp vpnv4 unicast soft {in | out}
# get router info filter show
# get router info filter vrf {vrf | all}
Examples
In example 1, multiple companies (or departments of a company) share the ADVPN. Company A and company B each
have two branches in two different locations. Company A's branches (A-1 and A-2) can talk to each other using the VPN
shortcut, but not to company B's branches (B-1 and B-2). Likewise, company B's branches can talk to each other using
the VPN shortcut, but not to company A's branches. Traffic can share the tunnels and shortcuts, but cannot be mixed up.
Example 2 shows that performance SLA health checks can be sent from a spoke's VRF to the loopback on the hub that
is in the same VRF.
Example 3 shows that when traffic is ingress on the hub on one overlay, it will preferably egress on the same overlay.
Example 1
In this example, two spokes each have two tunnels to the hub.
l Each spoke has two VRFs behind it that can use the same IP address or subnets.
l The computers in VRF1 behind spoke 1 can talk to the computers in VRF1 behind spoke 2, but not to any of the
computers in the VRF2s behind either spoke.

Fortinet Inc.
SD-WAN
config router bgp

set as 65505
set additional-path-vpnv4 enable
set cluster-id 11.12.13.14
edit "gr1"
set capability-graceful-restart enable
set next-hop-self-rr enable
set soft-reconfiguration-vpnv4 enable
set remote-as 65505
set additional-path-vpnv4 both
set route-reflector-client-vpnv4 enable
next
edit "gr2"
set next-hop-self-rr enable
set remote-as 65505
next
end
edit 1
set prefix 10.10.100.0 255.255.255.0
next
edit 2
set prefix 10.10.200.0 255.255.255.0
next
end
config network
edit 12
set prefix 11.11.11.11 255.255.255.255
next
edit 22
set prefix 11.11.22.11 255.255.255.255
next
edit 10
set prefix 100.1.1.0 255.255.255.0
next

Fortinet Inc.
SD-WAN
edit 33
set prefix 11.1.1.0 255.255.255.0
next
end
config vrf
edit "0"
set role pe
next
edit "1"
set role ce
set rd "1:1"
set export-rt "1:1"
set import-rt "1:1"
next
edit "2"
set role ce
set rd "2:1"
set export-rt "2:1"
set import-rt "2:1"
next
end
end
edit "p1"
set type dynamic
set interface "vd11-vlan1"
set peertype any
set proposal aes128-sha1
set dpd on-idle
set dhgrp 5
set auto-discovery-offer-interval 10
set psksecret **********
next
edit "p2"
set type dynamic
set peertype any
set dpd on-idle
set dhgrp 5
set auto-discovery-offer-interval 10
next
end

Fortinet Inc.
SD-WAN

edit "p1"
set phase1name "p1"
set dhgrp 5
next
edit "p2"
set phase1name "p2"
set dhgrp 5
next
end
config router bgp

set as 65505
set ebgp-multipath enable
set network-import-check disable
set additional-path6 enable
set recursive-next-hop enable
set graceful-restart enable
config neighbor
edit "10.10.100.254"
set capability-dynamic enable
set capability-graceful-restart-vpnv4 enable
set remote-as 65505
next
edit "10.10.200.254"
set remote-as 65505
next
end
config network
edit 3
set prefix 22.1.1.0 255.255.255.0
next
edit 4
set prefix 12.12.12.0 255.255.255.0
next

Fortinet Inc.
SD-WAN
end
config vrf
edit "0"
set role pe
next
edit "1"
set role ce
set rd "1:1"
set export-rt "1:1"
set import-rt "1:1"
next
edit "2"
set role ce
set rd "2:1"
set export-rt "2:1"
set import-rt "2:1"
next
end
end
edit "vd2-1"
set peertype any
set dhgrp 5
next
edit "vd2-2"
set peertype any
set dhgrp 5
next
end
edit "vd2-1"
set phase1name "vd2-1"
set dhgrp 5
next
edit "vd2-2"

Fortinet Inc.
SD-WAN

set dhgrp 5
next
end
config system sdwan
set status enable
config zone
next
edit "SASE"
next
edit "zon2"
next
end
config members
edit 1
set cost 10
next
edit 2
set cost 20
next
end
config health-check
edit "ping"
set server "11.11.11.11"
set members 1 2
config sla
edit 1
next
end
next
edit "1"
set vrf 1
set members 1 2
next
end
config service
edit 2
set mode sla
set dst "100-200"
config sla
edit "ping"
set id 1
next
end
next
edit 1

Fortinet Inc.
SD-WAN
set name "test-tag"

set mode sla
set dst "001-100"
config sla
edit "ping"
set id 1
next
end
next
end
end
To check the spoke 1 routes:

B* 0.0.0.0/0 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11 vrf 0),
04:42:57, [1/0]
[200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11 vrf 0),
04:42:57, [1/0]
B 1.1.1.1/32 [200/0] via 11.1.1.1 [2] (recursive via 12.1.1.1, vd2-vlan12), 04:42:57,
[1/0]
04:42:57, [1/0]
B 11.11.11.11/32 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11 vrf
0), 04:42:57, [1/0]
[200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11 vrf
0), 04:42:57, [1/0]
B 33.1.1.0/24 [200/0] via 10.10.100.254 [2] (recursive via vd2-1 tunnel 11.1.1.11 vrf
0), 04:42:57, [1/0]
[200/0] via 10.10.200.254 [2] (recursive via vd2-2 tunnel 11.1.2.11 vrf
0), 04:42:57, [1/0]
B 100.1.1.0/24 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11 vrf 0),
04:42:57, [1/0]
[200/0] via 10.10.200.254 (recursive via vd2-2 tunnel 11.1.2.11 vrf 0),
04:42:57, [1/0]

B V 33.1.1.0/24 [200/0] via 10.10.100.3 [2] (recursive via vd2-1 tunnel 11.1.1.11 vrf
0), 04:42:57, [1/0]
[200/0] via 10.10.200.3 [2] (recursive is directly connected, vd2-2_0),
04:42:57, [1/0]

0), 04:42:56, [1/0]
04:42:56, [1/0]
VRF1 routes:
# get router info filter vrf 1
0), 04:44:11, [1/0]

Fortinet Inc.
SD-WAN

04:44:11, [1/0]
To test the configuration on shortcut 1:
1. From VRF1 of spoke 1 ping VRF1 of spoke 2 and from VRF2 of spoke 1 ping VRF2 spoke 2. Both VRF1 and VRF2
source and destination IP addresses are the same, so you can see how the traffic is isolated
2. Check sessions on spoke 1:
The output vd=<vdom ID>:<VRF ID> indicates that sessions are created in and stay in the corresponding VRFs.
l User at 22.1.1.22 in VRF1 on spoke 1 pings 33.1.1.33 in VRF1 on spoke2.
origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty
gwy=10.10.200.3/22.1.1.22
src_mac=02:4c:a5:fc:6a:7f
misc=0 policy_id=1 pol_uuid_idx=566 auth_info=0 chk_client_info=0 vd=1:1
serial=00092eee tos=ff/ff app_list=0 app=0 url_cat=0
rpdb_link_id=ff000001 ngfwid=n/a
npu_state=0x5040001 no_offload
no_ofld_reason: disabled-by-policy non-npu-intf
l User at 22.1.1.22 in VRF2 on spoke 1 pings 33.1.1.33 in VRF2 on spoke2:

origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty
gwy=10.10.200.3/22.1.1.22
serial=00092f77 tos=ff/ff app_list=0 app=0 url_cat=0

Fortinet Inc.
SD-WAN

l User at 22.1.1.22 in VRF1 on spoke 1 pings 33.1.1.33 in VRF1 on spoke 2:
origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty npu
gwy=33.1.1.33/10.10.200.2
serial=000a29fd tos=ff/ff app_list=0 app=0 url_cat=0
rpdb_link_id=00000000 ngfwid=n/a
vlan=0x0000/0x0000
no_ofld_reason: disabled-by-policy

origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty npu
gwy=33.1.1.33/10.10.200.2
dst_mac=02:4c:a5:fc:6a:7f
serial=000a29ca tos=ff/ff app_list=0 app=0 url_cat=0
vlan=0x0000/0x0000

Fortinet Inc.
SD-WAN
1. From VRF1 of spoke 1 ping VRF1 of spoke 2 and from VRF2 of spoke 1 ping VRF2 spoke 2. Both VRF1 and VRF2
source and destination IP addresses are the same, so you can see how the traffic is isolated
# diagnose sys session listsession info: proto=1 proto_state=00 duration=17 expire=45
timeout=0 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty
gwy=10.10.200.3/22.1.1.22
serial=000aa475 tos=ff/ff app_list=0 app=0 url_cat=0

# diagnose sys session listsession info: proto=1 proto_state=00 duration=8 expire=53
timeout=0 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty
gwy=10.10.200.3/22.1.1.22
serial=000aa49f tos=ff/ff app_list=0 app=0 url_cat=0


Fortinet Inc.
SD-WAN

origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty npu
gwy=33.1.1.133/10.10.200.2
serial=000aa476 tos=ff/ff app_list=0 app=0 url_cat=0
vlan=0x0000/0x0000
l User at 22.1.1.22 in VRF2 on spoke 1 pings 33.1.1.133 in VRF2 on spoke2:

origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty npu
gwy=33.1.1.133/10.10.200.2
serial=000aa4a0 tos=ff/ff app_list=0 app=0 url_cat=0
vlan=0x0000/0x0000
Example 2
In this example, SLA health checks are sent from a spoke's VRF to the loopback on the hub that is in the same VRF.

Fortinet Inc.
SD-WAN
To configure the health check:
config system sdwan

config health-check
edit "1"
set server "11.11.22.11"
set vrf 1
set source 22.1.1.2
set members 1 2
config sla
edit 1
next
end
next
end
end
To check the health check status:
# diagnose sys sdwan health-check status 1

Health Check(1):
Seq(1 vd2-1): state(alive), packet-loss(0.000%) latency(0.023), jitter(0.002), mos(4.404),
Seq(2 vd2-2): state(alive), packet-loss(0.000%) latency(0.022), jitter(0.002), mos(4.404),
Example 3
In this example, when traffic from spoke 1 arrives at the hub on tunnel 1, it will egress the hub on tunnel 1 to go to other
spokes. If traffic arrives on tunnel 2, it will egress on tunnel 2, and not tunnel 1.
To configure SD-WAN on the hub:
config system sdwan

set status enable
config zone
set service-sla-tie-break input-device
next
end
config members
edit 1
set interface "p1"
next
edit 2
set interface "p2"
next
end
config health-check
edit "1"
set members 1 2
config sla

Fortinet Inc.
SD-WAN
edit 1
next
end
next
end
config service
edit 1
set mode sla
set dst "all"
config sla
edit "1"
set id 1
next
end
set tie-break input-device
next
end
end
To verify that traffic stays in the same overlay on ingress and egress on the hub:
1. Confirm that the SD-WAN service rule has Tie break set to input-device so that, when SLAs are met on all of
the members, traffic prefers to egress on the same member as the input device:

Tie break: input-device
Members(2):
1: Seq_num(1 p1), alive, sla(0x1), gid(0), cfg_order(0), local cost(0), selected
2: Seq_num(2 p2), alive, sla(0x1), gid(0), cfg_order(1), local cost(0), selected
Dst address(1):
0.0.0.0-255.255.255.255
2. Use diagnose sniffer packet commands to verify that traffic ingress and egress are on the same overlay.
SD-WAN segmentation over a single overlay using IPv6
IPv6 is supported for SD-WAN segmentation over a single overlay. This allows seamless communication between IPv6
devices within virtual routing and forwarding (VRF) overlay networks, benefiting organizations transitioning to IPv6 or
operating in a dual-stack environment.
Example
In this example, multiple companies (or departments of a company) share the ADVPN. Company A and company B each
have two branches in two different locations. Company A's branches (A-1 and A-2) can talk to each other using the VPN
shortcut, but not to company B's branches (B-1 and B-2). Likewise, company B's branches can talk to each other using
the VPN shortcut, but not to company A's branches. Traffic can share the tunnels and shortcuts, but cannot be mixed up.

Fortinet Inc.
SD-WAN
In this example, two spokes each have two tunnels to the hub.
l Each spoke has two VRFs behind it that can use the same IP address or subnets.
l Loopback addresses are used for communication between the spokes and the hub instead of tunnel IP address.
The exchange-ip-addr6 option allows a loopback IPv6 address to be exchanged between

the spokes and the hub in a network. This means that instead of using the tunnel IP address,
which is typically used for communication, the loopback IPv6 address is used.
See config router bgp and config router route-map in the CLI Reference for a comprehensive list of commands.

config router bgp
set as 65100
set network-import-check disable
set additional-path-select6 4

Fortinet Inc.
SD-WAN
edit "EDGEv6"
set activate-vpnv4 disable
set next-hop-self-rr6 enable
set soft-reconfiguration6 enable
set remote-as 65100
set update-source "vd11-lo1"
set additional-path6 both
set adv-additional-path6 4
set route-reflector-client6 enable
next
end
edit 2
set prefix6 2001::10:200:1:0/112
set neighbor-group "EDGEv6"
next
end
config network6
edit 1
set prefix6 2001::10:200:1:0/112
next
end
config vrf6
edit "0"
set role pe
next
edit "1"
set role ce
set rd "1:1"
set export-rt "1:1"
set import-rt "1:1"
next
edit "2"
set role ce
set rd "2:1"
set export-rt "2:1"
set import-rt "2:1"
next
end
end
2. Configure the IPsec phase 1 interface settings:

edit "p1"
set type dynamic
set ike-version 2
set peertype any
set exchange-ip-addr6 2001::10:200:1:1

Fortinet Inc.
SD-WAN
set dpd on-idle
set dhgrp 5
next
edit "p111"
set type dynamic
set ike-version 2
set peertype any
set dpd on-idle
set dhgrp 5
next
end

edit "p1-v6"
set phase1name "p1"
set replay disable
set src-addr-type subnet6
set dst-addr-type subnet6
next
edit "p111-v6"
set phase1name "p111"
set replay disable
next
end

config router bgp
set as 65100

Fortinet Inc.
SD-WAN
set recursive-next-hop enable
set tag-resolve-mode merge
set graceful-restart enable
set additional-path-select6 4
config neighbor
edit "2001::10:200:1:1"
set activate-vpnv4 disable
set capability-graceful-restart6 enable
set soft-reconfiguration6 enable
set remote-as 65100
set route-map-in6 "tag"
set route-map-in-vpnv6 "tag"
set update-source "vd2-lo1"
set additional-path6 both
next
end
config network6
edit 1
set prefix6 2001:22::/64
next
edit 2
set prefix6 2001::10:200:1:100/128
next
end
config vrf6
edit "0"
set role pe
next
edit "1"
set role ce
set rd "1:1"
set export-rt "1:1"
set import-rt "1:1"
next
edit "2"
set role ce
set rd "2:1"
set export-rt "2:1"
set import-rt "2:1"
next
end
end

edit "vd2-1"

Fortinet Inc.
SD-WAN

set ike-version 2
set peertype any
set dhgrp 5
next
edit "vd2-2"
set ike-version 2
set peertype any
set dhgrp 5
next
end

edit "vd2-1-6"
set dhgrp 5
set replay disable
next
edit "vd2-2-6"
set dhgrp 5
set replay disable
next
end

Fortinet Inc.
SD-WAN
config system sdwan

set status enable
config zone
next
end
config members
edit 1
set cost 10
next
edit 2
set cost 20
next
end
config health-check
edit "ping6"
set addr-mode ipv6
set server "2001::10:200:1:1"
set source6 2001::10:200:1:100
set members 1 2
config sla
edit 1
next
end
next
end
config service
edit 61
set addr-mode ipv6
set dst6 "6001-100"
next
edit 62
set addr-mode ipv6
set dst6 "6100-200"
next
end
end
To check the spoke 1 routes:

B 2001::10:200:1:0/112 [200/0] via 2001::10:200:1:1 tag 100 (recursive via vd2-1
tunnel ::11.1.1.11), 1d15h41m
(recursive via vd2-2 tunnel ::111.1.1.11), 1d15h41m, [1024/0]
B 2001::10:200:1:101/128 [200/0] via 2001::10:200:1:1 tag 100 (recursive via vd2-1
tunnel ::11.1.1.11), 1d15h41m

B V 2001:33::/64 [200/0] via 2001::10:200:1:101 tag 100 (recursive via vd2-1 tunnel
::11.1.1.11), 1d15h41m

Fortinet Inc.
SD-WAN

B V 2001:33::/64 [200/0] via 2001::10:200:1:101 tag 100 (recursive via vd2-1 tunnel
::11.1.1.11), 1d15h41m
1. From VRF1 of spoke 1, ping VRF1 of spoke 2.

2. From VRF2 of spoke 1, ping VRF2 spoke 2. Both VRF1 and VRF2 source and destination IP addresses are the
same, so you can see how the traffic is isolated.
3. Verify the session list:
# diagnose sys session6 list
session6 info: proto=58 proto_state=00 duration=3 expire=59 timeout=0 refresh_dir=both
flags=00000000 sockport=0 socktype=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
state=may_dirty
hook=pre dir=org act=noop 2001:22::55:398->2001:33::44:128(:::0)
hook=post dir=reply act=noop 2001:33::44:398->2001:22::55:129(:::0)
src_mac=02:4c:a5:fc:77:6f
serial=0001104d tos=ff/ff ips_view=0 app_list=0 app=0 url_cat=0
rpdb_link_id=ff00003d ngfwid=n/a
total session6: 1
In the output, vd=<vdom_ID>:<VRF_ID> indicates that sessions are created in and stay in the corresponding
VRFs.
Matching BGP extended community route targets in route maps
BGP extended community route targets can be matched in route maps. This can be applied in a scenario where the BGP
route reflector receives routes from many VRFs, and instead of reflecting all routes from all VRFs, users only want to
reflect routes based on a specific extended community route target.
To configure the extended community list:
config router extcommunity-list

edit <name>
set type {standard | expanded}
config rule
edit <id>
set action {deny | permit}

Fortinet Inc.
SD-WAN
set type {rt | soo}

set match <extended_community_specifications>
set regexp <ordered_list_of_attributes>
next
end
next
end
type {standard | Set the extended community list type (standard or expanded).
expanded}
action {deny | permit} Deny or permit route-based operations based on the route's extended community
attribute.
type {rt | soo} Set the extended community type:
l rt: route target
l soo: site of origin
match <extended_ Set the extended community specifications for matching a reserved extended
community_ community (community number in AA:NN format; use quotation marks complex
specifications>
expressions, "123:234 345:456").
regexp <ordered_list_of_ Set the ordered list of extended community attributes as a regular expression.
attributes>
To configure the BGP extended community list in the route map:

edit <name>
config rule
edit <id>
set match-extcommunity <list>
set match-extcommunity-exact {enable | disable}
next
end
next
end
match-extcommunity <list> Set the BGP extended community list to match to.
match-extcommunity-exact Enable/disable exact matching of extended communities.
{enable | disable}
Example
In this example, multiple companies (or departments of a company) share the same hub and spoke VPN infrastructure.
Company A and company B each have two branches in two different locations. The goal is for company A’s branches (A-
1 and A-2) to be able to communicate only with each other over VPN but not with company B’s branches. Likewise,
company B’s branches (B-1 and B-2) can only communicate with each other over VPN but not with company A’s. This is
accomplished by placing each branch VLAN into their respective VRFs (VRF1 and VRF2), and encapsulating the VRF
information within the VPN tunnel. The hub forms BGP peering with its neighbors, spoke 1 and spoke 2, over each IPsec
overlay. The hub’s BGP route reflector reflects the routes to the corresponding VRFs, allowing each spoke to form
ADVPN shortcuts with the other spoke for each VRF.

Fortinet Inc.
SD-WAN
However, in this scenario, we want A-1 and A-2 to use an ADVPN shortcut, but we do not want B-1 and B-2 to use
ADVPN. A route map is configured on the hub to match the desired extended community route target number where only
this route target is permitted, and others are denied. This allows the hub’s BGP route reflector to only reflect routes
associated with VRF1, allowing the spokes to form an ADVPN shortcut for VRF1. Routes associated with VRF2 are not
reflected, and each spoke must route traffic through the hub to reach VRF2 on the other spoke.
Configure the topology by following the instructions of Example 1 in SD-WAN segmentation over a single overlay on
page 1095. Note that when checking the spoke 1 routes in example 1, there is a VRF2 route:
Spoke 1 # get router info routing-table bgp
…
B V 33.1.1.0/24 [200/0] via 10.10.100.3 [2] (recursive via vd2-1 tunnel 11.1.1.11),
00:00:20, [1/0]
[200/0] via 10.10.200.3 [2] (recursive via vd2-2 tunnel 11.1.2.11),
00:00:20, [1/0]
The following procedure applies a route map on the hub’s BGP configurations to limit route reflection to only routes
matching the external community target of 1:1. This external community target corresponds to BGP paths for VRF1
learned from spoke 1 and spoke 2. The external community target of 2:1 corresponds to BGP paths for VRF2. By not
explicitly permitting this target (2:1) in the community list and denying everything other than the permitted target (1:1) in
the route map, the VRF2 BGP paths are essentially omitted from being reflected to the spokes.
To configure BGP filtering for an extended community route target on the hub:
1. Identify the external community target of VRF1 to be permitted:

# get router info bgp network 33.1.1.0/24
11.1.1.1
Advertised to peer-groups:

Fortinet Inc.
SD-WAN
gr1 gr2
…
Original VRF 0 external duplicated
0.0.0.0 from 10.10.100.3 (3.3.3.3)
Extended Community: RT:1:1
Receive Path ID: 1
Advertised Path ID: 1
Last update: Wed Aug 17 10:31:02 2022
0.0.0.0 from 10.10.200.3 (3.3.3.3)
Receive Path ID: 1
0.0.0.0 from 10.10.100.3 (3.3.3.3)
Receive Path ID: 1
0.0.0.0 from 10.10.200.3 (3.3.3.3)
Receive Path ID: 1
2. Configure the extended community list:

config router extcommunity-list
edit "extcomm1"
config rule
edit 1
set action permit
set match "1:1"
set type rt
next
end
next
end
3. Apply the extended community list to the route map:

Fortinet Inc.
SD-WAN

edit "rmp11"
config rule
edit 1
set match-extcommunity "extcomm1"
next
edit 2
set action deny
next
end
next
end
4. Update the related BGP neighbor group settings:

config router bgp
edit "gr1"
set route-map-out-vpnv4 "rmp11"
next
edit "gr2"
set route-map-out-vpnv4 "rmp11"
next
end
end
5. Refresh the routes:

# execute router clear bgp all vpnv4 unicast out
6. Check the spoke 1 routes. Since the extended community route target is applied, the VFR2 route does not appear in
the BGP routing table:
B* 0.0.0.0/0 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11),
03:47:50, [1/0]
03:47:50, [1/0]
03:47:50, [1/0]
03:47:50, [1/0]
03:47:50, [1/0]
03:47:50, [1/0]
B 33.1.1.0/24 [200/0] via 10.10.100.254 [2] (recursive via vd2-1 tunnel
11.1.1.11), 03:47:21, [1/0]
[200/0] via 10.10.200.254 [2] (recursive via vd2-2 tunnel
11.1.2.11), 03:47:21, [1/0]

B V 11.11.22.11/32 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11),
03:47:50, [1/0]
03:47:50, [1/0]
B V 33.1.1.0/24 [200/0] via 10.10.100.3 [2] (recursive via vd2-1 tunnel 11.1.1.11),

Fortinet Inc.
SD-WAN
03:47:21, [1/0]
[200/0] via 10.10.200.3 [2] (recursive via vd2-2 tunnel 11.1.2.11),
03:47:21, [1/0]
B V 100.1.1.0/24 [200/0] via 10.10.100.254 (recursive via vd2-1 tunnel 11.1.1.11),
03:47:50, [1/0]
03:47:50, [1/0]
Copying the DSCP value from the session original direction to its reply direction
In an SD-WAN scenario when DSCP tags are used to mark traffic from the spoke to the hub, it is sometimes desirable for
the hub to mark the reply traffic with the same DSSP tags. The diffserv-copy setting in firewall policy configurations
allows the DSCP tag to be copied to the reply direction.
edit <id>
set diffserv-copy {enable | disable}
next
end
Example
The use cases in this example are for a hub and spoke SD-WAN deployment. Traffic from the spoke (either real traffic or
SLA health check probes) can be marked with a certain DSCP tag when leaving the spoke. QoS may be applied by an
upstream device based on the DSCP tag. When the traffic arrives on the hub, the hub may also want to mark the reply
traffic to the spoke with the same DSCP tag. This would allow QoS to be applied to the traffic in the reply direction as
well, which is traffic in the hub to spoke direction associated with the same session in the spoke to hub direction.
While this topology simplifies the SD-WAN deployment into a single hub and spoke, this feature applies to the following
configurations:
l Multiple spokes (branch sites)
l One or more hubs (data center sites)
l Multiple overlays connecting spokes to hubs
l SD-WAN configured on spokes to pick the best overlay
Use case 1: typical forwarding traffic
Traffic originates from the spoke and is destined for a server behind the hub. The spoke marks the traffic with a DSCP
tag of 101010. This is done by enabling diffserv-foward on the spoke firewall policy. It can also be accomplished by
enabling dscp-forward in an SD-WAN rule.
The hub allows the traffic in through a firewall policy. By enabling diffserv-copy on the firewall policy, it will mark the
reply traffic on the corresponding sessions with the same DSCP tag in which it came.

Fortinet Inc.
SD-WAN
1. Configure the firewall policy on the spoke (FortiGate A):

edit 1
set srcintf "port1"
set action accept
set srcaddr "all"
set dstaddr "all"
set srcaddr6 "all6"
set dstaddr6 "all6"
set service "ALL"
set logtraffic all
set nat enable
set diffserv-forward enable
set diffservcode-forward 101010
next
end
2. Configure the firewall policy on the hub (FortiGate B):

edit 3
set dstintf "wan1"
set action accept
set srcaddr "all"
set dstaddr "all"
set srcaddr6 "all"
set dstaddr6 "all"
set service "ALL"
set logtraffic all
set auto-asic-offload disable
set nat enable
set diffserv-copy enable
next
end
1. Generate some forwarding traffic.

2. Verify that the session’s tos value from the original direction is applied to the reply direction:
# diagnose sys session filter policy 3
# diagnose sys session filter dst 172.16.200.55

origin-shaper=
reply-shaper=
per_ip_shaper=

Fortinet Inc.
SD-WAN
state=log may_dirty f00

hook=post dir=org act=snat 10.2.2.1:25290->172.16.200.55:8(172.16.200.2:25290)
hook=pre dir=reply act=dnat 172.16.200.55:25290->172.16.200.2:0(10.2.2.1:25290)
serial=0000a018 tos=6a/6a app_list=0 app=0 url_cat=0
total session 1
Use case 2: local-in traffic destined to a loopback interface
SLA health checks from the spoke are destined for a loopback interface on the hub. The health check is marked with a
DSCP tag of 000001 by the spoke. When the hub receives the probes to its loopback, it will mark the replies with the
same DSCP tags in which it came.
1. Configure the health check on the spoke (FortiGate A):

config system sdwan
config health-check
edit "ping"
set diffservcode 000001
set members 0
next
end
2. Configure the loopback interface on the hub (FortiGate B):

edit "loopback"
set vdom "vdom1"
set ip 1.1.1.1 255.255.255.255
set allowaccess ping https ssh http telnet
set type loopback
set role lan
set snmp-index 35
next
end
3. Configure the firewall policy on the hub:

edit 1
set dstintf "loopback"
set action accept
set srcaddr "all"
set dstaddr "all"
set srcaddr6 "all"
set dstaddr6 "all"

Fortinet Inc.
SD-WAN
set service "ALL"

set logtraffic all
set auto-asic-offload disable
set nat enable
set diffserv-copy enable
next
end
1. Generate some local-in traffic.

2. Verify that the session’s tos value from the original direction is applied to the reply direction:

origin-shaper=
reply-shaper=
per_ip_shaper=
state=log local may_dirty
orgin->sink: org pre->in, reply out->post dev=20->42/42->20 gwy=1.1.1.1/0.0.0.0
serial=0001a846 tos=41/41 app_list=0 app=0 url_cat=0
no_ofld_reason: local disabled-by-policy
total session 1
Capture packets can also be used verify that the DSCP value from the original direction is
applied to the reply direction.
SD-WAN cloud on-ramp
In this example, you configure a connection to a new cloud deployment that has some remote servers. SD-WAN is used
to steer traffic through the required overlay tunnel.
The on-premise FortiGate has two internet connections, each with a single VPN connection. The two VPN gateways are
configured on the cloud for redundancy, one terminating at the FortiGate-VM, and the other at the native AWS VPN
Gateway.
This example uses AWS as the Infrastructure as a Service (IaaS) provider, but the same configuration can also apply to
other services. A full mesh VPN setup is not shown, but can be added later if required.

Fortinet Inc.
SD-WAN
To connect to the servers that are behind the cloud FortiGate-VM, virtual IP addresses (VIPs) are configured on port2 to
map to the servers:
l VPN traffic terminating on port1 is routed to the VIP on port2 to access the web servers.
l VPN traffic terminating on the VPN gateway accesses the VIPs on port2 directly.
There are four major steps to configure this setup:
1. Configuring the VPN overlay between the HQ FortiGate and cloud FortiGate-VM on page 1126
2. Configuring the VPN overlay between the HQ FortiGate and AWS native VPN gateway on page 1131
3. Configuring the VIP to access the remote servers on page 1135
4. Configuring the SD-WAN to steer traffic between the overlays on page 1137
After the configuration is complete, verify the traffic to ensure that the configuration is working as expected, see Verifying
the traffic on page 1141.
Configuring the VPN overlay between the HQ FortiGate and cloud FortiGate-VM
Configure the cloud FortiGate-VM
To create an address for the VPN gateway:
1. Go to Policy & Objects > Addresses and select Address.

2. Click Create new.
3. Set Name to local_subnet_10_0_2_0.

Fortinet Inc.
SD-WAN
4. Set IP/Netmask to 10.0.2.0/24.
5. Click OK.
To configure a custom IPsec VPN:
1. Go to VPN > IPsec Wizard.

2. Set Name to Core_Dialup.
3. Set Template type to Custom.
4. Click Next.
5. Configure Network settings:
Remote Gateway Dialup User
Interface port1
NAT Traversal Enable
6. Configure Authentication settings:
Method Pre-shared Key
Pre-shared Key Enter the pre-shared key.
Version 1
Mode Aggressive
This setting allows the peer ID to be specified.
Accept Types Specific peer ID
Peer ID IaaS
The other end of the tunnel needs to have its local ID set to IaaS.

Fortinet Inc.
SD-WAN
7. Leave the default Phase 1 Proposal settings and disable XAUTH.

8. Configure the Phase 2 Selector settings:
Name Ent_Core
Local Address Named Address - local_subnet_10_0_2_0
Remote Address Named Address - all

This setting allows traffic originating from both the remote subnet 10.100.88.0
and the health checks from the VPN interface on the remote FortiGate. For
increased security, each subnet can be specified individually.
9. Click OK.
To configure remote and local tunnel IP addresses:
1. Go to Network > Interfaces and edit the Core_Dialup interface under port1.
2. Set IP to 172.16.200.1.
3. Set Remote IP/Netmask to 172.16.200.2 255.255.255.0. This is where remote health check traffic will come from.
4. Enable Administrative access for HTTPS, PING, and SSH.
5. Click OK.
To configure a route to the remote subnet through the tunnel:
1. Go to Network > Static Routes and click Create New.

2. Set Destination to Subnet and enter the IP address and netmask: 10.100.88.0/255.255.255.0.
3. Set Interface to Core_Dialup.

Fortinet Inc.
SD-WAN
4. Click OK.
To configure a firewall policy to allow traffic from the tunnel to port2:
2. Configure the following:
Name Core_Dialup-to-port2
Incoming Interface Core_Dialup
Outgoing Interface port2
Source all
Destination local_subnet_10_0_2_0
Schedule always
Service ALL
Action ACCEPT
3. Configure the remaining settings as required.

4. Click OK.
Configure the HQ FortiGate
To create an address for the VPN gateway:
1. Go to Policy & Objects > Addresses and select Address.

3. Set Name to remote_subnet_10_0_2_0.

Fortinet Inc.
SD-WAN
4. Set IP/Netmask to 10.0.2.0/24.

5. Click OK.

2. Set Name to FGT_AWS_Tun.
4. Click Next.
Remote Gateway Static IP Address
IP Address 100.21.29.17
Interface port5
Version 1
Mode Aggressive
This setting allows the peer ID to be specified.
Accept Types Any peer ID
7. Leave the default Phase 1 Proposal settings, except set Local ID to IaaS.
8. Disable XAUTH.
Name FGT_AWS_Tun
Local Address Named Address - all

This setting allows traffic originating from both the local subnet 10.100.88.0
and the health checks from the VPN interface. For increased security, each
subnet can be specified individually.
Remote Address Named Address - remote_subnet_10_0_2_0
10. Click OK.
To configure local and remote tunnel IP addresses:
1. Go to Network > Interfaces and edit the FGT_AWS_Tun interface under port5.
2. Set IP to 172.16.200.2.
3. Set Remote IP/Netmask to 172.16.200.1 255.255.255.0.
4. Enable Administrative access for HTTPS, PING, and SSH.
5. Click OK.

Fortinet Inc.
SD-WAN
Routing is defined when creating the SD-WAN interface. The firewall policy is created after the
SD-WAN interface is defined.
Configuring the VPN overlay between the HQ FortiGate and AWS native VPN
gateway
This example uses static routing. It is assumed that the AWS VPN Gateway is already configured, and that proper
routing is applied on the corresponding subnet.
Verify the AWS configuration
See Creating routing tables and associate subnets in the AWS Administration Guide for configuration details.
To check the AWS configuration:
1. Go to Virtual Private Network (VPN) > Customer Gateways to confirm that the customer gateway defines the
FortiGate IP address as its Gateway IP address, in this case 34.66.121.231.
2. Go to Virtual Private Network (VPN) > Virtual Private Gateways to confirm that a virtual private gateway (VPG) has
been created. In this case it is attached to the Cloud_onRamp VPC that contains the FortiGate and servers.
3. Go to Virtual Private Network (VPN) > Site-to-Site VPN Connections to confirm that site-to-site VPN connections
have been created and attached to the customer gateway and virtual private gateway.
If Routing Options is Static, the IP prefix of the remote subnet on the HQ FortiGate (10.100.88.0) is entered here.
AWS site-to-site VPN always creates two VPN tunnels for redundancy. In this example, only Tunnel 1 is used.

Fortinet Inc.
SD-WAN
4. Click Download Configuration to download the FortiGate's tunnel configurations. The configuration can be referred
to when configuring the FortiGate VPN.
5. The new VPG is attached to your VPC, but to successfully route traffic to the VPG, proper routing must be defined.
Go to Virtual Private Cloud > Subnets, select the Cloud-OnRamp-VPN, and select the Route Table tab to verify that
there are at least two routes to send traffic over the VPG.
l 169.254.0.0/24 defines the tunnel IP address. Health check traffic originating from the FortiGate will come from
this IP range.
l 10.100.0.0/16 defines the remote subnet from the HQ FortiGate.
l Both routes point to the just created VPG vgw-04xxxx.
6. On the cloud FortiGate-VM EC2 instances, ensure that port1 and port2 both have Source/Dest. Check set to false.
This allows the FortiGate to accept and route traffic to and from a different network.
If you launched the instance from the AWS marketplace, this setting defaults to true.

Fortinet Inc.
SD-WAN
Configure routing to the VPG on the cloud FortiGate-VM
To configure routing to the VPG on the cloud FortiGate-VM:

3. Set Gateway Address to Specify and enter 10.0.2.1.
4. Set Interface to port2.
The new route must have the same Administrative Distance as the route that was created for traffic through the
Core_Dialup tunnel to ensure that both routes are added to the routing table (see To configure a route to the remote
subnet through the tunnel).
The Gateway Address is arbitrarily set to 10.0.2.1. The VPG does not have an IP address, but the address defined
here allows the FortiGate to route traffic out of port2, while AWS routes the traffic based on its routing table.
5. Click OK.
6. Go to Network > Static Routes to view the configured static routes:
7. If Optimal dashboards is selected, go to Dashboard > Network and expand the Routing widget to view the routing
table.
If Comprehensive dashboards is selected, go to Dashboard > Routing Monitor and select Static & Dynamic in the
widget toolbar to view the routing table:

Fortinet Inc.
SD-WAN
Configure IPsec VPN on the HQ FortiGate

2. Set Name to AWS_VPG.
4. Click Next.
Remote Gateway Static IP Address
IP Address 34.210.19.225
This address is taken from the downloaded AWS configuration file.
Interface port1
Version 1
Mode Main
7. Configure the Phase 1 Proposal settings using information from the downloaded AWS configuration file.
8. Disable XAUTH.
Name AWS_VPG
Local Address Named Address - all

This setting allows traffic originating from both the local subnet 10.100.88.0
and the health checks from the VPN interface. For increased security, each
subnet can be specified individually.
Remote Address Named Address - remote_subnet_10_0_2_0
10. Click OK.
To configure local and remote tunnel IP addresses:
1. Go to Network > Interfaces and edit the AWS_VPG interface under port1.
2. Set IP to 169.254.55.154.
3. Set Remote IP/Netmask to 169.254.55.153 255.255.255.0.
4. Enable Administrative access for HTTPS and PING.
5. Click OK.

Fortinet Inc.
SD-WAN
Routing is defined when creating the SD-WAN interface. The firewall policy is created after the
SD-WAN interface is defined.
Configuring the VIP to access the remote servers
VIPs, interface IP addresses, and policies are created on the cloud FortiGate-VM to allow access to the remote servers.
To configure additional private IPs on AWS for the FortiGate VIP:
1. On the FortiGate EC2 instance, edit the Elastic Network Interface that corresponds to port2. In this example,
Network Interface eth1.
2. Go to Actions > Manage IP Addresses.
3. Add two private IP address in the 10.0.2.0/24 subnet.
These address will be used in the VIPs on the FortiGate. This ensures that traffic to these IP addresses is routed to
the FortiGate by AWS.
4. Click Yes, Update.
To configure VIPs on the cloud FortiGate-VM:
1. Go to Policy & Objects > Virtual IPs and select the Virtual IP tab.
Name VIP-HTTP
Interface port2
External IP address/range 10.0.2.20
Map to IPv4 address/range 10.0.3.33

Fortinet Inc.
SD-WAN
4. Click OK.
5. Create a second VIP for the FTP server with the following settings:
Name VIP-FTP
Interface port2
External IP address/range 10.0.2.21
Map to IPv4 address/range 10.0.3.44
To configure firewall policies to allow traffic from port2 to port3:
Name To-WebServer

Fortinet Inc.
SD-WAN
Source all
Destination VIP-HTTP
Schedule always
Service ALL
Action ACCEPT
NAT Enabled

4. Click OK.
5. Create a second policy for the FTP VIP with the following settings:
Name To-FTP
Source all
Destination VIP-FTP
Schedule always
Service ALL
Action ACCEPT
NAT Enabled
6. Click OK.
Configuring the SD-WAN to steer traffic between the overlays
Configure the HQ FortiGate to use two overlay tunnels for SD-WAN, steering HTTPS and HTTP traffic through the FGT_
AWS_Tun tunnel, and SSH and FTP throguh the AWS_VPG tunnel.
1. Add SD-WAN member interfaces
2. Configure a route to the remote network
3. Configure firewall policies

Fortinet Inc.
SD-WAN
4. Configure a health check

5. Configure SD-WAN rules
To add SD-WAN member interfaces:
2. Set Interface to AWS_VPG then click OK.
3. Click Create New > SD-WAN Member again.

4. Set Interface to FGT_AWS_Tun.
5. Set Gateway to 172.16.200.1.
6. Click OK.
To configure a route to the remote network 10.0.2.0/24:

3. Set Interface to virtual-wan-link.
4. Click OK.

Fortinet Inc.
SD-WAN
Individual routes to each tunnel are automatically added to the routing table with the same distance:
To configure firewall policies to allow traffic from the internal subnet to SD-WAN:
Name ISFW-to-IaaS
Outgoing Interface virtual-wan-link
Source all
Destination all
Schedule always
Service ALL
Action ACCEPT
NAT Enabled

4. Click OK.
Once the firewall policies are configured, the VPN tunnels should come up when there is traffic.
To configure a health check to monitor the status of the tunnels:
As you are accessing the servers on the 10.0.2.0/24 subnet, it is preferable to use the FortiGate port2 interface as the
ping server for detection. This ensures that, if the gateway is not reachable in either tunnel, its routes are brought down
and traffic continues on the other tunnel.
1. Go to Network > SD-WAN, select the Performance SLAs tab, and click Create New.
Name ping_AWS_Gateway
Protocol Ping

Fortinet Inc.
SD-WAN
Server 10.0.2.10
Participants Specify
Add AWS_VPG and FGT_AWS_Tun as participants.
3. Click OK.
Health check probes originate from the VPN interface's IP address. This is why the phase2 selectors are configured
with Local Address set to all.
To configure SD-WAN rules to steer traffic:
HTTPS and HTTP traffic is steered to the FGT_AWS_Tun tunnel, and SSH and FTP traffic is steered to the AWS_VPG
tunnel. The Manual algorithm is used in this example.
Name http-to-FGT_AWS_Tun
Source Address all
Address remote_subnet_10_0_2_0
Protocol TCP
Port range 80 - 80
Outgoing Interfaces Manual
Interface preference FGT_AWS_Tun
3. Click OK.

Fortinet Inc.
SD-WAN
4. Create other SD-WAN rules as required:
Verifying the traffic
To verify that pings are sent across the IPsec VPN tunnels
l On the HQ FortiGate, run the following CLI command:

# diagnose sniffer packet any 'host 10.0.2.10' 4 0 1 interfaces=[any]
interfaces=[any]
pcap_snapshot: snaplen raised from 0 to 262144
2021-06-05 11:35:14.822600 AWS_VPG out 169.254.55.154 -> 10.0.2.10: icmp: echo request
2021-06-05 11:35:14.822789 FGT_AWS_Tun out 172.16.200.2 -> 10.0.2.10: icmp: echo request
2021-06-05 11:35:14.877862 FGT_AWS_Tun in 10.0.2.10 -> 172.16.200.2: icmp: echo reply
2021-06-05 11:35:14.878887 AWS_VPG in 10.0.2.10 -> 169.254.55.154: icmp: echo reply
l On the cloud FortiGate-VM, run the following CLI command:

# diagnose sniffer packet any 'host 10.0.2.10' 4 0 1 interfaces=[any]
interfaces=[any]
pcap_snapshot: snaplen raised from 0 to 262144
2021-06-05 11:37:57.176329 port2 in 169.254.55.154 -> 10.0.2.10: icmp: echo request
2021-06-05 11:37:57.176363 port2 out 10.0.2.10 -> 169.254.55.154: icmp: echo reply
2021-06-05 11:37:57.176505 Core_Dialup in 172.16.200.2 -> 10.0.2.10: icmp: echo request
2021-06-05 11:37:57.176514 Core_Dialup out 10.0.2.10 -> 172.16.200.2: icmp: echo reply
To verify the SLA health checks on the HQ FortiGate:
1. Go to Network > SD-WAN, select the Performance SLAs tab, select Packet Loss, and click the ping_AWS_Gateway
SLA:

Fortinet Inc.
SD-WAN
2. Run the following CLI command:

…
Seq(1 AWS_VPG): state(alive), packet-loss(0.000%) latency(56.221), jitter(0.290) sla_
map=0x0
Seq(2 FGT_AWS_Tun): state(alive), packet-loss(0.000%) latency(55.039), jitter(0.223)
sla_map=0x0
To verify service rules:
1. Go to Network > SD-WAN and select the SD-WAN Rules tab:
2. Run the following CLI command:


Members:
1: Seq_num(2 FGT_AWS_Tun), alive, selected
Src address:
0.0.0.0-255.255.255.255
Dst address:
10.0.2.0-10.0.2.255

Members:
1: Seq_num(1 AWS_VPG), alive, selected
Src address:
0.0.0.0-255.255.255.255
Dst address:

Fortinet Inc.
SD-WAN
10.0.2.0-10.0.2.255

Members:
1: Seq_num(2 FGT_AWS_Tun), alive, selected
Src address:
0.0.0.0-255.255.255.255
Dst address:
10.0.2.0-10.0.2.255

Members:
Src address:
0.0.0.0-255.255.255.255
Dst address:
10.0.2.21-10.0.2.21
To verify that sessions are going to the correct tunnel:
1. Run the following CLI command to verify that HTTPS and HTTP traffic destined for the Web server at 10.0.2.20
uses FGT_AWS_Tun:

origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=FGT_AWS_Tun/ vlan_cos=0/255
state=log may_dirty npu f00 csf_syncd_log app_valid
src_mac=00:09:0f:00:03:01
misc=0 policy_id=32 auth_info=0 chk_client_info=0 vd=0
serial=00b7442c tos=ff/ff app_list=2000 app=34050 url_cat=0
rpdb_link_id= ff000001 rpdb_svc_id=2154552596 ngfwid=n/a
npu_state=0x3041008

origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=FGT_AWS_Tun/ vlan_cos=0/255
state=log may_dirty ndr f00 csf_syncd_log

Fortinet Inc.
SD-WAN

gwy=172.16.200.1/10.100.88.101
src_mac=00:09:0f:00:03:01
serial=00b74b50 tos=ff/ff app_list=2000 app=0 url_cat=0
npu_state=0x3041008
2. Run the following CLI command to verify that SSH and FTP traffic destined for the FTP server at 10.0.2.21 uses
AWS_VPG:
session info: proto=6 proto_state=11 duration=197 expire=3403 timeout=3600

flags=00000000 socktype=0 sockport=0 av_idx=0 use=4
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=AWS_VPG/ helper=ftp vlan_cos=0/255
state=log may_dirty ndr npu f00 csf_syncd_log app_valid
gwy=169.254.55.153/10.100.88.101
src_mac=00:09:0f:00:03:01
serial=00b72a5f tos=ff/ff app_list=2000 app=15896 url_cat=0
npu_state=0x3041008

origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=AWS_VPG/ vlan_cos=0/255
gwy=169.254.55.153/10.100.88.101

Fortinet Inc.
SD-WAN

src_mac=00:09:0f:00:03:01
serial=00b75287 tos=ff/ff app_list=2000 app=16060 url_cat=0
npu_state=0x3041008
To simulate an issue on an overlay VPN tunnel:
On the cloud FortiGate-VM, disable the firewall policy allowing Core_Dialup to port2.
1. Health-checks through the FGT_AWS_Tun tunnel fail:
a. Go to Network > SD-WAN, select the Performance SLAs tab, select Packet Loss, and click the ping_AWS_
Gateway SLA:
b. Run the following CLI command:

…
Seq(1 AWS_VPG): state(alive), packet-loss(0.000%) latency(52.746), jitter(0.713) sla_
map=0x0
Seq(2 FGT_AWS_Tun): state(dead), packet-loss(19.000%) sla_map=0x0
2. Service rules show that the member is down:

a. Go to Network > SD-WAN and select the SD-WAN Rules tab:


Members:

Fortinet Inc.
SD-WAN
1: Seq_num(2 FGT_AWS_Tun), dead

Src address:
0.0.0.0-255.255.255.255
Dst address:
10.0.2.0-10.0.2.255

Members:
Src address:
0.0.0.0-255.255.255.255
Dst address:
10.0.2.0-10.0.2.255

Members:
1: Seq_num(2 FGT_AWS_Tun), dead
Src address:
0.0.0.0-255.255.255.255
Dst address:
10.0.2.0-10.0.2.255

Members:
Src address:
0.0.0.0-255.255.255.255
Dst address:
10.0.2.21-10.0.2.21
3. Sessions are redirected to the working tunnel:

a. Run the following CLI command:
session info: proto=6 proto_state=11 duration=3 expire=3596 timeout=3600

flags=00000000 socktype=0 sockport=0 av_idx=0 use=4
origin-shaper=
reply-shaper=
per_ip_shaper=
gwy=169.254.55.153/0.0.0.0
src_mac=00:09:0f:00:03:01
serial=00b87199 tos=ff/ff app_list=2000 app=34050 url_cat=0

Fortinet Inc.
SD-WAN
rpdb_link_id= 80000000 rpdb_svc_id=0 ngfwid=n/a

npu_state=0x3041008

origin-shaper=
reply-shaper=
per_ip_shaper=
state=log may_dirty ndr f00 csf_syncd_log
gwy=169.254.55.153/10.100.88.101
src_mac=00:09:0f:00:03:01
serial=00b876bb tos=ff/ff app_list=2000 app=0 url_cat=0
rpdb_link_id= 80000000 rpdb_svc_id=0 ngfwid=n/a
npu_state=0x3041008
total session 2
4. Routes to the FGT_AWS_Tun tunnel are removed:

a. If Optimal dashboards is selected, go to Dashboard > Network and expand the Routing widget to view the
routing table.
If Comprehensive dashboards is selected, go to Dashboard > Routing Monitor and select Static & Dynamic in
the widget toolbar to view the routing table:

# get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
* - candidate default

S* 0.0.0.0/0 [1/0] via 10.100.64.254, port1
[1/0] via 10.100.65.254, port5
S 10.0.2.0/24 [1/0] via 169.254.55.153, AWS_VPG
C 10.0.10.0/24 is directly connected, Branch-HQ-A

Fortinet Inc.
SD-WAN
C 10.0.10.1/32 is directly connected, Branch-HQ-A

…
SD-WAN Network Monitor service
The SD-WAN Network Monitor service is a tool designed to determine upload and download speeds. Speed tests can be
conducted either on-demand or according to a predetermined schedule, measuring upload and download speeds of up
to 1 Gbps. The results of the tests can be used as reference for various applications, including the following:
l Configuring the estimated bandwidth of an interface, which can be employed in conjunction with various WAN
intelligence strategies. See Using speed test results with SD-WAN for more information.
l Configuring the inbandwidth and outbandwidth of an interface for use in traffic shaping. See Using speed test
results with traffic shaping for more information.
l Applying the speed test to dialup VPN tunnels in a hub and spoke deployment to conduct traffic shaping.
FortiOS offers a variety of methods for testing SD-WAN speed. The following table provides a brief overview of each
method and guidance on when it might be most advantageous to use one method over the others.
Service Overview License
CLI speed test l Provides the most flexibility and options, which Requires a valid SD-
enables the speed test to operate with user-defined WAN Network Monitor
parameters. license.
l Results can be used as reference to manually add to
the interface's estimated bandwidth, or inbandwidth
and outbandwidth.
l Server is on the cloud, which is maintained by Fortinet.
GUI speed test l Downloads the speed test server list automatically. Requires a valid SD-
l Results can be added to the interface's estimated WAN Network Monitor
bandwidth with one click. license.
l Results are automatically updated in the interface
measured-upstream-bandwidth and measured-
downstream-bandwidth fields.
l Results can be used as a reference to manually
configure an interface’s inbandwidth and
outbandwidth.
l Easier to use.
Scheduled interface speed l Speed tests can be scheduled to run automatically. Requires a valid SD-
test l Results are automatically updated in the interface WAN Network Monitor
measured-upstream-bandwidth and measured- license.
downstream-bandwidth fields.
outbandwidth.
l Possible to temporarily bypass the bandwidth limits set

Fortinet Inc.
SD-WAN
Service Overview License
on the interface and configure custom maximum

bandwidth limits.
Speed test from hub to l Server is the spoke. License not required.
spoke l Tests initiated from the hub.
l Results are cached for future use.
l Results can be dynamically applied to the dialup
tunnel for egress traffic shaping.
outbandwidth.
Speed test from spokes to l Server is the hub. License not required.
hub l Tests initiated from spokes, even when a spoke is
behind a NAT device.
l Results are cached on the spoke for future use and
sent to the hub.
l Results can be dynamically applied to the dialup
tunnel for egress traffic shaping.
outbandwidth.
CLI speed test
The speed test tool is compatible with iPerf3.6 with SSL support. It can test the upload bandwidth to the FortiGate Cloud
speed test service. It can initiate the server connection and send download requests to the server. The tool can be run up
to 10 times a day.
The FortiGate downloads the speed test server list. The list expires after 24 hours. One of the speed test servers is
selected based on user input.
To configure the speed test settings:
config system speed-test-setting

set latency-threshold <integer>
set multiple-tcp-stream <integer>
end
latency-threshold Set the speed test threshold for the auto mode, in milliseconds (0 - 2000, default =
<integer> 60). If the latency exceeds this threshold, the speed test will use the UDP
protocol; otherwise, it will use the TCP protocol.
multiple-tcp-stream Set the number of parallel client streams for the TCP protocol to run during the
<integer> speed test (1 - 64, default = 4).

Fortinet Inc.
SD-WAN
To download the server list of speed tests:
1. Download the server list from FortiCloud:

# execute speed-test-server download
Download completed.
2. Verify the list:

# execute speed-test-server list
...
FTNT_CA_Toronto valid
Host: 154.52.23.67 5200 fortinet
...
FTNT_CA_Vancouver valid
Host: 154.52.20.6 5200 fortinet
...
FTNT_Global valid
Host: 154.52.6.95 5203 fortinet
...
To run the speed test:
A speed test can be run with or without specifying a server. The system will automatically choose one server from the list
and run the speed test. The test results are shown in the command terminal.
# execute speed-test <interface> <server> {Auto | TCP | UDP}
# diagnose netlink interface speed-test <interface> <server> {Auto | TCP | UDP}
See Speed test examples on page 1171 for a sample configurations.
GUI speed test
An interface speed test can be manually performed on WAN interfaces in the GUI. The results of the test can be added to
the interface's Estimated bandwidth. The estimated upstream and downstream bandwidths can be used in SD-WAN
service rules to determine the best link to use when either load balancing or best quality strategies are selected.
To run an interface speed test on a WAN interface:
1. Go to Network > Interfaces.

2. Edit a WAN interface. The interfaces can be grouped by role using the grouping dropdown on the right side of the
toolbar.

Fortinet Inc.
SD-WAN
3. Click Execute speed test in the right pane.
The CLI equivalent to running a speed test in the GUI is execute speed-test <interface>.
4. When the test completes, click OK in the Confirm pane to apply the results to the estimated bandwidth.
The results can also be applied later by clicking Apply results to estimated bandwidth.
The speed test results are used to populate the Estimated bandwidth fields.
5. Click OK.
The FortiGate must be connected to FortiGuard, and able to reach either the AWS or Google
speed test servers.
Scheduled interface speed test
The SD-WAN Network Monitor service supports running a speed test based on a schedule. The test results are
automatically updated in the interface measured-upstream-bandwidth and measured-downstream-bandwidth
fields. These fields do not impact the interface inbound bandwidth, outbound bandwidth, estimated upstream bandwidth,
or estimated downstream bandwidth settings.
When the scheduled speed tests run, it is possible to temporarily bypass the bandwidth limits set on the interface and
configure custom maximum or minimum bandwidth limits. These configurations are optional.
config system speed-test-schedule
edit <interface>
set mode {Auto | TCP | UDP}
set schedules <schedule> ...
set update-inbandwidth enable {enable | disable}
set update-outbandwidth enable {enable | disable}
set update-inbandwidth-maximum <integer>
set update-outbandwidth-maximum <integer>
next
end

Fortinet Inc.
SD-WAN
mode {Auto | TCP | UDP} Set the protocol used for the speed test:
l Auto (default): dynamically select TCP or UDP based on the speed test
setting.
l UDP: use UDP.
l TCP: use TCP.
update-inbandwidth enable Enable/disable bypassing the interface's inbound bandwidth setting.

{enable | disable}
update-outbandwidth Enable/disable bypassing the interface's outbound bandwidth setting.
enable {enable |
disable}
update-inbandwidth- Set the maximum downloading bandwidth to be used in a speed test, in Kbps (0 -
maximum <integer> 16776000).
update-outbandwidth- Set the maximum uploading bandwidth to be used in a speed test, in Kbps (0 -
maximum <integer> 16776000).
See Example 4 for a sample configuration.
Hub and spoke speed tests
In an SD-WAN hub and spoke topology, connections between sites are typically made through VPN overlays. The hub
usually acts as the VPN gateway with spokes connecting as dial-up VPN clients. In order to estimate the speed of the
connection to each spoke, speed tests can be performed between the hub and each spoke. The results of the speed test
can be used for egress traffic shaping on the VPN overlay tunnel.
A speed test server can be enabled on the hub or spoke with custom speed-test listening ports. The test measures the
speeds of the link to each spoke so that QoS can be applied on the hub to the egress traffic shaping profile assigned to
the IPsec overlay tunnel interface and the respective tunnel. An egress-shaping profile can be applied to local, remote,
or both local and remote IPsec tunnels or no IPsec tunnels. Tests can be in upload or download direction and support
both TCP and UDP protocols.

Fortinet Inc.
SD-WAN
Speed test results are cached for future use. When speed tests are initiated from the hub, the results are cached on the
hub. When speed tests are initiated from the spoke, the results are cached on the spoke, but sent to the hub.
When a speed-test server is enabled on a hub or spoke, two speed test daemons are started and listen on different ports
for different purposes:
l The controller speed test daemon listens on the IPsec overlay interfaces to assign an access token to each
incoming speed test for authentication.
l The speed test daemon listens on the IPsec underlay interfaces to handle the speed tests.
Each incoming speed test request must present the obtained access token to prevent random, unauthorized requests.
Otherwise, the connection is closed immediately. As such, speed test access must be enabled on both the underlay and
the IPsec overlay tunnel interfaces.
edit <interface>
set allowaccess speed-test [other access] ...
next
end
If the IPsec tunnel has a configured exchange-ip, speed test access must also be
configured on the associated interface, such as the loopback interface.
The speed test client can be a hub or a spoke and must have system speed-test-schedule configured and the
dynamic-server setting enabled. The speed-test schedule initiates the test.
On the speed test client, specify whether and how to apply the test results in a shaping profile. The shaping profile must
be configured in the phase1 interface before it can be used with a speed test.
edit <interface>
set server-port <integer>
set ctrl-port <integer>
set update-shaper {disable | local | remote | both}
next
end
set server-port <integer> Specify the port number for the speed-test server used for speed tests (1 - 65535,
default = 5201).
set ctrl-port <integer> Specify the port number for the controller on the speed-test server used for
authentication (1 - 65535, default = 5200).
set update-shaper Set the egress shaper to use the speed test results:
{disable | local | l disable: Disable updating the egress shaper (default).
remote | both}
l local: Update the speed-test client egress shaper.
l remote: Update the speed-test server egress shaper.
l both: Update both the local and remote egress shapers.

Fortinet Inc.
SD-WAN
CLI commands
Enable the speed-test server:

set speedtest-server {enable | disable}
set speedtestd-server-port <integer>
set speedtestd-ctrl-port <integer>
end
speedtest-server {enable Enable/disable the speed test server on the hub or spoke (default = disable).This
| disable} enables iPerf in server mode, which listens on the default iPerf TCP port 5201.
set speedtestd-server- Specify a custom port number (1024 - 65535, default = 5201) for the speed test
port <integer> daemon. The port is used to perform the speed test.
set speedtestd-ctrl-port Specify a custom port number (1024 - 65535. default = 5200) for the controller
<integer> speed test daemon. The port is used to assign access tokens for authentication
prior to performing the speed test.
Enable the speed test client:

edit <interface>
set dynamic-server {enable | disable}
set ctrl-port <integer>
set server-port <integer>
set update-shaper {disable | local | remote | both}
next
end
<interface> The dial-up IPsec tunnel interface on the speed test client. The speed test client
can be the hub or the spokes.
dynamic-server {enable | Enable/disable the dynamic speed test server (default = disable).
disable}
Ctrl-port <integer> Specify the port number for the controller on the speed-test server used for
authentication (1 - 65535, default = 5200).
Server-port <integer> Specify the port number for the speed-test server used for speed tests (1 - 65535,
default = 5201).
Update-shaper {disable | Set the egress shaper to use the speed test results:
local | remote | l disable: Disable updating the egress shaper (default).
both}
l local: Update the speed-test client egress shaper.
l remote: Update the speed-test server egress shaper.
l both: Update both the local and remote egress shapers.
To limit the maximum bandwidth used in the speed test, enable set update-inbandwidth
and set update-outbandwidth. See Scheduled interface speed test on page 1151 for
more information.

Fortinet Inc.
SD-WAN
Enable speed test access on both the underlay and the IPsec overlay tunnel interfaces on the speed test
server:

edit <interface>
set allowaccess speed-test [other access] ...
next
end
<interface> The dial-up IPsec tunnel interface on the speed test server.
set allowaccess {speed- Enable speed-test access on the underlay and IPsec overlay interfaces.
test | [other
access]}
Allow an SD-WAN member on the spoke to switch routes when on speed test from the hub to spokes:
config system sdwan

set speedtest-bypass-routing {enable | disable}
config neighbor
edit <bgp neighbor>
set mode speedtest
next
end
end
speedtest-bypass-routing Enable/disable bypass routing when doing a speed test on an SD-WAN member
{enable | disable} (default = disable).
set mode speedtest Use the speed test to select the neighbor.
Manually run uploading speed test on the physical interfaces of each tunnel of a dial-up IPsec interface:
execute speed-test-dynamic <interface> <tunnel_name> <'y'/'n'> <max-out> <min-out>
<interface> IPsec phase1 interface name.

<tunnel_name> The tunnel name, or all for all tunnels.
<'y'/'n'> Apply the result to the tunnels' shaper or not.
<max-out> The maximum speed used in a speed test, in kbps.
<min-out> The minimum speed used in a speed test, in kbps.
Manually run a non-blocking uploading speed test:
diagnose netlink interface speed-test-tunnel <interface> <tunnel_name>
Debug and test commands:
diagnose debug application Enable debug of the speed test module in the forticron daemon.
speedtest <int>

Fortinet Inc.
SD-WAN
diagnose debug application Enable debug of the speed test server daemon.
speedtestd <int>
diagnose test application forticron List the scheduled speed tests.
9
diagnose test application forticron Show the cached speed test results.
10
diagnose test application forticron Write the cached speed test results to disk.
11
diagnose test application forticron Load the speed test results from disk.
12
diagnose test application forticron Cancel all pending speed tests.
99
Running speed tests from the hub to the spokes in dial-up IPsec tunnels
In this example, the hub is configured as a VPN dial-up server and both of the spokes are connected to the hub. It is
assumed that the VPN configuration is already done, with a dynamic gateway type and kernel device creation (net-
device) disabled. Only one SD-WAN interface is used, so there is only one VPN overlay member in the SD-WAN zone.
Multiple WAN interfaces and VPN overlays could be used.
The VPN interfaces and IP addresses are:
A recurring speed test is configured that runs on the hub over the dial-up interfaces. The speed tests are performed over
the underlay interface from the hub to the spoke. Each spoke is configured to operate as a speed test server and to allow

Fortinet Inc.
SD-WAN
the speed test to run on its underlay interface. The spokes establish BGP peering with the hub over the VPN interface,
and advertises its loopback network to the hub. The specific configuration is only shown for FGT_B.
When the speed test is running, routing through the VPN overlay can be bypassed, and route maps are used to filter the
routes that are advertised to peers. The spoke's route map does not advertise any routes to the peer, forcing the hub to
use others paths to reach the spoke's network.
When no speed tests are running, the spoke's route map allows its network to be advertised on the hub.
When the speed test is complete, the measured egress bandwidth is dynamically applied to the VPN tunnel on the hub,
and the result is cached for future use, in case the tunnel is disconnected and reconnected again.
To configure the hub FortiGate (FGT_A) as the speed-test client:
1. Configure a shaping profile:

edit "profile_1"
edit 1
set class-id 2
set priority low
next
end
next
end
Three classes are used in the profile for low, medium, and high priority traffic. Each class is assigned a guaranteed
and maximum bandwidth as a percentage of the measured bandwidth from the speed test.
2. Configure a shaping policy to assign certain traffic as a class ID:
In this example, all traffic destined to the dialup tunnels are assigned class 3.
edit 2
set service "ALL"
set dstintf "hub-phase1" "hub2-phase1"
set class-id 3
set srcaddr "all"
set dstaddr "all"
next
end
3. Use the shaping profile in the interface:

edit "hub-phase1"
set egress-shaping-profile "profile_1"
next
end
4. Configure a schedule to use for the speed tests:

config firewall schedule recurring
edit "speedtest_recurring"

Fortinet Inc.
SD-WAN
set start 01:00

set end 23:00
set day monday tuesday wednesday thursday friday saturday
next
end
5. Configure the speed test schedule:

The custom controller port used for authentication is set to 6000, and the custom port used to run the speed tests is
set to 7000. The shaping profile is set to local.
edit "hub-phase1"
set schedules "speedtest_recurring"
set dynamic-server enable
set ctrl-port 6000
set server-port 7000
set update-shaper remote
next
end
To configure the spoke FortiGates (FGT_B) as a speed test server:
1. Enable a speed test server with custom speed-test listening ports:

A speed test server is enabled on the hub. Port 7000 will run speed tests, and port 6000 will be the controller used to
issue access tokens for speed test authentication.
set speedtest-server enable
set speedtestd-ctrl-port 6000
set speedtestd-server-port 7000
end
2. Allow speed tests on the underlay:

edit "port1"
append allowaccess speed-test
next
end
3. Allow speed tests on the overlay:

edit "hub-spoke11-p1"
set allowaccess ping speed-test
...
next
end
4. Configure SD-WAN with bypass routing enabled for speed tests on member spoke11-p1:
config system sdwan
set speedtest-bypass-routing enable
config members
edit 1
next

Fortinet Inc.
SD-WAN
end
config neighbor
edit "10.10.100.254"
set member 1
set mode speedtest
next
end
end
5. Configure BGP routing:

edit "No_Speed-Test"
config rule
edit 1
set action permit
next
end
next
edit "Start_Speed-Test"
config rule
edit 1
set action deny
next
end
next
end
config router bgp
set as 65412
config neighbor
edit "10.10.100.254"
set remote-as 65412
set route-map-out "Start_Speed-Test"
set route-map-out-preferable "No_Speed-Test"
next
end
config network
edit 1
set prefix 2.2.2.2 255.255.255.255
next
edit 2
set prefix 10.1.100.0 255.255.255.0
next
end
end
Results
1. Before the speed test starts, FGT_A can receive the route from FGT_B by BGP:
B 2.2.2.2/32 [200/0] via 10.10.100.2 (recursive via 172.16.200.2, hub-phase1),
00:00:10
B 10.1.100.0/24 [200/0] via 10.10.100.2 (recursive via 172.16.200.2, hub-phase1),
00:00:10

Fortinet Inc.
SD-WAN
2. At the scheduled time, the speed test starts for the hub-phase1 interface from hub to spoke:
# diagnose test application forticron 9
Speed test schedules:
Interface Server Update Up/Down-limit (kbps) Days
H:M TOS Schedule
----------------------------------------------------------------------------------------
-----------------------------------
hub-phase1 dynamic 1111111
14:41 0x00 speedtest_recurring
Active schedules:
64002f: hub-phase1(port1) 172.16.200.2 hub-phase1_1
64002e: hub-phase1(port1) 172.16.200.4 hub-phase1_0
The diagnose debug application speedtest -1 command can be used on both the hub and spokes to
check the speed test execution.
3. While the speed test is running, FGT_A does not receive the route from FGT_B by BGP:
4. Speed tests results can be dynamically applied to the dial-up tunnel for egress traffic shaping:
------------------------------------------------------
...
------------------------------------------------------
...

Fortinet Inc.
SD-WAN
5. Speed test results can be cached, indexed, and written to disk:

Speed test results:
1: vdom=root, phase1intf=hub-phase1, peer-id='spoke11-p1', bandwidth=737210, last_
log=1624226603
2: vdom=root, phase1intf=hub-phase1, peer-id='spoke21-p1', bandwidth=726813, last_
log=1624226614
Write 2 logs to disk.

load 2 results.
Disable then reenable the IPsec VPN tunnel and the cached speed test results can be applied to the tunnel again:
------------------------------------------------------
...
------------------------------------------------------
...
Running speed tests from spokes to the hub in dial-up IPsec tunnels
In this hub and spoke example, the hub is configured as an IPsec VPN dial-up server with two IPsec tunnels, and each
tunnel is connected to a spoke. The VPN interfaces and IP addresses are:
The hub (FGT_A) is configured as a speed-test server to listen on custom ports (6000 and 7000), and the spokes (FGT_
B and FGT_D) are configured as speed-test clients. This setup allows speed tests to successfully perform when spokes

Fortinet Inc.
SD-WAN
are behind NAT devices. The results of the speed test will be applied to the hub-phase1 overlay tunnel(s) as specified by
the speed-test clients.
The spokes are configured to initiate speed tests on a schedule on UDP. After the speed test completes, the results are
sent to the hub, and the hub applies the results on its IPsec tunnels as egress traffic shaping. The results are also
cached and can be used if an IPsec tunnel is disconnected and reconnected again.
To configure the hub FortiGate (FGT_A) as the speed test server:
1. Configure a shaping profile:

In this example, the shaping profile is named profile_1.
edit "profile_1"
edit 1
set class-id 2
set priority low
next
edit 2
set class-id 3
set priority medium
next
edit 3
set class-id 4
next
end

Fortinet Inc.
SD-WAN
end
end
Three classes are used in the profile for low, medium, and high priority traffic. Each class is assigned a guaranteed
and maximum bandwidth as a percentage of the measured bandwidth from the speed test.
2. Configure a shaping policy to assign certain traffic as a class ID:
In this example, all traffic destined to the dialup tunnels are assigned class 3.
edit 2
set service "ALL"
set dstintf "hub-phase1" "hub2-phase1"
set class-id 3
set srcaddr "all"
set dstaddr "all"
next
end
3. Enable a speed test server with custom speed-test listening ports:

A speed test server is enabled on the hub. Port 7000 will run speed tests, and port 6000 will be the controller used to
issue access tokens for speed test authentication.
...
set speedtest-server enable
set speedtestd-ctrl-port 6000
set speedtestd-server-port 7000
end
4. Allow the speed test on the underlay:

edit "port1"
set ip 172.16.200.1 255.255.255.0
set allowaccess ping https ssh snmp http telnet fgfm radius-acct probe-response
fabric speed-test
...
next
end
5. Allow the speed test on the overlay and use the shaping profile in the interface:
In this example, speed tests are allowed on the overlay, and the shaping profile (profile_1) is used on the hub
phase1 interface (port1).
edit "hub-phase1"
set ip 10.10.15.253 255.255.255.255
set allowaccess ping speed-test
set egress-shaping-profile "profile_1"
...
next
end

Fortinet Inc.
SD-WAN
To configure the first spoke FortiGate (FGT_B) as a speed test client:
1. Configure system speed-test-schedule:

The protocol mode is set to UDP. The custom controller port used for authentication is set to 6000, and the custom
port used to run the speed tests is set to 7000. The shaping profile is set to remote.
edit "spoke11-p1"
set mode UDP
set schedules "1"
set ctrl-port 6000
next
end
2. Configure a recurring schedule for the speed tests:

Schedule 1 is set to start at 08:37 every day of the week.
edit "1"
set start 08:37
set day sunday monday tuesday wednesday thursday friday saturday
next
end
To configure the second spoke FortiGate (FGT_D) as a speed test client:
1. Configure a speed test schedule:

The protocol mode is set to UDP. The custom controller port used for authentication is set to 6000, and the custom
port used to run the speed tests is set to 7000. The shaping profile is set to remote.
edit "spoke21-p1"
set mode UDP
set schedules "1"
set ctrl-port 6000
next
end
2. Configure a recurring schedule for the speed tests:

Schedule 1 is set to start at 08:37 every day of the week.
edit "1"
set start 08:37
next
end

Fortinet Inc.
SD-WAN
To view the speed test results:
1. After the speed test schedule runs, view the result on spoke FGT_B:
On spoke FGT_B, authentication succeeds through port 6000, and the test runs on port 7000. UDP mode is used,
and the test is successful.
# diagnose debug application speedtest -1
......
fcron_speedtest_ipsec_request_init()-464: root: spoke11-p1(spoke11-p1) id=003900d5
fd=24, init request=0.0.0.0:0 -> 10.10.15.253:6000, test=172.16.200.2:0 ->
172.16.200.1:7000: succeed.
......
[speedtest(2181)] start uploading test.
[speedtest(2181)] Connecting to host 172.16.200.1, port 7000
[speedtest(2181)] [ 26] local 172.16.200.2 port 17553 connected to 172.16.200.1 port
7000
[speedtest(2181)] [ ID] Interval Transfer Bitrate Total Datagrams
[speedtest(2181)] [ 26] 0.00-1.00 sec 150 MBytes 1.26 Gbits/sec 107570
[speedtest(2181)] [ ID] Interval Transfer Bitrate Jitter
Lost/Total Datagrams
[speedtest(2181)] [ 26] 0.00-5.00 sec 747 MBytes 1.25 Gbits/sec 0.000 ms
0/536190 (0%) sender
[speedtest(2181)] [ 26] 0.00-5.00 sec 271 MBytes 454 Mbits/sec 0.000 ms
341627/535995 (64%) receiver
[speedtest(2181)] client(sender): bytes_recv=283777280, bytes_sent=782837400, sender_
time=5.000, recver_time=5.000
[speedtest(2181)] client(sender): up_speed: 454 Mbits/sec
[speedtest(2181)]
[speedtest(2181)] speed test Done.
[speedtest(2181)] start downloading test.
[speedtest(2181)] Reverse mode, remote host 172.16.200.1 is sending
[speedtest(2181)] [ 26] local 172.16.200.2 port 7998 connected to 172.16.200.1 port 7000
[speedtest(2181)] [ 26] 0.00-1.00 sec 54.6 MBytes 458 Mbits/sec 0.007 ms
70745/109978 (64%)
67547/106917 (63%)
67543/106940 (63%)
67636/107024 (63%)
67421/106842 (63%)
0/538540 (0%) sender
340892/537701 (63%) receiver

Fortinet Inc.
SD-WAN
[speedtest(2181)] client(recver): bytes_recv=287341140, bytes_sent=786268400, sender_

[speedtest(2181)] client(recver): down_speed: 460 Mbits/sec
[speedtest(2181)]
fcron_speedtest_notify_func()-1275: Speed test pid=2181 done
fcron_speedtest_on_test_finish()-1211: Test 3900d5 for 'spoke11-p1' succeed with

up=454043, down=459694
fcron_speedtest_save_results()-1144: Write logs to disk: succ=1, fail=0
fcron_speedtest_sync_results()-1172: Sync cached results to secondary devices.
2. After the speed test schedule runs, view the result on the spoke FGT_D:
On spoke FGT_D, authentication succeeds through port 6000, and the test runs on port 7000. UDP mode is used,
and the test is successful.
# diagnose debug application speedtest -1
......
fcron_speedtest_ipsec_request_init()-464: root: spoke21-p1(spoke21-p1) id=00380011
fd=25, init request=0.0.0.0:0 -> 10.10.15.253:6000, test=172.16.200.4:0 ->
172.16.200.1:7000: succeed.
......
[speedtest(4309)] start uploading test.
7000
[speedtest(4309)] [ ID] Interval Transfer Bitrate Total Datagrams
0/529840 (0%) sender
335130/529650 (63%) receiver
[speedtest(4309)] client(sender): bytes_recv=283999200, bytes_sent=773566400, sender_
[speedtest(4309)] client(sender): up_speed: 454 Mbits/sec
[speedtest(4309)]
[speedtest(4309)] start downloading test.
[speedtest(4309)] Reverse mode, remote host 172.16.200.1 is sending
7000
70258/110574 (64%)
66496/106740 (62%)

Fortinet Inc.
SD-WAN
66481/106736 (62%)
66403/106690 (62%)
65991/106454 (62%)
0/538110 (0%) sender
335629/537194 (62%) receiver
[speedtest(4309)] client(recver): bytes_recv=294284900, bytes_sent=785640600, sender_
[speedtest(4309)] client(recver): down_speed: 471 Mbits/sec
[speedtest(4309)]
fcron_speedtest_notify_func()-1275: Speed test pid=4309 done
fcron_speedtest_on_test_finish()-1211: Test 380011 for 'spoke21-p1' succeed with

up=454398, down=470794
fcron_speedtest_save_results()-1144: Write logs to disk: succ=1, fail=0
fcron_speedtest_sync_results()-1172: Sync cached results to secondary devices.
3. After the speed test schedule runs, view the result on the hub (FGT_A):
The server side uses speedtestd, while the client side uses speedtest.
The speed test results are applied on hub-phase1_0 and hub_phase1_1 as egress traffic shaping.
# diagnose debug application speedtestd -1
......
17553
......
[speedtest(2771)] [ 7] local 172.16.200.1 port 7000 connected to 172.16.200.2 port 7998
......
[sptestd::ctrl(0377):root] set shaper: if=hub-phase1, tun=hub-phase1_0, sp=profile_1,
bw=459745
......
15349
......
19586
......
[sptestd::ctrl(0377):root] set shaper: if=hub-phase1, tun=hub-phase1_1, sp=profile_1,
bw=470855
......
4. Verify the result is cached on the spokes.

Fortinet Inc.
SD-WAN
l On FGT_B, the speed test results are cached:

Speed test results:
1: vdom=root, phase1intf=spoke11-p1, peer-id='172.16.200.1', up=454043, dw=459694,
time=12/13 12:32:19
l On FGT_D, the speed test results are cached:

Speed test results:
1: vdom=root, phase1intf=spoke21-p1, peer-id='172.16.200.1', up=454398, dw=470794,
time=12/12 16:33:18
5. On the hub (FGT_A), verify the speed test results are applied to the hub's IPsec tunnels as egress traffic shaping:
On hub-phase1_0 and hub-phase1_1, the correct traffic control is displayed.
......
------------------------------------------------------
name=hub-phase1_0 ver=2 serial=16 172.16.200.1:0->172.16.200.2:0 tun_id=10.10.15.1 tun_
id6=2000:10:10:15::1 dst_mtu=1500 dpd-link=on weight=1
bound_if=11 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/74408 options[122a8]=npu
rgwy-chg frag-rfc run_state=0 role=primary accept_traffic=1 overlay_id=10
parent=hub-phase1 index=0
......
priority=low forwarded_bytes=86K
------------------------------------------------------
name=hub-phase1_1 ver=2 serial=17 172.16.200.1:0->172.16.200.4:0 tun_id=10.10.15.2 tun_
id6=2000:10:10:15::2 dst_mtu=1500 dpd-link=on weight=1
bound_if=11 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/74408 options[122a8]=npu
rgwy-chg frag-rfc run_state=0 role=primary accept_traffic=1 overlay_id=10
parent=hub-phase1 index=1
......

Fortinet Inc.
SD-WAN
priority=low forwarded_bytes=81K
Speed test usage
Using speed test results with SD-WAN
The interface speed test can be used to populate the bandwidth values based on the results.
To manually configure the upstream and downstream interface bandwidth values in the GUI:

2. Edit a WAN interface.
3. Input the results from the speed test in the Estimated bandwidth section, specifically into the appropriate kbps
Upstream or kbps Downstream fields.
4. Click OK.
Alternatively, use the GUI speed test on page 1150 to add the speed test results to the
interface's estimated bandwidth with one click.
To manually configure the upstream and downstream interface bandwidth values in the CLI:

edit <interface>
set estimated-upstream-bandwidth <speed in kbps>
set estimated-downstream-bandwidth <speed in kbps>
next
end
The populated bandwidth values can be employed in conjunction with various WAN intelligence strategies such as load
balancing without SLA targets, load balancing with SLA targets, and best quality. These strategies are used to select an
SD-WAN interface in SD-WAN service rules.
Example
In this example, SD-WAN interfaces, wan1 and wan2, are connected to two different internet service providers (ISPs),
both providing access to the public internet. The preference is for Gmail services to use the link with the highest
bandwidth.

Fortinet Inc.
SD-WAN
To apply speed test results to a best quality SD-WAN rule:
1. Run theGUI speed test on page 1150 on wan1 and wan2:

a. When the test is complete, click OK in the Confirm pane to apply the results to the estimated bandwidth.
2. Add wan1 and wan2 as SD-WAN members, then add a policy and static route. See SD-WAN quick start on page
789 for more information.
3. Configure the health check:
a. Go to Network > SD-WAN, select the Performance SLAs tab, and click Create New.
b. Enter a name for the performance SLA, such as google, and set the Server to google.com. See Health checks
for more information.
c. Click OK.
4. Configure an SD-WAN rule to use best quality with bandwidth set as the quality criteria:
a. Go to Network > SD-WAN, select the SD-WAN Rules tab, and click Create New.
b. Enter a name for the rule, such as gmail.
c. Configure the following settings:
Internet service Google-Gmail
Interface Best quality

selection strategy
Interface wan1 and wan2

preference
Measured SLA google
Quality criteria Bandwidth
d. Click OK.
5. Verify that the Estimated bandwidth values are used to select an SD-WAN interface in SD-WAN service rules.
a. Verify the health check status:
# diagnose sys sdwan health-check status
Health Check(google):
Seq(1 wan1): state(alive), packet-loss(0.000%) latency(4.996), jitter(0.824), mos
Seq(2 wan2): state(alive), packet-loss(0.000%) latency(4.634), jitter(0.340), mos
(4.402), bandwidth-up(443975), bandwidth-dw(699264), bandwidth-bi(1143239) sla_
map=0x0
b. Verify the service rule 1 diagnostics:

Fortinet Inc.
SD-WAN
# diagnose sys sdwan service4 1

Tie break: cfg
Gen(1), TOS(0x0/0x0), Protocol(0): src(1->65535):dst(1->65535), Mode(priority),
link-cost-factor(bibandwidth), link-cost-threshold(10), heath-check(google)
Members(2):
1: Seq_num(2 wan2 virtual-wan-link), alive, bibandwidth: 1143239Kbps, selected
2: Seq_num(1 wan1 virtual-wan-link), alive, bibandwidth: 819506Kbps, selected
Internet Service(1): Google-Gmail(65646,0,0,0,0)
Since wan2 has superior bandwidth, SD-WAN will prioritize Seq_num(2) over Seq_num(1), and wan2 will be
utilized to route Gmail traffic. The bandwidth is automatically determined by the Estimated bandwidth values, which
were populated by running a speed test in the GUI.
Using speed test results with traffic shaping
The interface speed test can be used to populate the bandwidth values based on the results.
To configure the outbandwidth value in the GUI:

2. Edit an interface.
3. In the Traffic Shaping section, enable Outbound bandwidth and enter the result obtained from the speed test’s
uploading mode.
4. Click OK.
To configure the outbandwidth and inbandwidth values in the CLI:

edit <interface>
set outbandwidth <speed_in_kbps>
set inbandwidth <speed_in_kbps>
next
end
The populated bandwidth values can be used in conjunction with either an egress or ingress traffic shaping profile.
Speed test examples
This topic includes examples that show various tests based on different modes (auto, TCP, UDP), latency thresholds,
and test servers. Some test protocols and servers are manually configured, while others are chosen by the FortiGate.

Fortinet Inc.
SD-WAN
These examples assume the FortiGate is connected to the internet, has a valid SD-WAN Network Monitor license, and
has downloaded the server list of speed tests from FortiCloud. See CLI speed test on page 1149 for more information.
l Example 1: executing a speed test without specifying the interface, server, and mode
l Example 2: executing a speed test with a lower latency threshold setting
l Example 3: executing the speed test with diagnose netlink interface speed-test
l Example 4: executing the speed test according to the schedule
l Example 5: executing multiple speed tests with TCP and UDP connections
Example 1: executing a speed test without specifying the interface, server, and mode
Geographically, the Vancouver server (154.52.20.6) has the smallest latency (around 7 ms) to FGT_A, so it will be
automatically selected for the speed test because the latency 7 ms to 154.52.20.6 is less than the default latency-
threshold of 60 ms. Meanwhile, four TCP connections will be initiated to perform the test since the default multiple-
tcp-stream is 4.
To execute the speed test without specifying parameters:
1. Configure the speed test settings:

set multiple-tcp-stream 4
end
2. Execute a ping to the closest test server, 154.52.20.6, to learn the latency for the connection:
# execute ping 154.52.20.6
PING 154.52.20.6 (154.52.20.6): 56 data bytes
64 bytes from 154.52.20.6: icmp_seq=0 ttl=50 time=7.5 ms
--- 154.52.20.6 ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 7.1/7.6/9.1 ms
3. Run the speed test with no parameters:

Fortinet Inc.
SD-WAN
# execute speed-test
Initializing speed test.
current vdom=root
Run in uploading mode.
Connecting to host 154.52.20.6, port 5203
[ 7] local 172.16.200.1 port 21219 connected to 154.52.20.6 port 5203
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 7] 0.00-1.00 sec 22.4 MBytes 188 Mbits/sec 17 140 KBytes
[ 9] 0.00-1.00 sec 9.71 MBytes 81.4 Mbits/sec 6 73.5 KBytes
[ 11] 0.00-1.00 sec 18.5 MBytes 155 Mbits/sec 12 117 KBytes
...
[SUM] 0.00-5.02 sec 321 MBytes 536 Mbits/sec receiver
speed test Done.

Run in reverse downloading mode.
Reverse mode, remote host 154.52.20.6 is sending
...
speed test Done.
The tested upload/download speed for port1 is 536 Mbps/555 Mbps when connecting to the closest server with four TCP
connections.
Example 2: executing a speed test with a lower latency threshold setting
The latency-threshold setting is changed to 5 ms, which is less than the latency 7 ms to 154.52.20.6. When
executing the speed test, one UDP connection will be initiated as expected.
To execute the speed test with a lower latency threshold setting:
1. Edit the speed test settings:

end
2. Run the speed test:

# execute speed-test
Speed test quota for 7/19 is 4
current vdom=root
[ ID] Interval Transfer Bitrate Total Datagrams
[ 7] 0.00-1.00 sec 111 MBytes 931 Mbits/sec 80337

Fortinet Inc.
SD-WAN

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 7] 0.00-5.00 sec 556 MBytes 932 Mbits/sec 0.000 ms 0/402364 (0%) sender
[ 7] 0.00-5.04 sec 550 MBytes 917 Mbits/sec 0.017 ms 3787/402339 (0.94%)
receiver
speed test Done.

Reverse mode, remote host 154.52.20.6 is sending
[ 7] 0.00-1.00 sec 72.4 MBytes 607 Mbits/sec 0.013 ms 59813/112240 (53%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 7] 0.00-5.04 sec 764 MBytes 1.27 Gbits/sec 0.000 ms 0/553023 (0%) sender
[SUM] 0.0- 5.0 sec 2 datagrams received out-of-order
[ 7] 0.00-5.00 sec 355 MBytes 595 Mbits/sec 0.014 ms 295028/552025 (53%)
receiver
speed test Done.
The tested upload/download speed for port1 is 917 Mbps/595 Mbps when connecting to the closest server with one UDP
connection.
Example 3: executing the speed test with diagnose netlink interface speed-test
After running this diagnose command, the results are recorded in the interface settings for reference as measured-
upstream-bandwidth and measured-downstream-bandwidth.
To execute the speed test:
# diagnose netlink interface speed-test port1 FTNT_CA_Vancouver TCP

speed-test test ID is b0066
...
To view the interface settings:
show system interface port1

edit "port1"
...
set measured-upstream-bandwidth 735682
set measured-downstream-bandwidth 746573
set bandwidth-measure-time 1689811319
...
next
end

Fortinet Inc.
SD-WAN
Example 4: executing the speed test according to the schedule
After running the speed test, the results are recorded in the interface settings for reference as measured-upstream-
bandwidth and measured-downstream-bandwidth.
To execute the speed test according to the schedule:
1. Configure the recurring schedule:

edit "speedtest_recurring"
set start 17:07
next
end
2. Configure the speed test schedule:

edit "port1"
set mode TCP
set schedules "speedtest_recurring"
next
end
The speed test will be initiated at 17:07 based on 10 TCP connections. The results will be recorded in port1's
interface settings.
3. Verify the speed test results:
show system interface port1
edit "port1"
...
set measured-upstream-bandwidth 715636
set measured-downstream-bandwidth 819682
set bandwidth-measure-time 1689811759
...
next
end
Example 5: executing multiple speed tests with TCP and UDP connections
A speed test is executed to the closest server using 64 TCP connections and another speed test is executed using one
UDP connection. The results can be checked with a third-party platform (such as Ookla), which returns comparable
results.

Fortinet Inc.
SD-WAN
To execute multiple speed tests with TCP and UDP connections:
1. Edit the speed test settings:

set multiple-tcp-stream 64
end
2. Run the TCP speed test:

# execute speed-test port1 FTNT_CA_Vancouver TCP
...
...
[SUM] 0.00-5.00 sec 559 MBytes 938 Mbits/sec 2165 sender
speed test Done.

...
[SUM] 0.00-5.01 sec 505 MBytes 846 Mbits/sec 9329 sender
3. Run the UDP speed test:

# execute speed-test port1 FTNT_CA_Vancouver UDP
...
...
[ 7] 0.00-5.00 sec 556 MBytes 933 Mbits/sec 0.000 ms 0/402727 (0%) sender
[ 7] 0.00-5.04 sec 556 MBytes 925 Mbits/sec 0.020 ms 393/402717 (0.098%)
receiver
...
...
[ 7] 0.00-5.04 sec 869 MBytes 1.45 Gbits/sec 0.000 ms 0/629383 (0%) sender
[SUM] 0.0- 5.0 sec 2 datagrams received out-of-order
[ 7] 0.00-5.00 sec 489 MBytes 821 Mbits/sec 0.005 ms 274103/628393 (44%)
receiver
speed test Done.

Fortinet Inc.
SD-WAN
Troubleshooting SD-WAN
The following topics provide instructions on SD-WAN troubleshooting:

l Tracking SD-WAN sessions on page 1177
l Understanding SD-WAN related logs on page 1177
l SD-WAN related diagnose commands on page 1180
l Using SNMP to monitor health check on page 1185
Tracking SD-WAN sessions
You can check the destination interface in Dashboard > FortiView Sessions in order to see which port the traffic is being
forwarded to.
The example below demonstrates a source-based load-balance between two SD-WAN members:
l If the source IP address is an even number, it will go to port13.
l If the source IP address is an odd number, it will go to port12.
Understanding SD-WAN related logs
This topic lists the SD-WAN related logs and explains when the logs will be triggered.
Health-check detects a failure:
l When health-check detects a failure, it will record a log:

logid="0100022921" type="event" subtype="system" level="critical" vd="root"
logdesc="Routing information changed" name="test" interface="R150" status="down"
msg="Static route on interface R150 may be removed by health-check test. Route:
(10.100.1.2->10.100.2.22 ping-down)"
l When health-check detects a recovery, it will record a log:

logid="0100022921" type="event" subtype="system" level="critical" vd="root"
logdesc="Routing information changed" name="test" interface="R150" status="up"

Fortinet Inc.
SD-WAN
msg="Static route on interface R150 may be added by health-check test. Route:

(10.100.1.2->10.100.2.22 ping-up)"
Health-check has an SLA target and detects SLA qualification changes:
l When health-check has an SLA target and detects SLA changes, and changes to fail:
logid="0113022923" type="event" subtype="sdwan" level="notice" vd="root" logdesc="SDWAN
status" eventtype="Health Check" healthcheck="test" slatargetid=1 oldvalue="2"
newvalue="1" msg="Number of pass member changed."
status" eventtype="Health Check" healthcheck="test" slatargetid=1 member="1" msg="Member
status changed. Member out-of-sla."
l When health-check has an SLA target and detects SLA changes, and changes to pass:
status" eventtype="Health Check" healthcheck="test" slatargetid=1 oldvalue="1"
newvalue="2" msg="Number of pass member changed."
logid="0113022923" type="event" subtype="sdwan" level="information" vd="root"
logdesc="SDWAN status" eventtype="Health Check" healthcheck="test" slatargetid=1
member="1" msg="Member status changed. Member in sla."
SD-WAN calculates a link's session/bandwidth over/under its ratio and stops/resumes traffic:
l When SD-WAN calculates a link's session/bandwidth over its configured ratio and stops forwarding traffic:
volume status" eventtype="Volume" interface="R160" member="2" msg="Member enters into
conservative status with limited ablity to receive new sessions for too much traffic."
l When SD-WAN calculates a link's session/bandwidth according to its ratio and resumes forwarding traffic:
volume status" eventtype="Volume" interface="R160" member="2" msg="Member resume normal
status to receive new sessions for internal adjustment"
The SLA mode service rule's SLA qualified member changes:
l When the SLA mode service rule's SLA qualified member changes. In this example R150 fails the SLA check, but is
still alive:
status" eventtype="Service" serviceid=1 service="test" seq="2,1" msg="Service
prioritized by SLA will be redirected in sequence order."
l When the SLA mode service rule's SLA qualified member changes. In this example R150 changes from fail to pass:

Fortinet Inc.
SD-WAN
status" eventtype="Service" serviceid=1 service="test" seq="1,2" msg="Service

prioritized by SLA will be redirected in sequence order."
The priority mode service rule member's link status changes:
l When priority mode service rule member's link status changes. In this example R150 changes to better than R160,
and both are still alive:
status" eventtype="Service" serviceid=1 service="test" metric="packet-loss" seq="2,1"
msg="Service prioritized by performance metric will be redirected in sequence order."
l When priority mode service rule member's link status changes. In this example R160 changes to better than R150,
and both are still alive:
status" eventtype="Service" serviceid=1 service="test" metric="packet-loss" seq="1,2"
msg="Service prioritized by performance metric will be redirected in sequence order."
SD-WAN member is used in service and it fails the health-check:
l When SD-WAN member fails the health-check, it will stop forwarding traffic:
status" eventtype="Service" interface="R150" member="1" serviceid=1 service="test"
gateway=10.100.1.1 msg="Member link is unreachable or miss threshold. Stop forwarding
traffic. "
l When SD-WAN member passes the health-check again, it will resume forwarding logs:
status" eventtype="Service" interface="R150" member="1" serviceid=1 service="test"
gateway=10.100.1.1 msg="Member link is available. Start forwarding traffic. "
Load-balance mode service rule's SLA qualified member changes:
l When load-balance mode service rule's SLA qualified member changes. In this example R150 changes to not meet
SLA:
status" eventtype="Service" serviceid=1 service="test" member="2(R160)" msg="Service
will be load balanced among members with available routing."
l When load-balance mode service rule's SLA qualified member changes. In this example R150 changes to meet
SLA:
status" eventtype="Service" serviceid=1 service="test" member="2(R160),1(R150)"
msg="Service will be load balanced among members with available routing."

Fortinet Inc.
SD-WAN
SLA link status logs, generated with interval sla-fail-log-period or sla-pass-log-period:
l When SLA fails, SLA link status logs will be generated with interval sla-fail-log-period:
SLA information" eventtype="SLA" healthcheck="test" slatargetid=1 interface="R150"
status="up" latency="0.061" jitter="0.004" packetloss="2.000%"
inbandwidthavailable="0kbps" outbandwidthavailable="200.00Mbps"
bibandwidthavailable="200.00Mbps" inbandwidthused="1kbps" outbandwidthused="1kbps"
bibandwidthused="2kbps" slamap="0x0" metric="packetloss" msg="Health Check SLA status.
SLA failed due to being over the performance metric threshold."
l When SLA passes, SLA link status logs will be generated with interval sla-pass-log-period:
logid="0113022925" type="event" subtype="sdwan" level="information" vd="root"
logdesc="SDWAN SLA information" eventtype="SLA" healthcheck="test" slatargetid=1
interface="R150" status="up" latency="0.060" jitter="0.003" packetloss="0.000%"
inbandwidthavailable="0kbps" outbandwidthavailable="200.00Mbps"
bibandwidthavailable="200.00Mbps" inbandwidthused="1kbps" outbandwidthused="1kbps"
bibandwidthused="2kbps" slamap="0x1" msg="Health Check SLA status."
SD-WAN related diagnose commands
This topic lists the SD-WAN related diagnose commands and related output.
To check SD-WAN health-check status:
FGT # diagnose sys sdwan health-check

Health Check(server):
Seq(1 R150): state(alive), packet-loss(0.000%) latency(0.110), jitter(0.024) sla_map=0x0
FGT # diagnose sys sdwan health-check
Health Check(ping):
Seq(2 R160): state(dead), packet-loss(100.000%) sla_map=0x0
FGT # diagnose sys sdwan health-check google
Health Check(google):
To check SD-WAN member status:
l When SD-WAN load-balance mode is source-ip-based/source-dest-ip-based.

FGT # diagnose sys sdwan member
Member(1): interface: R150, gateway: 10.100.1.1 2000:10:100:1::1, priority: 0 1024,
weight: 0
weight: 0

Fortinet Inc.
SD-WAN
l When SD-WAN load-balance mode is weight-based.

weight: 33
Session count: 15
weight: 66
Session count: 1
l When SD-WAN load-balance mode is measured-volume-based.

l Both members are under volume and still have room:

weight: 33
Config volume ratio: 33, last reading: 218067B, volume room 33MB
weight: 66
l Some members are overloaded and some still have room:

weight: 0
Config volume ratio: 33, last reading: 1287767633B, overload volume 517MB
weight: 63
l When SD-WAN load balance mode is usage-based/spillover.

l When no spillover occurs:

weight: 255
Egress-spillover-threshold: 400kbit/s, ingress-spillover-threshold: 300kbit/s
Egress-overbps=0, ingress-overbps=0
weight: 254
l When member has reached limit and spillover occurs:

weight: 255
weight: 254
l You can also use the diagnose netlink dstmac list command to check if you are over the limit.
FGT # diagnose netlink dstmac list R150
dev=R150 mac=00:00:00:00:00:00 vwl rx_tcp_mss=0 tx_tcp_mss=0 egress_overspill_

Fortinet Inc.
SD-WAN
threshold=50000 egress_bytes=100982 egress_over_bps=1 ingress_overspill_

threshold=37500 ingress_bytes=40 ingress_over_bps=0 sampler_rate=0 vwl_zone_id=1
intf_qua=0
To check SD-WAN service rules status:
l Manual mode service rules.

FGT # diagnose sys sdwan service4
Members(2):
1: Seq_num(1 R150), alive, selected
2: Seq_num(2 R160), alive, selected
Dst address(1):
10.100.21.0-10.100.21.255
l Auto mode service rules.

Gen(1), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(auto), link-cost-factor(latency),
link-cost-threshold(10), heath-check(ping)
Members(2):
1: Seq_num(2 R160), alive, latency: 0.066, selected
2: Seq_num(1 R150), alive, latency: 0.093
Dst address(1):
10.100.21.0-10.100.21.255
l Priority mode service rules.

Gen(1), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(priority), link-cost-factor
(latency), link-cost-threshold(10), heath-check(ping)
Members(2):
Dst address(1):
10.100.21.0-10.100.21.255
l Load-balance mode service rules.

Gen(1), TOS(0x0/0x0), Protocol(0: 1->65535), Mode(load-balance hash-mode=round-robin)
Members(2):
1: Seq_num(1 R150), alive, sla(0x1), gid(2), num of pass(1), selected
2: Seq_num(2 R160), alive, sla(0x1), gid(2), num of pass(1), selected
Dst address(1):
10.100.21.0-10.100.21.255
l SLA mode service rules.

Members(2):
1: Seq_num(1 R150), alive, sla(0x1), gid(0), cfg_order(0), cost(0), selected

Fortinet Inc.
SD-WAN
2: Seq_num(2 R160), alive, sla(0x1), gid(0), cfg_order(1), cost(0), selected

Dst address(1):
10.100.21.0-10.100.21.255
To check interface logs from the past 15 minutes:
FGT (root) # diagnose sys sdwan intf-sla-log R150

Timestamp: Wed Apr 21 16:58:27 2021, used inbandwidth: 655bps, used outbandwidth:
81655306bps, used bibandwidth: 81655961bps, tx bys: 3413479982bytes, rx bytes: 207769bytes.
To check SLA logs in the past 10 minutes:
FGT (root) # diagnose sys sdwan sla-log ping 1

Timestamp: Wed Apr 21 17:10:11 2021, vdom root, health-check ping, interface: R150, status:
up, latency: 0.079, jitter: 0.023, packet loss: 0.000%.
To check Application Control used in SD-WAN and the matching IP addresses:
FGT # diagnose sys sdwan internet-service-app-ctrl-list

Gmail(15817 4294836957): 64.233.191.19 6 443 Thu Apr 22 10:10:34 2021
Gmail(15817 4294836957): 142.250.128.83 6 443 Thu Apr 22 10:06:47 2021
Facebook(15832 4294836806): 69.171.250.35 6 443 Thu Apr 22 10:12:00 2021
Amazon(16492 4294836342): 3.226.60.231 6 443 Thu Apr 22 10:10:57 2021
Amazon(16492 4294836342): 52.46.135.211 6 443 Thu Apr 22 10:10:58 2021
Amazon(16492 4294836342): 52.46.141.85 6 443 Thu Apr 22 10:10:58 2021
Amazon(16492 4294836342): 52.46.155.13 6 443 Thu Apr 22 10:10:58 2021
Amazon(16492 4294836342): 54.82.242.32 6 443 Thu Apr 22 10:10:59 2021
YouTube(31077 4294838537): 74.125.202.138 6 443 Thu Apr 22 10:06:51 2021
YouTube(31077 4294838537): 108.177.121.119 6 443 Thu Apr 22 10:08:24 2021
YouTube(31077 4294838537): 142.250.136.119 6 443 Thu Apr 22 10:02:02 2021
YouTube(31077 4294838537): 142.250.136.132 6 443 Thu Apr 22 10:08:16 2021
YouTube(31077 4294838537): 142.250.148.100 6 443 Thu Apr 22 10:07:28 2021

Fortinet Inc.
SD-WAN
YouTube(31077 4294838537): 142.250.148.132 6 443 Thu Apr 22 10:10:32 2021

YouTube(31077 4294838537): 172.253.119.91 6 443 Thu Apr 22 10:02:01 2021
YouTube(31077 4294838537): 184.150.64.211 6 443 Thu Apr 22 10:04:36 2021
YouTube(31077 4294838537): 184.150.168.175 6 443 Thu Apr 22 10:02:26 2021
YouTube(31077 4294838537): 184.150.168.211 6 443 Thu Apr 22 10:02:26 2021
YouTube(31077 4294838537): 184.150.186.141 6 443 Thu Apr 22 10:02:26 2021
YouTube(31077 4294838537): 209.85.145.190 6 443 Thu Apr 22 10:10:36 2021
YouTube(31077 4294838537): 209.85.200.132 6 443 Thu Apr 22 10:02:03 2021
To check the dynamic tunnel status:
# diagnose sys link-monitor interface <name> <name>_0
For example:
# diagnose sys link-monitor interface vd2-2
Interface(vd2-2): state(up, since Tue Jun 15 12:31:28 2021), bandwidth(up:1299bps,
down:0bps), session count(IPv4:2, IPv6:0), tx(2409919 bytes), rx(5292290 bytes), latency
(0.03), jitter(0.00), packet-loss(0.00).
# diagnose sys link-monitor interface vd2-2 vd2-2_0

Interface(vd2-2_0): state(up, since Tue Jun 15 15:21:52 2021), bandwidth(up:640bps,
down:0bps), session count(IPv4:0, IPv6:0), tx(102242 bytes), rx(16388 bytes), latency(0.03),
jitter(0.00), packet-loss(0.00).
To check BGP learned routes and determine if they are used in SD-WAN service:
FGT # get router info bgp network 10.100.11.0/24

10.100.1.1
Original VRF 0
20 10
10.100.1.1 from 10.100.1.1 (5.5.5.5)
Origin incomplete metric 0, route tag 15, localpref 100, valid, external, best
Community: 30:5
Original VRF 0
20 10
10.100.1.5 from 10.100.1.5 (6.6.6.6)
Origin incomplete metric 0, route tag 15, localpref 100, valid, external, best
Community: 30:5
FGT # diagnose sys sdwan route-tag-list
Route-tag: 15, address: v4(1), v6(0)Last write/now: 6543391 6566007
service(1), last read route-tag 15 at 6543420
Prefix(24): Address list(1):
10.100.11.0-10.100.11.255 oif: 50 48
FGT # diagnose firewall proute list
id=2133196801(0x7f260001) vwl_service=1(DataCenter) vwl_mbr_seq=1 2 dscp_tag=0xff 0xff

Fortinet Inc.
SD-WAN
flags=0x40 order-addr tos=0x00 tos_mask=0x00 protocol=0 sport=0-65535 iif=0 dport=1-65535

oif=48(R150) oif=50(R160)
destination(1): 10.100.11.0-10.100.11.255
Using SNMP to monitor health check
You can monitor SD-WAN health check related statistics using SNMP. The MIB file can be downloaded by going to
System > SNMP and clicking Download FortiGate MIB File.
The following OIDs can be monitored:
Name OID Description
fgVWLHealthCheckLinkNumber .1.3.6.1.4.1.12356.101.4.9.1 The number of health check links

in fgVWLHealthCheckLinkTable
fgVWLHealthCheckLinkTable .1.3.6.1.4.1.12356.101.4.9.2 SD-WAN health check statistics

table.
This table has a dependent
expansion relationship with
fgVdTable.Only health checks with
a configured member link are
present in this table.
fgVWLHealthCheckLinkTableEntry .1.3.6.1.4.1.12356.101.4.9.2.1 SD-WAN health check statistics on

a virtual domain.
fgVWLHealthCheckLinkID .1.3.6.1.4.1.12356.101.4.9.2.1.1 SD-WAN health check link ID.

Only health checks with configured
member link are present in this
table. Virtual-wan-link health check
link IDs are only unique within a
virtual domain.
fgVWLHealthCheckLinkName .1.3.6.1.4.1.12356.101.4.9.2.1.2 Health check name.
fgVWLHealthCheckLinkSeq .1.3.6.1.4.1.12356.101.4.9.2.1.3 SD-WAN member link sequence.
fgVWLHealthCheckLinkState .1.3.6.1.4.1.12356.101.4.9.2.1.4 Health check state on a specific

member link.
fgVWLHealthCheckLinkLatency .1.3.6.1.4.1.12356.101.4.9.2.1.5 The average latency of a health

check on a specific member link
within last 30 probes, in float
number.
fgVWLHealthCheckLinkJitter .1.3.6.1.4.1.12356.101.4.9.2.1.6 The average jitter of a health check

on a specific member link within
last 30 probes, in float number.

Fortinet Inc.
SD-WAN
Name OID Description
fgVWLHealthCheckLinkPacketSend .1.3.6.1.4.1.12356.101.4.9.2.1.7 The total number of packets sent

by a health check on a specific
member link.
fgVWLHealthCheckLinkPacketRecv .1.3.6.1.4.1.12356.101.4.9.2.1.8 The total number of packets

received by a health check on a
specific member link.
fgVWLHealthCheckLinkPacketLoss .1.3.6.1.4.1.12356.101.4.9.2.1.9 The packet loss percentage of a

health check on a specific member
link within last 30 probes, in float
number.
fgVWLHealthCheckLinkVdom .1.3.6.1.4.1.12356.101.4.9.2.1.10 The VDOM that the link monitor

entry exists in.
This name corresponds to the
fgVdEntName used in fgVdTable.
fgVWLHealthCheckLinkBandwidthIn .1.3.6.1.4.1.12356.101.4.9.2.1.11 The available bandwidth of

incoming traffic detected by a
link, in Mbps,
fgVWLHealthCheckLinkBandwidthOut .1.3.6.1.4.1.12356.101.4.9.2.1.12 The available bandwidth of

outgoing traffic detected by a
link, in Mbps.
fgVWLHealthCheckLinkBandwidthBi .1.3.6.1.4.1.12356.101.4.9.2.1.13 The available bandwidth of bi-

direction traffic detected by a
link, in Mbps.
fgVWLHealthCheckLinkIfName .1.3.6.1.4.1.12356.101.4.9.2.1.14 SD-WAN member interface name.
Example
This example shows a SD-WAN health check configuration and its collected statistics.
To configure the SD-WAN health check:
config system sdwan

set status enable
config zone
next
end
config members
edit 1

Fortinet Inc.
SD-WAN
next
edit 2
set interface "MPLS"
set zone "SD-Zone2"
set cost 20
next
edit 3
next
end
config health-check
edit "pingserver"
set sla-fail-log-period 10
set sla-pass-log-period 20
set members 2 1 3
config sla
edit 1
set link-cost-factor jitter packet-loss
next
end
next
end
end
The collected statistics:
fgVWLHealthCheckLinkID .1.3.6.1.4.1.12356.101.4.9.2.1.1 1 2 3
fgVWLHealthCheckLinkName .1.3.6.1.4.1.12356.101.4.9.2.1.2 pingserver pingserver pingserver
fgVWLHealthCheckLinkSeq .1.3.6.1.4.1.12356.101.4.9.2.1.3 2 1 3
fgVWLHealthCheckLinkState .1.3.6.1.4.1.12356.101.4.9.2.1.4 0 0 0
fgVWLHealthCheckLinkLatency .1.3.6.1.4.1.12356.101.4.9.2.1.5 39.302 43.124 44.348
fgVWLHealthCheckLinkJitter .1.3.6.1.4.1.12356.101.4.9.2.1.6 4.346 3.951 5.05
fgVWLHealthCheckLinkPacketSend .1.3.6.1.4.1.12356.101.4.9.2.1.7 3657689 3657689 3657689
fgVWLHealthCheckLinkPacketRecv .1.3.6.1.4.1.12356.101.4.9.2.1.8 3196258 3220258 3219466
fgVWLHealthCheckLinkPacketLoss .1.3.6.1.4.1.12356.101.4.9.2.1.9 0 0 0
fgVWLHealthCheckLinkVdom .1.3.6.1.4.1.12356.101.4.9.2.1.1 root root root

0
fgVWLHealthCheckLinkBandwidthIn .1.3.6.1.4.1.12356.101.4.9.2.1.1 9999963 9999937 9999999

1
fgVWLHealthCheckLinkBandwidthO .1.3.6.1.4.1.12356.101.4.9.2.1.1 9999981 9999953 9999998

ut 2
fgVWLHealthCheckLinkBandwidthBi .1.3.6.1.4.1.12356.101.4.9.2.1.1 19999944 19999890 19999997

3

Fortinet Inc.
SD-WAN
fgVWLHealthCheckLinkIfName .1.3.6.1.4.1.12356.101.4.9.2.1.1 MPLS port1 port2

4

Fortinet Inc.
Zero Trust Network Access
This section includes information about ZTNA related new features:

l Zero Trust Network Access introduction on page 1189
l Basic ZTNA configuration on page 1192
l Establish device identity and trust context with FortiClient EMS on page 1203
l SSL certificate based authentication on page 1211
l Full versus simple ZTNA policies on page 1213
l ZTNA advanced configurations on page 1219
l ZTNA configuration examples on page 1232
l ZTNA troubleshooting and debugging commands on page 1314
l ZTNA troubleshooting scenarios on page 1319
This feature is not supported on FortiGate models with 2 GB RAM or less. See Proxy-related features not supported on
FortiGate 2 GB RAM models on page 101 for more information.
Zero Trust Network Access introduction
Zero Trust Network Access (ZTNA) is an access control method that uses client device identification, authentication, and
security posture tags (formerly ZTNA tags) to provide role-based application access. It gives administrators the flexibility
to manage network access for On-net local users and Off-net remote users. Access to applications is granted only after
device verification, authenticating the user’s identity, authorizing the user, and then performing context based posture
checks using security posture tags.
Traditionally, a user and a device have different sets of rules for on-net access and off-net VPN access to company
resources. With a distributed workforce and access that spans company networks, data centers, and cloud, managing
the rules can become complex. User experience is also affected when multiple VPNs are needed to get to various
resources. ZTNA can improve this experience.
ZTNA application gateway and IP/MAC based access control
l ZTNA application gateway allows users to securely access resources through an SSL encrypted access proxy. This
simplifies remote access by eliminating the use of VPNs.
l IP/MAC based access control combines IP/MAC with security posture tags for identification and security posture
check to implement role-based zero trust access.

Fortinet Inc.
ZTNA telemetry, tags, and policy enforcement
When On-net and Off-net FortiClient endpoints register to FortiClient EMS, device information, log on user information,
and security posture are all shared over ZTNA telemetry with the EMS server. Clients also make a certificate signing
request to obtain a client certificate from the EMS that is acting as the ZTNA Certificate Authority (CA).
Based on the client information, EMS applies matching Zero Trust tagging rules to tag the clients. These tags, and the
client certificate information, are synchronized with the FortiGate in real-time. This allows the FortiGate to verify the
client's identity using the client certificate, and grant access based on the security posture tags applied in the ZTNA
policy.
For more information, see Establish device identity and trust context with FortiClient EMS on page 1203.
Application gateway
The FortiGate application gateway can proxy HTTP, SSH, RDP, SMB, FTP, and other TCP traffic over secure
connections with the client. This enables seamless access from the client to the protected servers, without needing to
form IPsec or SSL VPN tunnels.

Fortinet Inc.
HTTPS access proxy
The FortiGate HTTPS access proxy works as a reverse proxy for the HTTP server. When a client connects to a webpage
hosted by the protected server, the address resolves to the FortiGate’s access proxy VIP. The FortiGate proxies the
connection and takes steps to authenticate the user. It prompts the user for their certificate on the browser, and verifies
this against the ZTNA endpoint record that is synchronized from the EMS. If an authentication scheme, such as SAML
authentication, is configured, the client is redirected to a captive portal for sign-on. If this passes, traffic is allowed based
on ZTNA policies, and the FortiGate returns the webpage to the client.
For example configurations, see ZTNA HTTPS access proxy example on page 1232, ZTNA HTTPS access proxy with
basic authentication example on page 1243, and ZTNA application gateway with SAML authentication example on page
1267.
TCP forwarding access proxy (TFAP)
The TCP forwarding access proxy works as a special type of HTTPS reverse proxy. Instead of proxying traffic to a web
server, TCP traffic is tunneled between the client and the access proxy over HTTPS, and forwarded to the protected
resource. The FortiClient on the remote endpoint intercepts traffic destined for the protected resources and routes them
to the FortiGate proxy gateway. An HTTPS connection is made to the FortiGate’s access proxy VIP, where the client
certificate is verified and access is granted based on the ZTNA policies. TCP traffic is forwarded from the FortiGate to the
protected resource, and an end to end connection is established. To reduce overhead, you can disable access proxy
encryption on the client, as some TCP protocols, like RDP, are already secure. The TCP forwarding access proxy
supports UTM scanning and deep inspection for HTTP, HTTPS, SMTP, SMTPS, IMAP, IMAPS, POP3, POP3S, SMB,
and CIFS.
For an example configuration, see ZTNA TCP forwarding access proxy example on page 1250.
SSH access proxy
The SSH access proxy provides some benefits to proxying SSH connections over TFAP, including allowing SSH deep
inspection, performing optional SSH host-key validation, and allowing one-time user authentication to authenticate the
ZTNA SSH access proxy connection and SSH server connection.
For an example configuration, see ZTNA SSH access proxy example on page 1260.
Basic ZTNA configuration components
The basic components required to configure ZTNA application gateway on the FortiGate are:
1. FortiClient EMS fabric connector and security posture tags.
2. FortiClient EMS running version 7.0.0 or later or FortiClient EMS Cloud.
3. FortiClient running 7.0.0 or later.
4. ZTNA server
5. ZTNA policy
6. (Optional) User authentication
7. (Optional) HA configurations
For configuration details, see Basic ZTNA configuration on page 1192.

Fortinet Inc.
Basic ZTNA configuration
To deploy a ZTNA application gateway, configure the following components on the FortiGate:
1. Configure a FortiClient EMS connector on page 1192
2. Configure a ZTNA server on page 1194
3. Configure a ZTNA policy on page 1197
4. Optional authentication on page 1201
5. Optional HA configurations on page 1203
To configure ZTNA in the GUI, go to System > Feature Visibility and enable Zero Trust
Network Access.
Configure a FortiClient EMS connector
To add an on-premise FortiClient EMS server in the GUI:
1. Go to Security Fabric > Fabric Connectors and double-click the FortiClient EMS card.
3. Enter a name for the connector and the IP address or FQDN of the EMS.
4. Click OK.
5. A window appears to verify the EMS server certificate. Click Accept.
See FortiClient EMS for more information.
To add an on-premise FortiClient EMS server in the CLI:

edit <ems-id>
set server <server IP or domain>
next
end
To add FortiClient EMS Cloud in the GUI:
1. Go to Security Fabric > Fabric Connectors and double-click the FortiClient EMS card.
3. Set the Type to FortiClient EMS Cloud.
4. Enter a name for the connector.
5. Click OK. A window appears to verify the EMS server certificate.

Fortinet Inc.
6. Click Accept.
See FortiClient EMS for more information.
To add FortiClient EMS Cloud in the CLI:

edit <ems-id>
set fortinetone-cloud-authentication enable
set certificate <string>
next
end
Security posture tags
After the FortiGate connects to the FortiClient EMS, it automatically synchronizes security posture tags (formerly
ZTNA tags). Security posture tags are generated from tagging rules configured on the FortiClient EMS. These tagging
rules are based on various posture checks that can be applied on the endpoints. See Endpoint Posture Check
Reference.
To view the synchronized security posture tags in the GUI:
1. Go to Policy & Objects > ZTNA and select the Security Posture Tags tab.
2. Hover the cursor over a tag name to view more information about the tag, such as its resolved addresses.
To create a security posture tag group in the GUI:
1. Go to Policy & Objects > ZTNA and select the Security Posture Tag Group tab.
3. Enter a name for the group and select the group members.

Fortinet Inc.
4. Click OK.
To view the synchronized security posture tags in the CLI:
# diagnose firewall dynamic address

# diagnose firewall dynamic list
To create a security posture tag group in the CLI:

edit <group name>
set category ztna-ems-tag
set member <members>
next
end
Configure a ZTNA server
To configure a ZTNA server, define the access proxy VIP and the real servers that clients will connect to. The access
proxy VIP is the FortiGate ZTNA gateway that clients make HTTPS connections to. The service/server mappings define
the virtual host matching rules and the real server mappings of the HTTPS requests.
To create a ZTNA server for HTTPS access proxy in the GUI:
1. Go to Policy & Objects > ZTNA and select the ZTNA Servers tab.
3. Enter a name for the server.
4. Select an Interface. The IP address and Port fields are automatically filled in based on the interface selection.
Verify that the IP address and port do not conflict with management access to the
interface. Otherwise, change the IP address to another address on that subnet.

Fortinet Inc.
You may specify the IP address as 0.0.0.0 if an interface other than Any is selected. The
ZTNA Application Gateway IP address will dynamically use the primary and secondary
IPv4 address of the interface as its external IP address.
5. Select the Default certificate. Clients will be presented with this certificate when they connect to the access proxy
VIP.
6. Add a server mapping:

a. In the Service/server mapping table, click Create New.
b. Set Service to HTTPS.
c. Set Virtual Host to Any Host or Specify.
l Any Host: Any request that resolves to the access proxy VIP will be mapped to your real servers. For
example, if both www.example1.com and www.example2.com resolve to the VIP, then both requests are
mapped to your real servers.
l Specify: Enter the name or IP address of the host that the request must match. For example, if
www.example1.com is entered as the host, then only requests to www.example1.com will match.
d. Configure the path as needed.
The path can be matched by substring, wildcard, or regular expression. For example, if the virtual host is
specified as www.example1.com, and the path substring is map1, then www.example1/map1 will be matched.

Fortinet Inc.
e. In the Server section, enter the server IP address and port number.
f. Click OK.
7. Click OK.
8. Use the CLI to add additional servers and mappings to the ZTNA server.
After additional servers and mappings are added, the Load balancing option is visible in the GUI.
config firewall access-proxy
edit "ZTNA-Server"
config api-gateway
edit 1
config realservers
edit 0
set ip <real server 2 address>
set port <real server 2 port>
next
end
next
end
next
end
To create a ZTNA server and access proxy VIP in the CLI:
1. Configure an access proxy VIP:

config firewall vip
edit <name>
set type access-proxy
set extip <external IP>

Fortinet Inc.
set extintf <external interface>

set server-type {https | ssh}
set extport <external port>
set ssl-certificate <certificate>
next
end
2. If the virtual host is specified, configure the virtual host:

config firewall access-proxy-virtual-host
edit <auto generated when configured from GUI>
set ssl-certificate <certificate>
set host <host name or IP>
set host-type {sub-string | wildcard}
next
end
3. Configure the server and path mapping:

config firewall access-proxy
edit <name>
set vip <vip name>
set client-cert {enable | disable}
set empty-cert-action {accept | block}
set log-blocked-traffic {enable | disable}
config api-gateway
edit 1
set url-map <mapped path>
set service {http | https | tcp-forwarding | samlsp}
set virtual-host <name of virtual-host if specified>
set url-map-type {sub-string | wildcard | regex}
config realservers
edit 1
set addr-type ip
set ip <ip of real server>
set port <port>
set status {active | standby | disable}
set health-check {enable | disable}
set translate-host {enable | disable}
next
end
set ldb-method static
set persistence none
set ssl-dh-bits 2048
set ssl-algorithm high
set ssl-min-version tls-1.1
set ssl-max-version tls-1.3
next
end
next
end
Configure a ZTNA policy
A ZTNA policy is used to enforce zero trust role based access control by using security posture tags or tag groups to
verify a device’s security posture. A ZTNA policy can also utilize security profiles to protect this traffic.

Fortinet Inc.
In earlier versions, ZTNA rules were special proxy policies that controlled access to the ZTNA
servers, and they could be configured from the Policy & Objects > ZTNA > ZTNA Rules tab.
However, on this version and above, these special proxy policies are now configured from
Policy & Objects > Proxy Policy page.
There are two ways to configure ZTNA rules in the GUI by using a full or simple ZTNA policy:
l Full ZTNA policy: The legacy method for configuring access-proxy policies.
l Simple ZTNA policy: A simplified method for configuring a ZTNA policy using firewall policies. This method covers
most functionality of a Full ZTNA policy, except it cannot control access based on destination interface or real
server’s destination address.
To configure a simple ZTNA policy in the GUI:
1. Go to Policy & Objects > Firewall Policy.

3. Enter a name for the ZTNA policy.
4. Set Type to ZTNA.
5. Select an Incoming Interface and Source.
6. Add the security posture tags or tag groups that are allowed access. If multiple tags are included, the matching
method is set to Any by default.
7. Select the ZTNA Server.
8. Select the Action.
9. Configure the remaining options as needed.

10. Click OK.

Fortinet Inc.
To configure a simple ZTNA policy in the CLI:

edit 1
set name <ZTNA policy name>
set srcintf <interface>
set dstintf "any"
set action accept
set srcaddr "all"
set dstaddr <access proxy vip>
set ztna-ems-tag <security posture tags>
set nat enable
next
end
The dstintf command cannot be modified.
To configure a full ZTNA policy in the GUI:
1. Go to System > Feature Visibility.

2. Under Security Features, enabled Explicit Proxy.
3. Click Apply.
4. Go to Policy & Objects > Proxy Policy.
6. Enter a name for the rule.
7. Select an Incoming Interface and Source.
8. Add the security posture tags or tag groups that are allowed access. If multiple tags are included, the matching
method is set to Any by default.
9. Select the Destination.
10. Select the ZTNA Server.
11. Select the Action.

Fortinet Inc.
12. Configure the remaining options as needed.

13. Click OK.
To configure a full ZTNA policy in the CLI:
config firewall proxy-policy

edit 1
set name <ZTNA policy name>
set proxy access-proxy
set access-proxy <access proxy>
set srcintf <interface>
set srcaddr "all"
set transparent {enable | disable}
set dstaddr "all"
set ztna-ems-tag <security posture tag(s)>
set ztna-tags-match-logic {or | and}
set action accept
set logtraffic all
set poolname <ip_pool>
set ssl-ssh-profile <inspection profile>
next
end
The transparent and poolname settings cannot be enabled at the same time. Use one
setting at a time when configuring ZTNA policies.

Fortinet Inc.

FortiOS-7.4.5-Administration - Guide (2) - Parte4

Uploaded by

Document Informationclick to expand document informationFortiOS-7.4.5-Administration_Guide (2)_Parte4

Document Informationclick to expand document information

Copyright:

Available Formats

FortiOS-7.4.5-Administration - Guide (2) - Parte4

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

FortiOS-7.4.5-Administration - Guide (2) - Parte4

Uploaded by

Copyright:

Available Formats

SD-WAN

To verify applications identified by Application Control in SD-WAN:

# diagnose sys sdwan internet-service-app-ctrl-list

Steam(16518 4294838108): 23.6.148.10 6 443 Thu Apr 15 08:51:54 2021

DSCP tag-based traffic steering in SD-WAN

FortiOS 7.4.5 Administration Guide 901

Name SD-WAN-OUT Overlay-OUT

FortiOS 7.4.5 Administration Guide 902

From port3 port3

Source all all

Destination all all

Schedule always always

Service all all

Action Accept Accept

NAT enabled enabled

Configuring SD-WAN rules

VoIP traffic is steered to the Overlay zone.

config sys sdwan

FortiOS 7.4.5 Administration Guide 903

set tos 0x70

Social media traffic

Social media traffic is steered to the virtual-wan-link zone.

config system sdwan

Other web traffic

Other web traffic is steered to the virtual-wan-link zone.

config system sdwan

FortiOS 7.4.5 Administration Guide 904

set name "All-traffic"

Verifying the DSCP tagged traffic on FortiGate

VoIP traffic marked with DSCP tag 0x70:

FortiOS 7.4.5 Administration Guide 905

Web traffic marked with DSCP tag 0x30:

Verifying the service rules

# diagnose sys sdwan service4

Service(5): Address Mode(IPV4) flags=0x0

Service(3): Address Mode(IPV4) flags=0x0

Service(2): Address Mode(IPV4) flags=0x0

FortiOS 7.4.5 Administration Guide 906

Verifying traffic steering on the SD-WAN rules

Verifying that steered traffic is leaving from the expected interface

VoIP traffic is expected to leave on the preferred VPN_B_Tunnel (Branch-HQ-B) interface:

FortiOS 7.4.5 Administration Guide 907

ECMP support for the longest match in SD-WAN rule matching

To configure the SD-WAN:

config system sdwan

FortiOS 7.4.5 Administration Guide 908

To check the results:

Service(1): Address Mode(IPV4) flags=0x200 use-shortcut-sla

FortiOS 7.4.5 Administration Guide 909

4. On FGT_A, sniff for traffic sent to PC_2:

Override quality comparisons in SD-WAN longest match rule matching

FortiOS 7.4.5 Administration Guide 910

To configure the SD-WAN:

config system sdwan

FortiOS 7.4.5 Administration Guide 911

To check the results:

FortiOS 7.4.5 Administration Guide 912