Parsing the most democratic of SD-WAN: architecture, configuration, administration and pitfalls

Judging by the number of questions that began to arrive to us via SD-WAN, the technology began to take root in Russia. Vendors, of course, are not asleep and offer their concepts, and some daredevil pioneers are already implementing them on their networks.



We work with almost all vendors, and over the years in our lab, I've had time to dig into the architecture of every major software-defined solution developer. Fortinet's SD-WAN is a bit apart here, which simply built the functionality of balancing traffic between communication channels into the firewall software. The solution is rather democratic, so it is usually looked at in companies where they are not yet ready for global changes, but want to use their communication channels more effectively.



In this article, I want to tell you how to configure and work with Fortinet SD-WAN, who is the right solution for this solution, and what pitfalls you can face here.



The most notable players in the SD-WAN market can be classified into one of two types:



1. Startups that created SD-WAN solutions from scratch. The most successful of these receive a huge impetus to development when bought by large companies - this is the history of Cisco / Viptela, VMWare / VeloCloud, Nuage / Nokia



2. Large network vendors who have created SD-WAN solutions, developing the programmability and manageability of their traditional routers are history Juniper, Huawei



Fortinet has managed to find its way. The software of firewalls has built-in functionality that allows combining their interfaces into virtual channels and balancing the load between them using complex algorithms compared to the usual routing. This functionality is called SD-WAN. Could what Fortinet did be called SD-WAN? The market is gradually developing an understanding that Software-Defined means the separation of Control Plane from Data Plane, dedicated controllers, orchestrators. Fortinet doesn't have anything like that. Centralized management is optional and is offered using the traditional Fortimanager tool. But in my opinion, one should not look for abstract truth and waste time arguing about terms. In the real world, any approach has its merits and demerits. The best way out is to understand them and be able to choose solutions that match the tasks.



I'll try to tell you, with screenshots in hand, what SD-WAN from Fortinet looks like and can do.

How it works

Suppose you have two branches connected by two data links. These data transmission channels are grouped in the same way as ordinary Ethernet interfaces are grouped in the LACP-Port-Channel. Old-timers will remember PPP Multilink, which is also a suitable analogy. Channels can be physical ports, VLAN SVIs, VPNs or GRE tunnels.



VPN or GRE is typically used when connecting branch office LANs over the Internet. And physical ports - if there are L2 connections between sites, or when connecting over a dedicated MPLS / VPN, if we are satisfied with the connection without Overlay and encryption. Another scenario in which physical ports are used in the SD-WAN group is balancing the local access of users to the Internet.



We have four firewalls and two VPN tunnels at our booth, working through two "telecom operators". The scheme looks like this:







VPN tunnels are configured in front-end mode, so that it looks like Point-to-Point connections between devices with IP addresses on P2P interfaces that can be pinged and made sure that the connection is through a specific the tunnel is working. For the traffic to be encrypted and go to the opposite side, it is enough to route it into the tunnel. The alternative - selecting traffic for encryption using subnet lists, greatly confuses the administrator as the configuration becomes more complex. In a large network, to build a VPN, you can use ADVPN technology, this is an analogue of DMVPN from Cisco or DVPN from Huawei, which simplifies the configuration.





I present the config in text form, because, in my opinion, it is more convenient to configure VPN this way. Almost all settings are the same on both sides; in text form, they can be made with copy-paste. If you do the same thing in the web interface, it is easy to make a mistake - to forget a checkmark somewhere, to enter the wrong value.



After we added interfaces to the bundle,







all routes and security policies can refer to it, and not to the interfaces included in it. At a minimum, traffic from internal networks to SD-WAN should be allowed. When you create rules for them, you can apply protective measures such as IPS, antivirus, and HTTPS disclosure.







SD-WAN Rules are configured for the bundle. These are the rules that determine the balancing algorithm for a specific traffic. They are similar to routing policies in Policy-Based Routing, only as a result of traffic falling under the policy, not next-hop or a regular outgoing interface is installed, but the interfaces added to the SD-WAN bundle plus the traffic balancing algorithm between these interfaces.



Traffic can be separated from the general stream by L3-L4 information, by recognized applications, Internet services (URL and IP), as well as by recognized users of workstations and laptops. After that, one of the following balancing algorithms can be assigned to dedicated traffic:







The Interface Preference list selects those interfaces from those already added to the bundle that will serve this type of traffic. By adding not all interfaces, you can restrict which channels you use, say, e-mail, if you do not want to load it with expensive channels with a high SLA. In FortiOS 6.4.1, it became possible to group interfaces added to the SD-WAN bundle into zones, creating, for example, one zone for communication with remote sites, and another for local Internet access using NAT. Yes, the traffic that goes to the regular Internet can also be balanced.

About balancing algorithms

In terms of how Fortigate (a firewall from Fortinet) is able to split traffic between channels, there are two interesting options that are not very common in the market:



Lowest Cost (SLA) - from all interfaces that satisfy the SLA at the moment, the one with the lower weight is chosen (cost) manually set by the administrator; this mode is suitable for "heavy" traffic such as backups and file transfers.



Best Quality (SLA) - this algorithm, in addition to the usual delay, jitter and packet loss Fortigate, can use the current channel load to assess the quality of channels; this mode is suitable for sensitive traffic such as VoIP and video conferencing.



These algorithms require setting up the performance SLA. This meter periodically (check interval) monitors SLA compliance information: packet loss, latency and jitter in the communication channel - and can “reject” those channels that currently do not meet the quality thresholds - are losing too many packets or giving too much delay. In addition, the meter monitors the status of the channel, and can temporarily remove it from the bundle in case of repeated loss of responses (failures before inactive). When restoring, after several successively received responses (restore link after), the meter will automatically return the channel to the bundle, and data will be transferred over it again.



This is how the "meter" setting looks like:







In the web interface, ICMP-Echo-request, HTTP-GET and DNS request are available as a test protocol. There are slightly more options on the command line: TCP-echo and UDP-echo options are available, as well as a specialized quality measurement protocol - TWAMP.







The measurement results can be seen in the web interface:







And in the command line:

Trouble shooting

If you have created a rule and everything does not work as expected, you should look at the Hit Count value in the SD-WAN Rules list. It will show whether the traffic falls into this rule at all:







On the meter's setup page, you can see the change in channel parameters over time. The dotted line indicates the threshold value of the parameter.In the







web interface, you can see how the traffic is distributed by the volume of transmitted / received data and the number of sessions:







In addition to all this, there is an excellent opportunity to track the passage of packets with maximum detail. When working in a real network, the device configuration accumulates a lot of routing policies, firewalling and traffic distribution over SD-WAN ports. All this interacts with each other in a complex way, and although the vendor provides detailed block diagrams of packet processing algorithms, it is very important to be able not to build and test theories, but to see where the traffic really goes.



For example, the following set of commands Will track two packets with a source address of 10.200.64.15 and a destination address of 10.1.7.2. Ping 10.7.1.2 from 10.200.64.15 twice and look at the output on the console. First package: Second package:



diagnose debug flow filter saddr 10.200.64.15

diagnose debug flow filter daddr 10.1.7.2

diagnose debug flow show function-name

diagnose debug enable

diagnose debug trace 2

























Here is the first packet received by the firewall: A new session was created for it: And a match was found in the settings of the routing policies It turns out that the packet needs to be sent to one of the VPN tunnels: An allow rule is found in the firewall policies: The packet is encrypted and sent to the VPN tunnel: Encrypted the packet is sent to the gateway address for this WAN interface: For the second packet, everything is the same, but it is sent to a different VPN tunnel and leaves through a different firewall port:

id=20085 trace_id=475 func=print_pkt_detail line=5605 msg="vd-Internet:0 received a packet(proto=1, 10.200.64.15:42->10.1.7.2:2048) from DMZ-Office. type=8, code=0, id=42, seq=0."

VDOM – Internet, Proto=1 (ICMP), DMZ-Office – L3-. Type=8 – Echo.





msg="allocate a new session-0006a627"





msg="Match policy routing id=2136539137: to 10.1.7.2 via ifindex-110"





"find a route: flag=04000000 gw-192.168.254.1 via DC-Ph1-1"





msg="Allowed by Policy-3:"





func=ipsecdev_hard_start_xmit line=789 msg="enter IPsec interface-DC-Ph1-1"

func=_ipsecdev_hard_start_xmit line=666 msg="IPsec tunnel-DC-Ph1-1"

func=esp_output4 line=905 msg="IPsec encrypt/auth"





msg="send to 2.2.2.2 via intf-WAN1"





func=ipsecdev_hard_start_xmit line=789 msg="enter IPsec interface-DC-Ph1-2"

func=_ipsecdev_hard_start_xmit line=666 msg="IPsec tunnel-DC-Ph1-2"

func=esp_output4 line=905 msg="IPsec encrypt/auth"

func=ipsec_output_finish line=622 msg="send to 4.4.4.2 via intf-WAN2"

Pros of the solution

Reliable functionality and user-friendly interface. The feature set that was available in FortiOS before the advent of SD-WAN is completely preserved. That is, we do not have a newly developed software, but a mature system from a proven firewall vendor. With the traditional set of networking functions, a convenient and easy-to-learn web interface. How many SD-WAN vendors have, say, Remote-Access VPN functionality on their terminal devices?



Level 80 security. FortiGate is among the top firewall solutions. There are many materials on the Internet for configuring and administering firewalls, and there are many security specialists on the labor market who have already mastered the vendor's solutions.



Zero cost for SD-WAN functionality.It costs the same to build an SD-WAN on FortiGate as to build a regular WAN on it, since no additional licenses are needed to implement SD-WAN functionality.



Low entry threshold for the price. Fortigate has a good gradation of devices for different performance. The youngest and most inexpensive models are quite suitable for raising an office or point of sale for, say, 3-5 employees. Many vendors simply do not have such low-performance and affordable models.



High performance. The reduction of SD-WAN functionality to traffic balancing allowed the company to release a specialized SD-WAN ASIC, thanks to which the operation of SD-WAN does not reduce the performance of the firewall in general.



The ability to implement an entire office on Fortinet equipment.These are a couple of firewalls, switches, Wi-Fi access points. This office is easy and convenient to manage - switches and access points are registered on firewalls and controlled from them. For example, this is how a switch port may look from the firewall interface that controls this switch:







No controllers as a single point of failure. The vendor himself focuses on this, but this can be called a plus only partly, because those vendors who have controllers, ensuring their fault tolerance is inexpensive, most often at the price of a small amount of computing resources in a virtualization environment.

What to look for

Lack of separation of Control Plane and Data Plane . This means that the network must be configured either manually or using the existing traditional management tools - FortiManager. For vendors who have implemented such a separation, the network is assembled by itself. The administrator may only need to adjust its topology, prohibit something somewhere, nothing more. However, FortiManager's trump card is that it can manage not only firewalls, but also switches and Wi-Fi access points, that is, almost the entire network.



Conditional growth of controllability.Due to the fact that traditional tools are used to automate network configuration, network manageability with the introduction of SD-WAN does not increase significantly. On the other hand, new functionality becomes available faster, since the vendor first releases it only for the firewall operating system (which immediately makes it possible to use it), and only then supplements the control system with the necessary interfaces.



Some of the functionality may be available from the command line, but not available from the web interface. Sometimes it’s not so scary to go to the command line to configure something, as it’s scary not to see in the web interface that something has already been configured by someone from the command line. But this usually concerns the newest features and gradually, with FortiOS updates, the capabilities of the web interface are tightened.

Who is it for

For those who do not have many branches. Implementing an SD-WAN solution with complex central components on a network of 8-10 branches may not be worth the candle — you will have to spend money on licenses for SD-WAN devices and resources of the virtualization system to host central components. A small company usually has a shortage of free computing resources. In the case of Fortinet, you just need to buy firewalls.



Those who have a lot of small branches. For many vendors, the minimum price of the solution for a branch is quite high and may not be interesting from the point of view of the end customer's business. Fortinet offers small devices at a very attractive price.



Those who are not ready to step too far yet.Implementing SD-WAN with controllers, proprietary routing, and a new approach to network planning and management may be too big for some customers. Yes, such an implementation will ultimately help optimize the use of communication channels and the work of administrators, but first you have to learn a lot. For those who are not yet ready for the paradigm shift, but want to squeeze more out of their communication channels, the solution from Fortinet is just the right fit.