<< Back to Soekris net4501 FreeBSD Router Project Page

How to Aggregate the Bandwidth of

Multiple Internet Connections using
FreeBSD 4.x
By Michael R. Brumm

Introduction

Often an organization will require more bandwidth than a single DSL or cable Internet
connection can provide, but higher bandwidth connections (T1 or T3) are beyond the
reach of their budget. An alternative (and much less costly) solution is to combine
multiple slower Internet connections using some form of bandwidth aggregation. This is
often called "load balancing" or "site multihoming", and the specific type of aggregation I
describe in this tutorial is often called "teaming" or "connection teaming".

Another possible benefit of bandwidth aggregation might be increased reliability.

Because multiple separate connections are used, if one of them becomes unavailable, the
others can takeover (at an obvious cost to overall bandwidth).

Ideally bandwidth aggregation would be supported by your ISP, and performed at the link
layer (using multilink PPP) or at least with BGP routing support. However, most ISPs are
selling their lines with only the most basic of services, so customers are left with basically
only one choice: connection teaming using network address translation (NAT).

Here are the project requirements:

 Allow NAT clients to share bandwidth from multiple Internet connections with
dynamic IP addresses (cheap DSL, cable, or other Internet service) and no
upstream routing support from the ISPs
 FreeBSD 4.x
 ipf for packet filtering
 ipnat for network address and port translation
 ipfw for traffic shaping
 No natd (ipnat provides vastly better performance)
 Minimize disruption to NAT clients if one or more Internet connections are
unavailable
  Use only a single Ethernet port on the router for all the Internet connections

It might be hard to believe, but all of these things are achievable using FreeBSD 4.x and
no other additional software. You just have to learn how to make all its various pieces
and parts work smoothly together.

Newcomers to FreeBSD might be wondering where I found a bandwidth aggregator in

the packages library (nope, there isn't one). Experienced users of FreeBSD will wonder
how I got ipnat to handle translation to multiple dynamic IP addresses on a single
Ethernet port. Hardcore users will want to know how I allowed the Internet connections
to be disconnected with minimal disruption to the NAT clients. Well, step right up ladies
and gentlemen. The show is about to start.

The Secrets to My Success

The first thing to recognize is that policy routing can be used together with network
address translation to handle bandwidth aggregation. Policy routing allows you to route
packets based on policies rather than route just based on the destination address and the
routing tables. This isn't a tutorial on policy routing, so if you are unfamiliar about policy
routing, read the FreeBSD man pages for ipf and ipfw. ipf uses the "fastroute" or "to"
command. ipfw uses the "fwd" or "forward" command.

The second thing to realize is that it is impossible to use ipnat's network address
translation with multiple external IP addresses on the same Ethernet interface. The ipnat
configuration doesn't even have a way to describe such a situation. However, you can
create multiple virtual Ethernet interfaces on the same Ethernet port, and have ipnat use
each interface independently.

Also, keep in mind that dissimilar Internet services use dissimilar authentication and
connection protocols. Many DSL connections use PPPoE, some cable ISPs authenticate
using a MAC address, and a few of the cheapest ISPs require you to use their
router/modems. Supporting all these different environments on the FreeBSD router can
be done, however it would also make this tutorial unnecessarily complicated. Instead, I'm
going to use the ISP supplied NAT router/modem and perform a second network address
translation.

Topology

In this tutorial, I've configured the ISP supplied routers to translate class C (/24) subnets
within the unroutable 192.168.0.0/16 subnet. For the clients, I've used a class B subnet
(10.0.0.0/16) in the unroutable 10.0.0.0/8 subnet. This means the FreeBSD router will
translate the NAT clients in 10.0.0.0/16 to an IP address on one of its virtual Ethernet
interfaces (each of which have an IP address within 192.168.0.0/16). The corresponding
ISP supplied NAT router will then translate the 192.168.0.0/16 address into its public
dynamic IP address.

Many organizations need at least one static IP address for providing services, so I've also
included a static public IP address on sis0 (12.34.56.7). This is also configured as a
backup for the NAT clients in case the connections on sis1 are down.

Of course, a simple diagram is worth a thousand words:

Requirements

Your kernel must have been built with support for ipf, ipfw, ipnat, dummynet, and
netgraph. This includes (but is not limited to the following kernel options):

options IPFIREWALL
options IPFIREWALL_FORWARD
 options IPFILTER
options DUMMYNET
options NETGRAPH

In addition, you will have to build and install the following kernel module: ng_eiface.ko
(not a kernel option as far as I know).

Finally, you will have to fix the order in which ipf and ipfw handle packets by following
this tutorial: How to Reorder the FreeBSD 4.x ipf and ipfw Packet Filters

Create Multiple Virtual Ethernet Interfaces

We need to create and configure virtual ethernet interfaces on boot. To have the
interfaces automatically created, create a file named /etc/start_if.sis1 and add this:

ngctl mkpeer . eiface hook ether

ngctl mkpeer . eiface hook ether
ifconfig ngeth0 link 00:5c:12:34:56:78
ifconfig ngeth1 link 00:5c:12:34:56:79
ifconfig ngeth0 up
ifconfig ngeth1 up

To enable bridging on these two new interfaces, install the following script in
/usr/local/etc/rc.d: ether.bridge.0.sh

Also, configure the interfaces (and a few other things) in /etc/rc.conf with these lines:

ipfilter_enable="YES"
firewall_enable="YES"
ipnat_enable="YES"

ifconfig_sis0="inet 12.34.56.7/24"
ifconfig_ngeth0="inet 192.168.0.100/24"
ifconfig_ngeth1="inet 192.168.1.100/24"
ifconfig_sis2="inet 10.0.0.1/24"
 gateway_enabled="YES"
defaultroute="12.34.56.1"

firewall_type="/etc/ipfw.rules"

Configure ipnat to Perform Network Address Translation

To allow network address translation on all the Internet connections, add these lines to
/etc/ipnat.rules:

map sis0 10.0.0.0/24 -> 12.34.56.7/32 proxy port ftp ftp/tcp mssclamp 1360
map sis0 10.0.0.0/24 -> 12.34.56.7/32 portmap tcp/udp auto mssclamp 1360
map sis0 10.0.0.0/24 -> 12.34.56.7/32 mssclamp 1360

map ngeth0 10.0.0.0/24 -> 192.168.0.100/32 proxy port ftp ftp/tcp mssclamp
1360
map ngeth0 10.0.0.0/24 -> 192.168.0.100/32 portmap tcp/udp auto mssclamp
1360
map ngeth0 10.0.0.0/24 -> 192.168.0.100/32 mssclamp 1360

map ngeth1 10.0.0.0/24 -> 192.168.1.100/32 proxy port ftp ftp/tcp mssclamp
1360
map ngeth1 10.0.0.0/24 -> 192.168.1.100/32 portmap tcp/udp auto mssclamp
1360
map ngeth1 10.0.0.0/24 -> 192.168.1.100/32 mssclamp 1360

Configure ipfw with Policy Routing

Policy routing is used to balance the NAT client requests between the Internet
connections. The contents of /etc/ipfw.rules:

flush
# ipfw is used only for traffic shaping (ipf is used for filtering)

# Everything gets passed by default

add 65000 pass all from any to any

# None of the checks below are for these interfaces

 add 100 pass all from any to any via lo0
add 120 pass all from any to any via ngeth0
add 130 pass all from any to any via ngeth1

# Policy routing
add 500 forward 192.168.0.1 all from 192.168.0.100 to any out xmit sis0
add 510 forward 192.168.1.1 all from 192.168.1.100 to any out xmit sis0

# None of the checks below are for these interfaces

add 700 pass all from any to any via sis0

# Incoming: coming from the Internet.

# Outgoing: going to the Internet.
#
# sis2 is the LAN side, so
# Incoming packets out xmit to sis2
# Outgoing packets in recv on sis2

# Load balance between the dynamic IP connections

# Use state for existing connections

add 1500 check-state

# Allows use of the regular routing when load balancing cannot be done
# rule 1799 reserved for check-loadbalance
add 1800 skipto 2000 all from any to any in recv sis2
add 1900 skipto 10000 all from any to any in recv sis2 keep-state

# HTTP/HTTPS balanced using probability

# rule 2000 reserved for check-loadbalance
# rule 2001 reserved for check-loadbalance
add 2100 prob 0.5 forward 192.168.0.1 tcp from any to any 80,443 in recv sis2
keep-state
add 2200 forward 192.168.1.1 tcp from any to any 80,443 in recv sis2 keep-state
add 2300 forward 192.168.0.1 tcp from any to any 80,443 in recv sis2 keep-state

# Other protocols balanced based on source IP address

# rule 5000 reserved for check-loadbalance
# rule 5001 reserved for check-loadbalance
add 5100 forward 192.168.0.1 all from 0.0.0.0:0.0.0.1 to any in recv sis2 keep-
state
add 5200 forward 192.168.1.1 all from 0.0.0.1:0.0.0.1 to any in recv sis2 keep-
state
add 5300 forward 192.168.0.1 all from any to any in recv sis2 keep-state
add 5400 forward 192.168.1.1 all from any to any in recv sis2 keep-state

The method of balancing based on services could be improved, but this gives you a good
general idea of how to balance (both based on probability and on odd/even IP addresses).
Stateless protocols like HTTP and HTTPS tend to be very forgiving about receiving the
same client requests from different IP addresses. Other protocols, like FTP, are not.

Additional Internet connections can be added, but if you want to balance them evenly, the
number of rules and probability values get complicated (especially with check-
loadbalance script enabling and disabling the rules ). Remember that the probabilities
must be in the series [1/n, 1/(n-1), 1/(n-2), ... , 1]. For example, with four connections you
would want to use these lines:

add 2100 prob 0.25 forward ...

add 2200 prob 0.3333 forward ...
add 2300 prob 0.5 forward ...
add 2400 forward ...
...

Also, with four Internet connections, balancing with odd and even source IP addresses
won't work, but you can use mod 4 (least signifigant bits: 1 + 2) with these lines:

add 5100 forward 192.168.0.1 all from 0.0.0.0:0.0.0.3 ...

add 5200 forward 192.168.1.1 all from 0.0.0.1:0.0.0.3 ...
add 5300 forward 192.168.0.1 all from 0.0.0.2:0.0.0.3 ...
add 5400 forward 192.168.1.1 all from 0.0.0.3:0.0.0.3 ...
...

Script Watchdog for Internet Connection Outages

Now that you've got your clients connection teaming these multiple Internet connections,
you'll want to examine reliability. Imagine one of the cheap Internet connections become
unavailable. You'll get some unusual problems on your NAT clients: half of all
HTTP/HTTPS connections will fail, and half of all NAT clients won't be able to use other
protocols at all.

This would obviously not be acceptable. To handle this problem, I've scheduled a script
in the crontab to watch these connections every few minutes and modify the policy routes
when a connection becomes unavailable: check-loadbalance.sh

Change the variable definitions DST1_IP and DST2_IP to suit your situation. Ideally they
should be ping-able IP addresses within the respective ISP's (like routers one hop away,
or the ISP's DNS servers). They could be the same destination, if necessary.

This script is designed only for checking two Internet connections, but a similar idea
could be applied to handle more connections. Try unplugging one of the Internet
connections and running the script from the console to test it before scheduling it in the
 crontab.

Conclusion

As you can see, even without installing any additional software, FreeBSD can be a very
powerful and flexible operating system for routing. It is always a surprise what you can
do with it if you think outside the box.

<< Back to Soekris net4501 FreeBSD Router Project Page

All Material Copyright © Michael R. Brumm