HamWAN Remote Site

It's become a need for HamWAN to expand over existing internet links, as a backup and in areas we cannot hit with radio. This has shown a need at some of our other radio sites across the state, and in many cases where we can't get a good internet connection unless via restrictive NAT.

Design Requirements:

• IPv4 and IPv6
• IPv4 DHCP and IPv6 SLAAC for clients
• Transparent routing over the underlay network (people shouldn't be able to tell it's a VPN)
• Traverse NAT, even NAT 4444!
• Local Managed Switch
• POE source on the switch
• Conserve IP space in the design
• Integrate to the existing HamWAN network
• Hub and spoke no spoke to spoke direct breakout or on demand tunneling (SDWAN)
• support for up to 10 remote locations

Thoughts on Hardware

Thought was given to this for hardware and in general we favor used routing equipment which is past it's useful life from eBay. This invariably means Cisco or lower end Juniper, but Cisco has the largest amount of gear out there.

We did configure, and deploy a network based on Mikrotik routers to test this on. While we found this would "work" it leaked information from the Mikrotik as it cannot do a VRF properly. We found a number of other issues, and I've documented some here. MT might work for you, if you're ok with it, and you can get new in the box replacements from Amazon Prime for $99.

For Cisco hardware we've settled on the Cisco 2921/51 for the Spoke routers and a 3945e for the HUB. These routers are capable of doing 300 mbit+ of traffic over the VPN, and support the routing protocols we require to do dual stack IPv4 and IPv6.

Hub

The hub will plug into our core Juniper in Tampa via a ptp interface. This will speak ISIS, our IGP of choice for IPv4 and IPv4, and let the rest of HamWAN know as the sites come online.

We made the decision to use a multi-point GRE tunnel interface and run NHRP for the remote links. This allows us to use a /28 on the Tunnel, and support up to 13 remote locations without re-configuring. If we needed more remote sites, we can renumber or just use IPv6 :)

The one disadvantage to running multi-point GRE is we cannot run ISIS directly as ISIS doesn't use IP but rather CLNS for a transport. This means for the Tunnel interface and remote spoke sites we'll run OSPFv3 in a dual stack configuration. From the perspective of the spoke, they will get a default route and "announce" their routes to the hub.

Interconnection with hamwan. We prefer the hub to speak ISIS to the core, and handle both address families in the same process. Our soultion to this is redistributing the learned OSPFv3 routes into the ISIS process on the Hub.

The 3945e router was chosen for the hub. The 3945e is a 3945 which had the SPE-250 processing card in it. Like all 29/3900 routers they support various service modules from ATM to Ethernet switching interfaces, and even server blades. With the right power supply the router will even support POE or POE+ depending on the switch module installed. There are several Licenses and RTU's used on this, but by default the 3945 supports SEC/K9 and will handle a hundred tunnels at 150 mbit/s of throughput. The router is able to support well over 1gb/s of throughput and up to 3000 tunnels if the HSEC/K9 license is added to it. This license is locked to the CPU and must be generated from Cisco. As we don't need much more performance here we will not be licensing this.

FYY these are all known and the ISR/G2 routers. The next generation is the 4000 and 4400 ISR routers, which support 3 gb/s+ of crypto. As of writing they are still quite pricey on the used market.

Spoke

Our spoke site router is designed to provide us a number of Ethernet ports which serve up access to 44 net and IPv6 directly at the remote site with minimal config. We also want to support local breakout via NAT if needed too.

For HamWAN we're not concerned with encryption, so we could build a GRE tunnel without IPSEC and assuming we have an unfiltered public IPv4 at the spoke site, it would work. This would avoid the limitations of the crypto license limits as well. GRE has no ability to traverse NAT as a UDP packet, and IPSEC handles this NAT traversal quite well. Now this doesn't fix remote sites where there is layer 7 firewalling, ALG's and the like.

For IPsec we've choses to use pre-shared keys and IKEv2 vs ISAKMP as IKEv2 supports NAT traversal as via standard encapsulation of the IPSEC as UDP port 4500. It also is better in terms of us running a well known IP listener from a service denial or DOS perspective.

At the spoke we'll have a routed subnet to a VLAN interface on the router. This will bridge into the switch module and the local router will runs DHCP to hand out IPv4 and SLACC for IPv6. As this routed subnet will burn 2 IP's the management interfaces on the switch and the router, a /29 will only provide space for 4 connected devices. This may be fine at some sites, but others will need a /28 or /27. The HUB router will learn of these subnets via OSPFv3.

The routing config will be a bit complex as we want any traffic into Ethernet to not go to the default routing table. This means a VRF (or separate routing table) is needed for these interfaces. On the Spoke a VRF, HamWAN is created and the Tunnel and Vlan interface are placed inside it. The OSPFv3 process must run inside this table as well as it must not leak any routes from or into the default table on the router since the default table is how the tunnel traverses the underlay network (internet).

The decision here was made to go with the Cisco 2921 or 2951 routers. There is not much perforamcne difference in these, but they are limited to 85mbit/s of IPSEC unless they have the HSEC/K9 license AND the ISM-VPN-29 crypto accelerator module. This is known as the "CISCO2951-HSEC+/K9" bundle. Also if you intend to run the POE switch module a special power supply "PWR-2921-51-POE" is required. This this supply supplies 48v in addition to the standard 12v and 5v voltages of the standard supply.

For the local switch breakout there are several options:

SM-X-ES3-24-P   - Based on a 3560X switch - "SM-X-ES3-24-P: EtherSwitch SM L3 + PoEPlus + MACSec + 24 10/100/1000"
SM-ES3G-24-P—24 - Based on a 3560e Switch - "SM-ES3G-24-P: EtherSwitch SM L3 + POE + 24 10/100/1000"
SM-ES3-24-P—23  - Based on the 3560 - SM-ES3G-24-P: EtherSwitch SM L2 + POE + 24 10/100/1000" 
SM-ES2-24-P     - L2 only 2960sm based - "SM-ES2-24-P: EtherSwitch SM L2 + PoE + 23 10/100 + 1 10/100/1000" 

There are other switch modules, but these are the most popular. In our case we're running the SM-ES2-24-P as we don't require layer 3 on the switch but do require POE. These are managed on their own IP and boot their own IOS. They have two virtual 1g interfaces which interconnect with the router via the backplane and trunk to the switch. This way a Vlan73 on the router will correspond to vlan 73 on the switch.