HamWAN Remote Site

It's become a need for HamWAN to expand over existing internet links, as a backup and in areas we cannot hit with radio. This has shown a need at some of our other radio sites across the state, and in many cases where we can't get a good internet connection unless via restrictive NAT.

Design Requirements:

• IPv4 and IPv6
• IPv4 DHCP and IPv6 SLAAC for clients
• Transparent routing over the underlay network (people shouldn't be able to tell it's a VPN)
• Traverse NAT, even NAT 4444!
• Local Managed Switch
• POE source on the switch
• Conserve IP space in the design
• Integrate to the existing HamWAN network
• Hub and spoke no spoke to spoke direct breakout or on demand tunneling (SDWAN)
• support for up to 10 remote locations

Thoughts on Hardware

Thought was given to this for hardware and in general we favor used routing equipment which is past it's useful life from eBay. This invariably means Cisco or lower end Juniper, but Cisco has the largest amount of gear out there.

We did configure, and deploy a network based on Mikrotik routers to test this on. While we found this would "work" it leaked information from the Mikrotik as it cannot do a VRF properly. We found a number of other issues, and I've documented some here. MT might work for you, if you're ok with it, and you can get new in the box replacements from Amazon Prime for $99.

For Cisco hardware we've settled on the Cisco 2921/51 for the Spoke routers and a 3945e for the HUB. These routers are capable of doing 300 mbit+ of traffic over the VPN, and support the routing protocols we require to do dual stack IPv4 and IPv6.

Hub

The hub will plug into our core Juniper in Tampa via a ptp interface. This will speak ISIS, our IGP of choice for IPv4 and IPv4, and let the rest of HamWAN know as the sites come online.

We made the decision to use a multi-point GRE tunnel interface and run NHRP for the remote links. This allows us to use a /28 on the Tunnel, and support up to 13 remote locations without re-configuring. If we needed more remote sites, we can renumber or just use IPv6 :)

The one disadvantage to running multi-point GRE is we cannot run ISIS directly as ISIS doesn't use IP but rather CLNS for a transport. This means for the Tunnel interface and remote spoke sites we'll run OSPFv3 in a dual stack configuration. From the perspective of the spoke, they will get a default route and "announce" their routes to the hub.

Interconnection with hamwan. We prefer the hub to speak ISIS to the core, and handle both address families in the same process. Our soultion to this is redistributing the learned OSPFv3 routes into the ISIS process on the Hub.

The 3945e router was chosen for the hub. The 3945e is a 3945 which had the SPE-250 processing card in it. Like all 29/3900 routers they support various service modules from ATM to Ethernet switching interfaces, and even server blades. With the right power supply the router will even support POE or POE+ depending on the switch module installed. There are several Licenses and RTU's used on this, but by default the 3945 supports SEC/K9 and will handle a hundred tunnels at 150 mbit/s of throughput. The router is able to support well over 1gb/s of throughput and up to 3000 tunnels if the HSEC/K9 license is added to it. This license is locked to the CPU and must be generated from Cisco. As we don't need much more performance here we will not be licensing this.

FYY these are all known and the ISR/G2 routers. The next generation is the 4000 and 4400 ISR routers, which support 3 gb/s+ of crypto. As of writing they are still quite pricey on the used market.

Spoke

Our spoke site router is designed to provide us a number of Ethernet ports which serve up access to 44 net and IPv6 directly at the remote site with minimal config. We also want to support local breakout via NAT if needed too.

For HamWAN we're not concerned with encryption, so we could build a GRE tunnel without IPSEC and assuming we have an unfiltered public IPv4 at the spoke site, it would work. This would avoid the limitations of the crypto license limits as well. GRE has no ability to traverse NAT as a UDP packet, and IPSEC handles this NAT traversal quite well. Now this doesn't fix remote sites where there is layer 7 firewalling, ALG's and the like.

For IPsec we've choses to use pre-shared keys and IKEv2 vs ISAKMP as IKEv2 supports NAT traversal as via standard encapsulation of the IPSEC as UDP port 4500. It also is better in terms of us running a well known IP listener from a service denial or DOS perspective.

At the spoke we'll have a routed subnet to a VLAN interface on the router. This will bridge into the switch module and the local router will runs DHCP to hand out IPv4 and SLACC for IPv6. As this routed subnet will burn 2 IP's the management interfaces on the switch and the router, a /29 will only provide space for 4 connected devices. This may be fine at some sites, but others will need a /28 or /27. The HUB router will learn of these subnets via OSPFv3.

The routing config will be a bit complex as we want any traffic into Ethernet to not go to the default routing table. This means a VRF (or separate routing table) is needed for these interfaces. On the Spoke a VRF, HamWAN is created and the Tunnel and Vlan interface are placed inside it. The OSPFv3 process must run inside this table as well as it must not leak any routes from or into the default table on the router since the default table is how the tunnel traverses the underlay network (internet).

The decision here was made to go with the Cisco 2921 or 2951 routers. There is not much perforamcne difference in these, but they are limited to 85mbit/s of IPSEC unless they have the HSEC/K9 license AND the ISM-VPN-29 crypto accelerator module. This is known as the "CISCO2951-HSEC+/K9" bundle. Also if you intend to run the POE switch module a special power supply "PWR-2921-51-POE" is required. This this supply supplies 48v in addition to the standard 12v and 5v voltages of the standard supply.

For the local switch breakout there are several options:

SM-X-ES3-24-P   - Based on a 3560X switch - "SM-X-ES3-24-P: EtherSwitch SM L3 + PoEPlus + MACSec + 24 10/100/1000"
SM-ES3G-24-P—24 - Based on a 3560e Switch - "SM-ES3G-24-P: EtherSwitch SM L3 + POE + 24 10/100/1000"
SM-ES3-24-P—23  - Based on the 3560       - "SM-ES3G-24-P: EtherSwitch SM L2 + POE + 24 10/100/1000" 
SM-ES2-24-P     - L2 only 2960sm based    - "SM-ES2-24-P: EtherSwitch SM L2 + PoE + 23 10/100 + 1 10/100/1000" 

There are other switch modules, but these are the most popular. In our case we're running the SM-ES2-24-P as we don't require layer 3 on the switch but do require POE. These are managed on their own IP and boot their own IOS. They have two virtual 1g interfaces which interconnect with the router via the backplane and trunk to the switch. This way a Vlan73 on the router will correspond to vlan 73 on the switch.

General management

We want to have some basic security and monitoring.

• SNMPv3
• SSH Key auth
• NMS
• ACL's on lines

Configs

Hub

The is the standard Hub Config we're running with annotations on the config.

service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
#set the hostname to what you want
hostname Tampa-VPN
!
boot-start-marker
# this is the latest code from cisco as of 7-APR-2022
boot system flash:c3900e-universalk9-mz.SPA.157-3.M8.bin
boot-end-marker
!
!
logging persistent
# enable password here
enable secret 5 $1$Rr7R$3h5Yz1xtc8Ne/eY1EHODw.
!
#enable the new model for auth
aaa new-model
!
!
#tell the AAA to use the local user list for auth
aaa authentication login default local
aaa authentication login vpnclient local
aaa authorization console
aaa authorization config-commands
aaa authorization exec default local
aaa authorization network localgroups local
!
aaa session-id common
!
#log user failures
login on-failure log
login on-success log
# don't look up names in DNS
no ip domain lookup
# the domain name you need, this is needed for SSH
ip domain name tampa.flscg.org
#enable CEF
ip cef
#enable IPv6 routing
ipv6 unicast-routing
ipv6 cef
!
#enable CLNS for ISIS
clns routing
!
!
!
!
key chain ISIS_HAMWAN
 key 1
  key-string ISIS-PAssword
cts logging verbose
!
# this next sets up config archiving on the router.  Don't do this on anything where you can't remove the disk
# First you need to mkdir flash0:/cfgs if it's not there.
archive
 path flash0:/cfgs/config
 maximum 14
 write-memory
 time-period 3600
#usernames and passwords.  Note that RANCID only uses an SSH key.
username bryan privilege 15 secret 5 $1$gryM$uDSpJxJrvdTLynieY/E.V/
username dd privilege 15 secret 5 $1$/v6S$0cHfkn//80wOOUF8Eg/Iy0
username rancid privilege 15
!
# now we get into the crypto
# this first sets up a IKEv2 Proposal named IKEv2-VPN with AES256 and SHA512 
crypto ikev2 proposal IKEv2-VPN
 encryption aes-cbc-256
 integrity sha512
 group 19
!
# This makes a IKEv2 Policy and ties the proposal into the policy.
crypto ikev2 policy IKEv2-VPN-POLICY
 proposal IKEv2-VPN
!
# This sets the PSK for the remote sites
crypto ikev2 keyring HamWAN
 peer REMOTE-ROUTERS
  description REMOTE-ROUTERS
  # the address is all since we can't control the IP they come in from.  Only if the PSK matches will they be permitted
  address 0.0.0.0 0.0.0.0
  pre-shared-key <key here>
 !

# Now we make a Profile IKEv2-VPN-PROFILE and tie keyring HamWAN to it.
crypto ikev2 profile IKEv2-VPN-PROFILE
 # this must be the Gi0/0 interface address, loopbacks will not work
 match address local 44.98.249.197
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local HamWAN
!
#now attache the ike2 provite to the ipsec profile HamWAN-DMVPN
crypto ipsec profile HamWAN-DMVPN
 set ikev2-profile IKEv2-VPN-PROFILE
!
# Now into the interfaces
# loop back0 is used for logging and all sourced IP's from the router.  Note that v6 will be configured here too
interface Loopback0
 ip address 44.98.249.162 255.255.255.255
 #it must patricpate in ISIS
 ip router isis
 ipv6 address 2607:F3F0:2:400F::1/128
 ipv6 enable
 #needed for the ospfv3 process
 ipv6 ospf 1 area 1
 #needed for ISIS
 clns router isis
!
#this is the mac-daddy where all the magic happens, Tu73
interface Tunnel73
 #the subnet chosen
 ip address 44.98.249.97 255.255.255.240
 no ip redirects
 # select a smaller MTU since we're running encap.
 ip mtu 1400
 #this is the auth "key" for NHRP
 ip nhrp authentication HamWAN
 #the NHRP Network ID for 
 ip nhrp network-id 73
 #defining the next hop server as the Gi0/0 interface
 ip nhrp nhs 44.98.249.97
 # There is 74 bytes of IPv4 + GRE + IPSEC total, so this makes the max segment for tcp 1360.
 ip tcp adjust-mss 1360
 # Not 100% sure this is needed as we only use OSPFv3.
 ip ospf network broadcast
 # cuz it's cisco and sh cdp neig is nice
 cdp enable
 # This is the IPv6 Address, I've cut everything from 2607:f3f0:0002:4000::/52  subnet
 # note how we encode the ipv4 address in the device.  You could use slaac too. 
 ipv6 address 2607:F3F0:2:4000:44:98:249:97/64
 # needed to enable IPv6
 ipv6 enable
 # same as v4
 ipv6 mtu 1400
 # IPv6 is 20 bytes larger than v4
 ipv6 tcp adjust-mss 1340
 # same as v4
 ipv6 nhrp authentication HamWAN
 ipv6 nhrp network-id 73
 # this configures OSPFv3
 ospfv3 1 network broadcast
 #This ensures the HUB is the Designated Router, higher priority wins
 ospfv3 1 priority 255
 # below we setup the areas for the address families in ospfv3
 ospfv3 1 ipv4 area 1
 ospfv3 1 ipv6 area 1
 #Set the Tunnel Source 
 tunnel source GigabitEthernet0/0
 # this is letting it know it's a dmvpn and that it should use NHRP to resolve next hop
 tunnel mode gre multipoint
 # this really isn't needed and adds some over head, but I like to leave it as some basic auth
 # if you test without ipsec, it's nice to have
 tunnel key 7373
 # This enables IPSEC on the interface. 
 tunnel protection ipsec profile HamWAN-DMVPN ikev2-profile IKEv2-VPN-PROFILE
!
interface GigabitEthernet0/0
 # the /31 north bound to the juniper core router
 ip address 44.98.249.197 255.255.255.254
 # we're enabling ipv4 isis on the interface
 ip router isis
 duplex auto
 speed auto
 # v6 address to the core, note the IP used here is not from the /52 subnet.
 ipv6 address 2607:F3F0:2:1005::2/64
 # enable IPv6 AF in the ISIS on this interface
 ipv6 router isis
 # set the circuit as ISIS level 2 only
 isis circuit-type level-2-only
 # now ISIS auth on cisco is a bit fucked
 # cisco configures the hello and the PDU auth diferently
 # this command sets the hello auth, PDU is configured in the isis
 isis authentication mode md5
 isis authentication key-chain ISIS_HAMWAN
!
router ospfv3 1
 #Give it a router ID of loopback 0
 router-id 44.98.249.162
 # base the ID of the snmp-if-index for reporting
 interface-id snmp-if-index
 # limit the queue depth to 1500 TLV's to prevent overload of signaling
 queue-depth update 1500
 # update the reference bandwidth 
 auto-cost reference-bandwidth 1000
 !
 # below is where each address family is configured
 address-family ipv4 unicast
  # we don't run this on interfaces by default
  passive-interface default
  # define the interfaces OSPFv3 runs on
  no passive-interface Loopback0
  no passive-interface Tunnel73
  # send a default IPv4 Route
  default-information originate always
 exit-address-family
 !
 address-family ipv6 unicast
  # this is all the same as IPv4
  passive-interface default
  no passive-interface Tunnel73
  no passive-interface Loopback0
  default-information originate always
 exit-address-family
!
router isis
 # set the area 49.0001 and the NSAP address of the node
 net 49.0001.0440.9824.9162.00
 # this tells ISIS to authenticat all PDU's, not just hellos
 # https://netquirkengineering.files.wordpress.com/2020/07/is-is-md5-authentication.pdf
 authentication mode md5
 authentication key-chain ISIS_HAMWAN
 # use the wide metric style, it's what juniper uses by default
 metric-style wide
 # Log changes
 log-adjacency-changes
 redistribute connected
 # this takes anything the router learns from OSPFv3 (v4)
 redistribute ospfv3 1
 !
 # this is where ipv6 gets configured in ISIS
 address-family ipv6
  # we're going to use both AF's in the same process
  multi-topology
  redistribute connected
  # this takes anything the router learns from OSPFv3 (v6) including connected routes
  redistribute ospf 1 include-connected
 exit-address-family
!
# don't need support for the Sun Network Disk Protocol 
no ip forward-protocol nd
!
# nope, not serving up http
no ip http server
no ip http secure-server
!
# the below is how to setup ssh on the router. 
# the crypto key generate rsa modulus 4096 must be run once
!crypto key generate rsa modulus 4096
ip ssh time-out 60
ip ssh authentication-retries 2
ip ssh version 2
# the below is config to add the keys.  Note that a user name must be configured too.
# note that IOS has a 250 char limit on the cli, so we break them up.
ip ssh pubkey-chain
username bryan
key-string  
AAAAB3NzaC1yc2EAAAADAQABAAABAQC3ruIvMI+gYdCvOr3S5GDi4J93W3+KfePp
dK8WdLzsGyq4/fl9EnI284NVxEiNPCupxen6yJ4yruu6J+TXCaBW77m2MNyV4qo5V4qxLxGU
le3sq1AfmD6vFyMY6XTSY9+JI6Mu022uxixlILWiqTvKh31HkOM8Ui1Fb7wdjqMYEUa3snwX
SDvQKUq0ioeEvy2EdhIinGLDG8EGSR2hmqoXu6D0cfe3/zQ1kvT4lii8j5cWjA9++Ac5tHr5
rKrAUzj0+4fPKsL5bTeT5uC0e8puWNXhgn27ecv4Bx9D+KtoTaCAx//+5Q7EPTeY3ehuyO2i
K3uqH/kP4cdnCtZlm2L3
exit
exit
exit
ip ssh pubkey-chain
username rancid
key-string  
AAAAB3NzaC1yc2EAAAADAQABAAABAQDZSbQaXcziig2UlwR5cs3ihvRIGRtl8+Cc
MsgXHu3WVOt0RAt8m8egiS07BN684FazM611Y4jAN1XafYWdRNHR+8dOm77dZqsoErMynQo0
2x4nUAQUaZfRZmStYmYS0dj2wqueuKUEub29dAMfWC/rdHMI7Y7+CLXRD1W1j50SmGsKBYH2
ZWTT/UGAIQL25dYsj+rWFUk/V+Kf/oDJEBd46MpL8zVLlmL4Ft3HaFjoFasdHOf1vGvE8gAO
VuQMtCRExvdTWTH3pZEg0aNBBwCkgJaxmJw+JNbYuPvd2CiKWxxJZvInmNC3U2T7Yy/1SsA1
QZvAEG4ShtbfDWYdZx8v
exit
exit
exit
!
# setup a SNMP ACL.
ip access-list standard snmp-acl
 permit 47.206.239.202
 permit 199.47.174.149
 permit 44.98.0.0 0.0.255.255
!
# now one for the VTY's
ip access-list extended VTY
 permit tcp 199.47.174.0 0.0.0.255 any
 permit tcp 44.98.0.0 0.0.255.255 any
 permit tcp host 47.206.239.202 any
 permit tcp host 96.254.123.27 any
 deny   ip any any log-input
!
# some basic logging to the hamwan log server
logging facility user
logging source-interface Loopback0
logging host 44.98.254.1
ipv6 ioam timestamp
!
!
# ok this below is how we configure SNMPv3 with auth.
# group HamWAN, allow it to read the v1default mibs and apply the ACL "snmp-acl"
snmp-server group HamWAN v3 priv read v1default access snmp-acl
# configure a user HamWANv3, with view 'HamWAN' and use sha and aes128.  
# Note that none of the snmp utils support 192 or 256.
snmp-server user  HamWANv3 HamWAN v3 auth sha HamWAN-SNMPv3-PA55 priv aes 128 HamWAN-SNMPv3-PA55
snmp-server ifindex persist
snmp-server trap-source Loopback0
snmp-server source-interface informs Loopback0
!
line vty 0 4
 exec-timeout 180 0
 # enable the snmp acl
 access-class VTY in
 transport input all
line vty 5 15
 access-class VTY in
 transport input all
!
scheduler allocate 20000 1000
# configure this router for NTP and have it function as an NTP SERVER
ntp source Loopback0
ntp master
ntp update-calendar
ntp server 45.79.214.107
ntp server 138.236.128.36
ntp server 162.248.241.94
ntp server 172.104.193.207
!
end

Core Juniper Config

This is a config snippit of what we needed to add to our existing config. Note the ISO MTU under the vlan 26 interface. CISCO and JUNIPER will not talk if this isn't set.

interfaces {
    ge-0/0/46 {
        unit 0 {
            family ethernet-switching {
                port-mode access;
                vlan {
                    members CISCO-VPN;
                }
            }
        }
    }
    vlan {
        unit 26 {
            family inet {
                address 44.98.249.196/31;
            }
            family iso {
              # holy shit this is important.  CISCO and Juniper will not talk unless the MTU is set
		mtu 1492; 
            }
            family inet6 {
                address 2607:f3f0:0002:1005::1/64;
            }
        }
    }
}
protocols {
    isis {
        interface vlan.26;
    }
}
vlans {
    CISCO-VPN {
        vlan-id 26;
        interface {
            ge-0/0/46.0;
        }
        l3-interface vlan.26;
    }
}