Difference between revisions of "HamWAN Remote Site"

From W9CR
Jump to navigation Jump to search
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
[[Category:HamWAN]]
 +
 
It's become a need for HamWAN to expand over existing internet links, as a backup and in areas we cannot hit with radio.  This has shown a need at some of our other radio sites across the state, and in many cases where we can't get a good internet connection unless via restrictive NAT.   
 
It's become a need for HamWAN to expand over existing internet links, as a backup and in areas we cannot hit with radio.  This has shown a need at some of our other radio sites across the state, and in many cases where we can't get a good internet connection unless via restrictive NAT.   
 +
 +
[[File:HamWAN VPN Diagram.png|thumb|500px|HamWAN VPN Diagram]]
  
 
:Design Requirements:
 
:Design Requirements:
Line 19: Line 23:
 
We did configure, and deploy a network based on Mikrotik routers to test this on.  While we found this would "work" it leaked information from the Mikrotik as it cannot do a VRF properly.  We found a number of other issues, and [[MikroTik-Fail|I've documented some here]].  MT might work for you, if you're ok with it, and you can get new in the box replacements from Amazon Prime for $99.   
 
We did configure, and deploy a network based on Mikrotik routers to test this on.  While we found this would "work" it leaked information from the Mikrotik as it cannot do a VRF properly.  We found a number of other issues, and [[MikroTik-Fail|I've documented some here]].  MT might work for you, if you're ok with it, and you can get new in the box replacements from Amazon Prime for $99.   
  
−
For Cisco hardware we've settled on the Cisco 2921/51 for the Spoke routers and a 3945e for the HUB.  These routers are capable of doing 300 mbit+ of traffic over the VPN, and support the routing protocols we require to do dual stack IPv4 and IPv6.
+
For Cisco hardware we've settled on the [https://web.archive.org/web/20140907071122/http://www.cisco.com/c/en/us/products/routers/3900-series-integrated-services-routers-isr/series-comparison.html Cisco 2921/51 for the Spoke routers and a 3945e for the HUB].  These routers are capable of doing 300 mbit+ of traffic over the VPN, and support the routing protocols we require to do dual stack IPv4 and IPv6.
 +
 
 +
[https://www.cisco.com/c/en/us/td/docs/routers/access/2900/hardware/installation/guide/Hardware_Installation_Guide/Overview.html Here is a link to the hardware overview of the routers]
 +
 
 +
<gallery mode=packed-hover widths=350px heights=300px>
 +
3845e in rack.jpg| Cisco 3945e
 +
2921 in rack.jpg|Cisco 2921
 +
3845e skinless.png|"skinless" 3545e testing doublewide card in bottom slot.
 +
</Gallery>
  
 
= Hub =
 
= Hub =
  
−
The hub will plug into our core Juniper in Tampa via a ptp interface.  This will speak ISIS, our IGP of choice for IPv4 and IPv4, and let the rest of HamWAN know as the sites come online.   
+
The hub will plug into our core Juniper in Tampa via a PtP (/31) interface.  This will speak ISIS, our IGP of choice for IPv4 and IPv6, and let the rest of HamWAN know as the sites come online.   
  
 
We made the decision to use a multi-point GRE tunnel interface and run NHRP for the remote links.  This allows us to use a /28 on the Tunnel, and support up to 13 remote locations without re-configuring.  If we needed more remote sites, we can renumber or just use IPv6 :)
 
We made the decision to use a multi-point GRE tunnel interface and run NHRP for the remote links.  This allows us to use a /28 on the Tunnel, and support up to 13 remote locations without re-configuring.  If we needed more remote sites, we can renumber or just use IPv6 :)
Line 29: Line 41:
 
The one disadvantage to running multi-point GRE is we cannot run ISIS directly as ISIS doesn't use IP but rather CLNS for a transport.  This means for the Tunnel interface and remote spoke sites we'll run OSPFv3 in a dual stack configuration.  From the perspective of the spoke, they will get a default route and "announce" their routes to the hub.   
 
The one disadvantage to running multi-point GRE is we cannot run ISIS directly as ISIS doesn't use IP but rather CLNS for a transport.  This means for the Tunnel interface and remote spoke sites we'll run OSPFv3 in a dual stack configuration.  From the perspective of the spoke, they will get a default route and "announce" their routes to the hub.   
  
−
Interconnection with hamwan.  We prefer the hub to speak ISIS to the core, and handle both address families in the same process.  Our soultion to this is redistributing the learned OSPFv3 routes into the ISIS process on the Hub.  
+
Interconnection with HamWAN.  We prefer the hub to speak ISIS to the core, and handle both address families in the same process.  Our soultion to this is redistributing the learned OSPFv3 routes into the ISIS process on the Hub.
 +
 
 +
The 3945e router was chosen for the hub.  The 3945e is a 3945 which had the SPE-250 processing card in it.  Like all 29/3900 routers they support various service modules from ATM to Ethernet switching interfaces, and even server blades.  With the right power supply the router will even support POE or POE+ depending on the switch module installed.  There are several Licenses and RTU's used on this, but by default the 3945 supports SEC/K9 and will handle 225 tunnels at 150 mbit/s of throughput.  The router is able to support well over 1gb/s of throughput and up to 3000 tunnels if the HSEC/K9 license is added to it.  This license is locked to the CPU and must be generated from Cisco.  We have aquired these licenses and are using them.  The pricing is reasonable for ham radio use.
 +
 
 +
[[Media:ISR G2 Perfomance.pdf]] - This is the official Cisco Statement of performance of tthe ISR G2.  Note that Cisco lies.
  
−
The 3945e router was chosen for the hub.  The 3945e is a 3945 which had the SPE-250 processing card in it.  Like all 29/3900 routers they support various service modules from ATM to Ethernet switching interfaces, and even server blades. With the right power supply the router will even support POE or POE+ depending on the switch module installed.  There are several Licenses and RTU's used on this, but by default the 3945 supports SEC/K9 and will handle a hundred tunnels at 150 mbit/s of throughput.  The router is able to support well over 1gb/s of throughput and up to 3000 tunnels if the HSEC/K9 license is added to it. This license is locked to the CPU and must be generated from Cisco.  As we don't need much more performance here we will not be licensing this.
+
[http://miercom.com/pdf/reports/20100528.pdf Miercom has a better report] (but still paid for by Cisco)
  
 
FYY these are all known and the ISR/G2 routers.  The next generation is the 4000 and 4400 ISR routers, which support 3 gb/s+ of crypto.  As of writing they are still quite pricey on the used market.  
 
FYY these are all known and the ISR/G2 routers.  The next generation is the 4000 and 4400 ISR routers, which support 3 gb/s+ of crypto.  As of writing they are still quite pricey on the used market.  
Line 41: Line 57:
 
For HamWAN we're not concerned with encryption, so we could build a GRE tunnel without IPSEC and assuming we have an unfiltered public IPv4 at the spoke site, it would work.  This would avoid the limitations of the crypto license limits as well.  GRE has no ability to traverse NAT as a UDP packet, and IPSEC handles this NAT traversal quite well.  Now this doesn't fix remote sites where there is layer 7 firewalling, ALG's and the like.   
 
For HamWAN we're not concerned with encryption, so we could build a GRE tunnel without IPSEC and assuming we have an unfiltered public IPv4 at the spoke site, it would work.  This would avoid the limitations of the crypto license limits as well.  GRE has no ability to traverse NAT as a UDP packet, and IPSEC handles this NAT traversal quite well.  Now this doesn't fix remote sites where there is layer 7 firewalling, ALG's and the like.   
  
−
For IPsec we've choses to use pre-shared keys and IKEv2 vs ISAKMP as IKEv2 supports NAT traversal as via standard encapsulation of the IPSEC as UDP port 4500.  It also is better in terms of us running a well known IP listener from a service denial or DOS perspective.   
+
For IPsec we've chose to use pre-shared keys and IKEv2 vs ISAKMP as IKEv2 supports NAT traversal as via standard encapsulation of the IPSEC as UDP port 4500.  It also is better in terms of us running a well known IP listener from a service denial or DOS perspective.   
  
 
At the spoke we'll have a routed subnet to a VLAN interface on the router.  This will bridge into the switch module and the local router will runs DHCP to hand out IPv4 and SLACC for IPv6.  As this routed subnet will burn 2 IP's the management interfaces on the switch and the router, a /29 will only provide space for 4 connected devices.  This may be fine at some sites, but others will need a /28 or /27.  The HUB router will learn of these subnets via OSPFv3.
 
At the spoke we'll have a routed subnet to a VLAN interface on the router.  This will bridge into the switch module and the local router will runs DHCP to hand out IPv4 and SLACC for IPv6.  As this routed subnet will burn 2 IP's the management interfaces on the switch and the router, a /29 will only provide space for 4 connected devices.  This may be fine at some sites, but others will need a /28 or /27.  The HUB router will learn of these subnets via OSPFv3.
Line 47: Line 63:
 
The routing config will be a bit complex as we want any traffic into Ethernet to not go to the default routing table.  This means a [https://en.wikipedia.org/wiki/Virtual_routing_and_forwarding VRF] (or separate routing table) is needed for these interfaces.  On the Spoke a VRF, HamWAN is created and the Tunnel and Vlan interface are placed inside it.  The OSPFv3 process must run inside this table as well as it must not leak any routes from or into the default table on the router since the default table is how the tunnel traverses the underlay network (internet).   
 
The routing config will be a bit complex as we want any traffic into Ethernet to not go to the default routing table.  This means a [https://en.wikipedia.org/wiki/Virtual_routing_and_forwarding VRF] (or separate routing table) is needed for these interfaces.  On the Spoke a VRF, HamWAN is created and the Tunnel and Vlan interface are placed inside it.  The OSPFv3 process must run inside this table as well as it must not leak any routes from or into the default table on the router since the default table is how the tunnel traverses the underlay network (internet).   
  
−
The decision here was made to go with the Cisco 2921 or 2951 routers.  There is not much perforamcne difference in these, but they are limited to 85mbit/s of IPSEC unless they have the HSEC/K9 license AND the ISM-VPN-29 crypto accelerator module.  This is known as the "CISCO2951-HSEC+/K9" bundle.  Also if you intend to run the POE switch module a special power supply "PWR-2921-51-POE" is required.  This this supply supplies 48v in addition to the standard 12v and 5v voltages of the standard supply.   
+
The decision here was made to go with the Cisco 2921 or 2951 routers.  There is not much performance difference in these, but they are limited to 85mbit/s of IPSEC unless they have the HSEC/K9 license AND the ISM-VPN-29 crypto accelerator module.  This is known as the "CISCO2951-HSEC+/K9" bundle.  Also if you intend to run the POE switch module a special power supply "PWR-2921-51-POE" is required.  This this supply supplies 48v in addition to the standard 12v and 5v voltages of the standard supply.   
  
−
For the local switch breakout there are several options:
+
In testing we found the 2921 with the HSEC/K9 license would do about 150 Mbit/s of realized throughput using http://fast.com for speed tests with out the ISM-VPN-29 module.  Without the HSEC/K9 license it would cap at 65 Mbit/s.  Bear in mind this is traffic over the VPN connection so we expect a 20% overhead or so due to encapsulation.    We also tested a 3945e as a spoke router and saw 700 mbit/s, over a 750 mbit/s PON connection, and was likely limited by it.  Considering the 100-150 dollar price for a 3945e on eBay, this is great performance.
  
−
  SM-X-ES3-24-P   - Based on a 3560X switch - "SM-X-ES3-24-P: EtherSwitch SM L3 + PoEPlus + MACSec + 24 10/100/1000"
+
<gallery heights=300px widths=400px mode="packed">
 +
64mbit.png|2921 performance prior to HSEC/K9 license
 +
 
 +
150mbit.png|2921 performance after HSEC/K9 license
 +
700 mbit-s.png|3945e with HSEC/K9 as spoke performance.  
 +
</gallery>
 +
 
 +
[https://www.cisco.com/c/en/us/td/docs/routers/access/interfaces/software/feature/guide/eesm_sw.html For the local switch breakout there are several options]:
 +
 
 +
[https://www.cisco.com/c/en/us/products/collateral/routers/4000-series-integrated-services-routers-isr/datasheet-c78-730357.html SM-X-ES3-24-P]  - Based on a 3560X switch - "SM-X-ES3-24-P: EtherSwitch SM L3 + PoEPlus + MACSec + 24 10/100/1000"
 
  SM-ES3G-24-P—24 - Based on a 3560e Switch - "SM-ES3G-24-P: EtherSwitch SM L3 + POE + 24 10/100/1000"
 
  SM-ES3G-24-P—24 - Based on a 3560e Switch - "SM-ES3G-24-P: EtherSwitch SM L3 + POE + 24 10/100/1000"
 
  SM-ES3-24-P—23  - Based on the 3560      - "SM-ES3G-24-P: EtherSwitch SM L2 + POE + 24 10/100/1000"  
 
  SM-ES3-24-P—23  - Based on the 3560      - "SM-ES3G-24-P: EtherSwitch SM L2 + POE + 24 10/100/1000"  
 
  SM-ES2-24-P    - L2 only 2960sm based    - "SM-ES2-24-P: EtherSwitch SM L2 + PoE + 23 10/100 + 1 10/100/1000"  
 
  SM-ES2-24-P    - L2 only 2960sm based    - "SM-ES2-24-P: EtherSwitch SM L2 + PoE + 23 10/100 + 1 10/100/1000"  
  
−
There are other switch modules, but these are the most popular.  In our case we're running the SM-ES2-24-P as we don't require layer 3 on the switch but do require POE.  These are managed on their own IP and boot their own IOS.  They have two virtual 1g interfaces which interconnect with the router via the backplane and trunk to the switch.  This way a Vlan73 on the router will correspond to vlan 73 on the switch.  
+
There are other switch modules, but these are the most popular.  In our case we're running the SM-ES2-24-P as we don't require layer 3 on the switch but do require POE.  These are managed on their own IP and boot their own IOS.  They have two virtual 1g interfaces which interconnect with the router via the backplane and trunk to the switch.  This way a Vlan73 on the router will correspond to vlan 73 on the switch.
  
 
= General management =  
 
= General management =  
Line 65: Line 90:
 
* SSH Key auth
 
* SSH Key auth
 
* NMS
 
* NMS
−
* ACL's on lines
+
* ACL's on lines.
 +
* population of DNS to support troubleshooting
 +
 
 +
This is reflected in the configuration below, but further setup is left to the implementation here.
 +
 
 +
[https://www.cisco.com/c/en/us/support/docs/ip/access-lists/13608-21.html Cisco has a good document on secure configuration practices that you should review.]
 +
 
 +
[https://team-cymru.com/community-services/templates/secure-bgp-template/ Team CYMRU's Secure BGP Template] is a worth a review too.
 +
 
 +
= Operational thoughts/gotchas =
 +
 
 +
We've noted the following operational issues or notes for improvement
 +
 
 +
* ISPEC is used as a nat translation device, primarialy
 +
* only one spoke will work per tunnel behind a single IP, even through nat.
 +
* this will only work over IPv4.  If a remote site has IPv6 (not common in the USA) but no IPv4 this will not work. It would be easy to setup ipv6 as a transport for the remote sites.  It is very common outside the US for people to have broken IPv4 via CGN but working public unfiltered IPv6, so it may be a better option there.
 +
 
 +
We may be well served to configure an inbound VRF on the hub router only for inbound VPN clients to connect to.  Any traffic would then be on a different port out to the HamWAN.
 +
 
 +
* IPSEC + Xauth could be added or even SSL VPN for end user access.
  
 
= Configs =
 
= Configs =
Line 129: Line 173:
 
  !
 
  !
 
  # this next sets up config archiving on the router.  Don't do this on anything where you can't remove the disk
 
  # this next sets up config archiving on the router.  Don't do this on anything where you can't remove the disk
 +
# First you need to mkdir flash0:/cfgs if it's not there.
 
  archive
 
  archive
−
   path flash0:archived-config
+
   path flash0:/cfgs/config
 
   maximum 14
 
   maximum 14
 
   write-memory
 
   write-memory
−
   time-period 1440
+
   time-period 3600
 
  #usernames and passwords.  Note that RANCID only uses an SSH key.
 
  #usernames and passwords.  Note that RANCID only uses an SSH key.
 
  username bryan privilege 15 secret 5 $1$gryM$uDSpJxJrvdTLynieY/E.V/
 
  username bryan privilege 15 secret 5 $1$gryM$uDSpJxJrvdTLynieY/E.V/
Line 395: Line 440:
 
  ntp server 172.104.193.207
 
  ntp server 172.104.193.207
 
  !
 
  !
 +
end
 +
 +
== Core Juniper Config ==
 +
 +
This is a config snippit of what we needed to add to our existing config. 
 +
 +
'''Note the ISO MTU under the vlan 26 interface.  CISCO and JUNIPER will not talk if this isn't set.'''
 +
 +
interfaces {
 +
    ge-0/0/46 {
 +
        unit 0 {
 +
            family ethernet-switching {
 +
                port-mode access;
 +
                vlan {
 +
                    members CISCO-VPN;
 +
                }
 +
            }
 +
        }
 +
    }
 +
    vlan {
 +
        unit 26 {
 +
            family inet {
 +
                address 44.98.249.196/31;
 +
            }
 +
            family iso {
 +
              # holy shit this is important.  CISCO and Juniper will not talk unless the MTU is set
 +
mtu 1492;
 +
            }
 +
            family inet6 {
 +
                address 2607:f3f0:0002:1005::1/64;
 +
            }
 +
        }
 +
    }
 +
}
 +
protocols {
 +
    isis {
 +
        interface vlan.26;
 +
    }
 +
}
 +
vlans {
 +
    CISCO-VPN {
 +
        vlan-id 26;
 +
        interface {
 +
            ge-0/0/46.0;
 +
        }
 +
        l3-interface vlan.26;
 +
    }
 +
}
 +
 +
== Spoke ==
 +
 +
# HUB
 +
# Tun 73 - 44.98.249.98/29, 2607:F3F0:2:4000:44:98:249:97/64
 +
# routed subnets  44.98.249.112/29 and 2607:F3F0:2:4010::/64
 +
# Vlan73 - 44.98.249.113/29, 2607:F3F0:2:4010::1/64
 +
# Switch Module - Vlan 73 - 44.98.249.114/29
 +
# I'll not be commenting here on what's the same as the hub.
 +
 +
service timestamps debug datetime msec
 +
service timestamps log datetime msec
 +
service password-encryption
 +
!
 +
#Hostname
 +
hostname LKLDFLRXRT0
 +
!
 +
boot-start-marker
 +
# this is the latest stable code from Cisco as of 7-APR-2022
 +
boot system flash:c2900-universalk9-mz.SPA.157-3.M8.bin
 +
boot-end-marker
 +
!
 +
# This is the improtant VRF defined, and since it's both IPv4 and V6 we have to
 +
# enable both.  'ip vrf' will not work for this
 +
vrf definition HamWAN
 +
  description HamWAN VRF
 +
  !
 +
  address-family ipv4
 +
  exit-address-family
 +
  !
 +
  address-family ipv6
 +
  exit-address-family
 +
!
 +
enable secret 4 AbuNhklLMiNLBIJkVTLQE5Nrkq2yZUtUJsb.Jp9iAzs
 +
!
 +
aaa new-model
 +
!
 +
aaa authentication login default local
 +
aaa authentication login vpnclient local
 +
aaa authorization console
 +
aaa authorization config-commands
 +
aaa authorization exec default local
 +
aaa authorization network localgroups local
 +
!
 +
aaa session-id common
 +
!
 +
#exclude the Vlan interfaces on the router and on the switch
 +
ip dhcp excluded-address 44.98.249.113 44.98.249.114
 +
!
 +
# setup a dhcp pool for the routed subnet
 +
ip dhcp pool HAMWAN
 +
  vrf HamWAN
 +
  network 44.98.249.112 255.255.255.248
 +
  # this is the site domain name, it should match DNS
 +
  domain-name lkld.flscg.org
 +
  default-router 44.98.249.113
 +
  dns-server 1.1.1.1
 +
  lease 0 0 15
 +
!
 +
no ip domain lookup
 +
#again, should match the dns
 +
ip domain name lkld.flscg.org
 +
ip cef
 +
login on-failure log
 +
login on-success log
 +
ipv6 unicast-routing
 +
ipv6 multicast-routing
 +
ipv6 cef
 +
!
 +
multilink bundle-name authenticated
 +
!
 +
!
 +
!
 +
!
 +
key chain ISIS_HAMWAN
 +
  key 1
 +
  key-string 7 143F13063B25046B08213D3E315333170E1157197A524D55433E035A5317475A50142D6925477E5208100E0149185D4F077F69491F0A05171D0916
 +
cts logging verbose
 +
!
 +
# this should be added by default, but it's how you run the switch in slot 1
 +
hw-module sm 1
 +
!
 +
!
 +
!
 +
archive
 +
  path flash0:/cfgs/config
 +
  maximum 14
 +
  write-memory
 +
  time-period 3600
 +
!
 +
username bryan privilege 15 secret 5 $1$gryM$uDSpJxJrvdTLynieY/E.V/
 +
username dd privilege 15 secret 5 $1$f2AV$7sJDI32RDPaF9bgQv6qdh0
 +
username rancid privilege 15
 +
!
 +
!
 +
crypto ikev2 proposal IKEv2-VPN
 +
  encryption aes-cbc-256
 +
  integrity sha512
 +
  group 19
 +
!
 +
crypto ikev2 policy IKEv2-VPN-POLICY
 +
  proposal IKEv2-VPN
 +
!
 +
crypto ikev2 keyring HamWAN
 +
  peer REMOTE-ROUTERS
 +
  description REMOTE-ROUTERS
 +
  address 44.98.249.197
 +
  pre-shared-key PSK
 +
  !
 +
!
 +
crypto ikev2 profile IKEv2-VPN-PROFILE
 +
  match identity remote address 44.98.249.197 255.255.255.255
 +
  authentication remote pre-share
 +
  authentication local pre-share
 +
  keyring local HamWAN
 +
!
 +
# setup the VLAN 73 for HamWAN, this is where the routed subnets live
 +
vlan 73
 +
  name HamWAN
 +
!
 +
crypto ipsec profile HamWAN-DMVPN
 +
  set ikev2-profile IKEv2-VPN-PROFILE
 +
!
 +
# tunnel 73, this must be in the VRF and have an IP in the same subnet as the HUB
 +
interface Tunnel73
 +
  # put this in the HamWAN VRF
 +
  vrf forwarding HamWAN
 +
  #IP of the tunnel
 +
  ip address 44.98.249.98 255.255.255.240
 +
  ip mtu 1400
 +
  ip nhrp authentication HamWAN
 +
  ip nhrp network-id 73
 +
  ip nhrp nhs 44.98.249.97 nbma 44.98.249.197
 +
  ip tcp adjust-mss 1360
 +
  cdp enable
 +
  # again, for the v6 interface
 +
  ipv6 address 2607:F3F0:2:4000:44:98:249:98/64
 +
  ipv6 enable
 +
  ipv6 mtu 1400
 +
  ipv6 tcp adjust-mss 1340
 +
  ipv6 nhrp authentication HamWAN
 +
  ipv6 nhrp network-id 73
 +
  ipv6 nhrp nhs 2607:F3F0:2:4000:44:98:249:97 nbma 44.98.249.197
 +
  ospfv3 1 network broadcast
 +
  # this sets the priority to 0, meaning this will not become a DR or BDR
 +
  ospfv3 1 priority 0
 +
  ospfv3 1 ipv4 area 1
 +
  ospfv3 1 ipv6 area 1
 +
  # what ever source you want.
 +
  tunnel source GigabitEthernet0/1
 +
  tunnel destination 44.98.249.197
 +
  tunnel key 7373
 +
  tunnel protection ipsec profile HamWAN-DMVPN ikev2-profile IKEv2-VPN-PROFILE
 +
!
 +
# this is the outside interface set to DHCP
 +
interface GigabitEthernet0/1
 +
  ip dhcp client default-router distance 10
 +
  ip address dhcp
 +
  duplex auto
 +
  speed auto
 +
!
 +
# this is a special interface used to manage the service module
 +
interface GigabitEthernet1/0
 +
  ip address 10.0.0.1 255.255.255.252
 +
!
 +
# This is the trunk interface to the service module
 +
interface GigabitEthernet1/1
 +
  description LINK TO SWITCH
 +
  switchport mode trunk
 +
  no ip address
 +
!
 +
interface Vlan1
 +
  no ip address
 +
!
 +
# the Vlan used for the hamwan clients.  The subnets should live here
 +
interface Vlan73
 +
  #put it in the VRF
 +
  vrf forwarding HamWAN
 +
  # give it some IP's
 +
  ip address 44.98.249.113 255.255.255.248
 +
  ipv6 address 2607:F3F0:2:4010::1/64
 +
  ipv6 enable
 +
  #enable OSPFv3 for both families
 +
  ospfv3 1 network broadcast
 +
  ospfv3 1 ipv4 area 1
 +
  ospfv3 1 ipv6 area 1
 +
!
 +
# ospfv3 config
 +
router ospfv3 1
 +
  # router ID should be the tunnel interface, no loopback here
 +
  router-id 44.98.249.98
 +
  interface-id snmp-if-index
 +
  queue-depth update 1500
 +
  auto-cost reference-bandwidth 1000
 +
  !
 +
  # now this is basically the same as the hub, but the OSPFv3 is running in the
 +
  # HamWAN VRF, not the global table. 
 +
  address-family ipv4 unicast vrf HamWAN
 +
  redistribute connected
 +
  passive-interface default
 +
  no passive-interface Tunnel73
 +
  no passive-interface Vlan73
 +
  router-id 44.98.249.98
 +
  exit-address-family
 +
  !
 +
  address-family ipv6 unicast vrf HamWAN
 +
  passive-interface default
 +
  no passive-interface Tunnel73
 +
  no passive-interface Vlan73
 +
  redistribute connected
 +
  router-id 44.98.249.98
 +
  exit-address-family
 +
!
 +
no ip forward-protocol nd
 +
no ip http server
 +
no ip http secure-server
 +
!
 +
# the below is how to setup ssh on the router.
 +
# the crypto key generate rsa modulus 4096 must be run once
 +
!crypto key generate rsa modulus 4096
 +
ip ssh time-out 60
 +
# shource is the tunnel interface.
 +
ip ssh source-interface Tunnel73
 +
ip ssh authentication-retries 2
 +
ip ssh version 2
 +
# the below is config to add the keys.  Note that a user name must be configured too.
 +
# note that IOS has a 250 char limit on the cli, so we break them up.
 +
ip ssh pubkey-chain
 +
username bryan
 +
key-string 
 +
AAAAB3NzaC1yc2EAAAADAQABAAABAQC3ruIvMI+gYdCvOr3S5GDi4J93W3+KfePp
 +
dK8WdLzsGyq4/fl9EnI284NVxEiNPCupxen6yJ4yruu6J+TXCaBW77m2MNyV4qo5V4qxLxGU
 +
le3sq1AfmD6vFyMY6XTSY9+JI6Mu022uxixlILWiqTvKh31HkOM8Ui1Fb7wdjqMYEUa3snwX
 +
SDvQKUq0ioeEvy2EdhIinGLDG8EGSR2hmqoXu6D0cfe3/zQ1kvT4lii8j5cWjA9++Ac5tHr5
 +
rKrAUzj0+4fPKsL5bTeT5uC0e8puWNXhgn27ecv4Bx9D+KtoTaCAx//+5Q7EPTeY3ehuyO2i
 +
K3uqH/kP4cdnCtZlm2L3
 +
exit
 +
exit
 +
exit
 +
ip ssh pubkey-chain
 +
username rancid
 +
key-string 
 +
AAAAB3NzaC1yc2EAAAADAQABAAABAQDZSbQaXcziig2UlwR5cs3ihvRIGRtl8+Cc
 +
MsgXHu3WVOt0RAt8m8egiS07BN684FazM611Y4jAN1XafYWdRNHR+8dOm77dZqsoErMynQo0
 +
2x4nUAQUaZfRZmStYmYS0dj2wqueuKUEub29dAMfWC/rdHMI7Y7+CLXRD1W1j50SmGsKBYH2
 +
ZWTT/UGAIQL25dYsj+rWFUk/V+Kf/oDJEBd46MpL8zVLlmL4Ft3HaFjoFasdHOf1vGvE8gAO
 +
VuQMtCRExvdTWTH3pZEg0aNBBwCkgJaxmJw+JNbYuPvd2CiKWxxJZvInmNC3U2T7Yy/1SsA1
 +
QZvAEG4ShtbfDWYdZx8v
 +
exit
 +
exit
 +
exit
 +
# enable SCP
 +
ip scp server enable
 +
!
 +
# note on this, you have at allow the 10 space or you can't get to the
 +
# console of the switch
 +
ip access-list standard snmp-acl
 +
  permit 47.206.239.202
 +
  permit 199.47.174.149
 +
  permit 44.98.0.0 0.0.255.255
 +
  permit 10.0.0.0 0.0.0.255
 +
!
 +
ip access-list extended VTY
 +
  permit ip 199.47.174.0 0.0.0.255 any
 +
  permit ip 44.98.0.0 0.0.255.255 any
 +
  permit ip host 47.206.239.202 any
 +
  permit ip host 96.254.123.27 any
 +
  # same thing here for the 10/8 space and accessing the switch
 +
  permit ip 10.0.0.0 0.0.0.255 any
 +
  deny  ip any any log-input
 +
!
 +
#  Give the logging info, but use the tun73 and the HamWAN VRF
 +
logging source-interface Tunnel73 vrf HamWAN
 +
logging host 44.98.254.1 vrf HamWAN
 +
ipv6 ioam timestamp
 +
!
 +
# ok this below is how we configure SNMPv3 with auth.
 +
# group HamWAN, allow it to read the v1default mibs and apply the ACL "snmp-acl"
 +
snmp-server group HamWAN v3 priv read v1default access snmp-acl
 +
# configure a user HamWANv3, with view 'HamWAN' and use sha and aes128. 
 +
# Note that none of the snmp utils support 192 or 256.
 +
snmp-server user  HamWANv3 HamWAN v3 auth sha HamWAN-SNMPv3-PA55 priv aes 128 HamWAN-SNMPv3-PA55
 +
snmp-server ifindex persist
 +
!
 +
line con 0
 +
  speed 115200
 +
line aux 0
 +
line 2
 +
  no activation-character
 +
  no exec
 +
  transport preferred none
 +
  transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 +
  stopbits 1
 +
# so this the line for the console of the switch.  It's listening on port 2067 and you need the VTY ACL on it was well.  Note the VRF-also, if it's not there, you'll be locked out for management. 
 +
line 67
 +
  access-class VTY in vrf-also
 +
  no activation-character
 +
  no exec
 +
  transport preferred none
 +
  transport input all
 +
  transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 +
  stopbits 1
 +
  flowcontrol software
 +
line vty 0 4
 +
  access-class VTY in vrf-also
 +
  exec-timeout 180 0
 +
  transport input all
 +
line vty 5 15
 +
  access-class VTY in vrf-also
 +
  exec-timeout 180 0
 +
  transport input all
 +
!
 +
scheduler allocate 20000 1000
 +
# use the hub as an ntp server.
 +
ntp server 44.98.249.162
 +
!
 +
end
 +
 +
== Spoke Switch ==
 +
# Spoke Switch
 +
# this is simple, it's a 2960g L2 switch. 
 +
 +
no service pad
 +
service timestamps debug datetime msec
 +
service timestamps log datetime msec
 +
service password-encryption
 +
!
 +
hostname LKLDFLRXSW0
 +
!
 +
enable secret 5 $1$EJed$cqOt4TVA/BQV2kQkBSyiw/
 +
!
 +
username bryan privilege 15 secret 5 $1$gryM$uDSpJxJrvdTLynieY/E.V/
 +
username rancid privilege 15
 +
aaa new-model
 +
!
 +
aaa authentication login default local
 +
aaa authentication login vpnclient local
 +
aaa authorization console
 +
aaa authorization config-commands
 +
aaa authorization exec default local
 +
aaa authorization network localgroups local
 +
!
 +
aaa session-id common
 +
system mtu routing 1500
 +
authentication mac-move permit
 +
!
 +
no ip domain-lookup
 +
ip domain-name lkld.flscg.org
 +
!
 +
spanning-tree mode pvst
 +
spanning-tree extend system-id
 +
!
 +
vlan internal allocation policy ascending
 +
!
 +
ip ssh time-out 60
 +
ip ssh authentication-retries 2
 +
ip ssh version 2
 +
ip ssh pubkey-chain
 +
  username bryan
 +
    key-hash ssh-rsa FD61A6D6FFD817666D6E420C5E7029CB
 +
  username rancid
 +
    key-hash ssh-rsa 83FC42333036E83C116A8AAD1CAFFF15
 +
ip scp server enable
 +
!
 +
!
 +
!
 +
!
 +
interface FastEthernet0/2
 +
  shut
 +
!
 +
interface FastEthernet0/3
 +
  shut
 +
!
 +
interface FastEthernet0/4
 +
  shut
 +
!
 +
interface FastEthernet0/5
 +
  shut
 +
!
 +
interface FastEthernet0/6
 +
  shut
 +
!
 +
interface FastEthernet0/7
 +
  shut
 +
!
 +
interface FastEthernet0/8
 +
  shut
 +
!
 +
interface FastEthernet0/9
 +
  shut
 +
!
 +
interface FastEthernet0/10
 +
  shut
 +
!
 +
interface FastEthernet0/11
 +
  shut
 +
!
 +
interface FastEthernet0/12
 +
  shut
 +
!
 +
interface FastEthernet0/13
 +
  shut
 +
!
 +
interface FastEthernet0/14
 +
  shut
 +
!
 +
interface FastEthernet0/15
 +
  shut
 +
!
 +
interface FastEthernet0/16
 +
  shut
 +
!
 +
interface FastEthernet0/17
 +
  shut
 +
!
 +
interface FastEthernet0/18
 +
  shut
 +
!
 +
interface FastEthernet0/19
 +
  shut
 +
!
 +
interface FastEthernet0/20
 +
  shut
 +
!
 +
interface FastEthernet0/21
 +
  switchport access vlan 73
 +
  spanning-tree portfast
 +
!
 +
interface FastEthernet0/22
 +
  switchport access vlan 73
 +
  spanning-tree portfast
 +
!
 +
interface FastEthernet0/23
 +
  switchport access vlan 73
 +
  spanning-tree portfast
 +
!
 +
interface FastEthernet0/24
 +
  switchport access vlan 73
 +
  spanning-tree portfast
 +
!
 +
# Note that this switch only has one exposed gige port on it, the rest are fastE
 +
interface GigabitEthernet0/1
 +
  switchport access vlan 73
 +
  spanning-tree portfast
 +
!
 +
#these are the two special links to the backplane of the router.  They must be trunks
 +
interface GigabitEthernet0/25
 +
  switchport mode trunk
 +
!
 +
interface GigabitEthernet0/26
 +
  switchport mode trunk
 +
!
 +
#Shutdown Vlan1
 +
interface Vlan1
 +
  no ip address
 +
  no ip route-cache
 +
  shutdown
 +
!
 +
# this will be the management IP of the device.  There's no reouting on this box, and no need to worry about the VRF like on the router.
 +
interface Vlan73
 +
  ip address 44.98.249.114 255.255.255.248
 +
  no ip route-cache
 +
!
 +
ip default-gateway 44.98.249.113
 +
no ip http server
 +
no ip http secure-server
 +
!
 +
ip access-list standard snmp-acl
 +
  permit 47.206.239.202
 +
  permit 199.47.174.149
 +
  permit 44.98.0.0 0.0.255.255
 +
  permit 10.0.0.0 0.0.0.255
 +
!
 +
ip access-list extended VTY
 +
  permit tcp 199.47.174.0 0.0.0.255 any
 +
  permit tcp 44.98.0.0 0.0.255.255 any
 +
  permit tcp host 47.206.239.202 any
 +
  permit tcp host 96.254.123.27 any
 +
  permit tcp 10.0.0.0 0.0.0.255 any
 +
  deny  ip any any log-input
 +
logging source-interface Vlan73
 +
logging host 44.98.254.1
 +
snmp-server group HamWAN v3 priv access snmp-acl
 +
snmp-server view SNMPv3-VIEW mib-2 included
 +
snmp-server view SNMPv3-VIEW cisco included
 +
snmp ifmib ifindex persist
 +
!
 +
# no smart install
 +
no vstack
 +
!
 +
line con 0
 +
  speed 115200
 +
  flowcontrol software
 +
line vty 0 4
 +
  access-class VTY in
 +
  exec-timeout 180 0
 +
  transport input all
 +
line vty 5 15
 +
  access-class VTY in
 +
  exec-timeout 180 0
 +
  transport input all
 +
!
 +
ntp source Vlan73
 +
ntp server 44.98.249.162 prefer
 
  end
 
  end

Latest revision as of 16:05, 29 November 2022


It's become a need for HamWAN to expand over existing internet links, as a backup and in areas we cannot hit with radio. This has shown a need at some of our other radio sites across the state, and in many cases where we can't get a good internet connection unless via restrictive NAT.

HamWAN VPN Diagram
Design Requirements:
  • IPv4 and IPv6
  • IPv4 DHCP and IPv6 SLAAC for clients
  • Transparent routing over the underlay network (people shouldn't be able to tell it's a VPN)
  • Traverse NAT, even NAT 4444!
  • Local Managed Switch
  • POE source on the switch
  • Conserve IP space in the design
  • Integrate to the existing HamWAN network
  • Hub and spoke no spoke to spoke direct breakout or on demand tunneling (SDWAN)
  • support for up to 10 remote locations

Thoughts on Hardware

Thought was given to this for hardware and in general we favor used routing equipment which is past it's useful life from eBay. This invariably means Cisco or lower end Juniper, but Cisco has the largest amount of gear out there.

We did configure, and deploy a network based on Mikrotik routers to test this on. While we found this would "work" it leaked information from the Mikrotik as it cannot do a VRF properly. We found a number of other issues, and I've documented some here. MT might work for you, if you're ok with it, and you can get new in the box replacements from Amazon Prime for $99.

For Cisco hardware we've settled on the Cisco 2921/51 for the Spoke routers and a 3945e for the HUB. These routers are capable of doing 300 mbit+ of traffic over the VPN, and support the routing protocols we require to do dual stack IPv4 and IPv6.

Here is a link to the hardware overview of the routers

  • Cisco 3945e

  • Cisco 2921

  • "skinless" 3545e testing doublewide card in bottom slot.

Hub

The hub will plug into our core Juniper in Tampa via a PtP (/31) interface. This will speak ISIS, our IGP of choice for IPv4 and IPv6, and let the rest of HamWAN know as the sites come online.

We made the decision to use a multi-point GRE tunnel interface and run NHRP for the remote links. This allows us to use a /28 on the Tunnel, and support up to 13 remote locations without re-configuring. If we needed more remote sites, we can renumber or just use IPv6 :)

The one disadvantage to running multi-point GRE is we cannot run ISIS directly as ISIS doesn't use IP but rather CLNS for a transport. This means for the Tunnel interface and remote spoke sites we'll run OSPFv3 in a dual stack configuration. From the perspective of the spoke, they will get a default route and "announce" their routes to the hub.

Interconnection with HamWAN. We prefer the hub to speak ISIS to the core, and handle both address families in the same process. Our soultion to this is redistributing the learned OSPFv3 routes into the ISIS process on the Hub.

The 3945e router was chosen for the hub. The 3945e is a 3945 which had the SPE-250 processing card in it. Like all 29/3900 routers they support various service modules from ATM to Ethernet switching interfaces, and even server blades. With the right power supply the router will even support POE or POE+ depending on the switch module installed. There are several Licenses and RTU's used on this, but by default the 3945 supports SEC/K9 and will handle 225 tunnels at 150 mbit/s of throughput. The router is able to support well over 1gb/s of throughput and up to 3000 tunnels if the HSEC/K9 license is added to it. This license is locked to the CPU and must be generated from Cisco. We have aquired these licenses and are using them. The pricing is reasonable for ham radio use.

Media:ISR G2 Perfomance.pdf - This is the official Cisco Statement of performance of tthe ISR G2. Note that Cisco lies.

Miercom has a better report (but still paid for by Cisco)

FYY these are all known and the ISR/G2 routers. The next generation is the 4000 and 4400 ISR routers, which support 3 gb/s+ of crypto. As of writing they are still quite pricey on the used market.

Spoke

Our spoke site router is designed to provide us a number of Ethernet ports which serve up access to 44 net and IPv6 directly at the remote site with minimal config. We also want to support local breakout via NAT if needed too.

For HamWAN we're not concerned with encryption, so we could build a GRE tunnel without IPSEC and assuming we have an unfiltered public IPv4 at the spoke site, it would work. This would avoid the limitations of the crypto license limits as well. GRE has no ability to traverse NAT as a UDP packet, and IPSEC handles this NAT traversal quite well. Now this doesn't fix remote sites where there is layer 7 firewalling, ALG's and the like.

For IPsec we've chose to use pre-shared keys and IKEv2 vs ISAKMP as IKEv2 supports NAT traversal as via standard encapsulation of the IPSEC as UDP port 4500. It also is better in terms of us running a well known IP listener from a service denial or DOS perspective.

At the spoke we'll have a routed subnet to a VLAN interface on the router. This will bridge into the switch module and the local router will runs DHCP to hand out IPv4 and SLACC for IPv6. As this routed subnet will burn 2 IP's the management interfaces on the switch and the router, a /29 will only provide space for 4 connected devices. This may be fine at some sites, but others will need a /28 or /27. The HUB router will learn of these subnets via OSPFv3.

The routing config will be a bit complex as we want any traffic into Ethernet to not go to the default routing table. This means a VRF (or separate routing table) is needed for these interfaces. On the Spoke a VRF, HamWAN is created and the Tunnel and Vlan interface are placed inside it. The OSPFv3 process must run inside this table as well as it must not leak any routes from or into the default table on the router since the default table is how the tunnel traverses the underlay network (internet).

The decision here was made to go with the Cisco 2921 or 2951 routers. There is not much performance difference in these, but they are limited to 85mbit/s of IPSEC unless they have the HSEC/K9 license AND the ISM-VPN-29 crypto accelerator module. This is known as the "CISCO2951-HSEC+/K9" bundle. Also if you intend to run the POE switch module a special power supply "PWR-2921-51-POE" is required. This this supply supplies 48v in addition to the standard 12v and 5v voltages of the standard supply.

In testing we found the 2921 with the HSEC/K9 license would do about 150 Mbit/s of realized throughput using http://fast.com for speed tests with out the ISM-VPN-29 module. Without the HSEC/K9 license it would cap at 65 Mbit/s. Bear in mind this is traffic over the VPN connection so we expect a 20% overhead or so due to encapsulation. We also tested a 3945e as a spoke router and saw 700 mbit/s, over a 750 mbit/s PON connection, and was likely limited by it. Considering the 100-150 dollar price for a 3945e on eBay, this is great performance.

  • 2921 performance prior to HSEC/K9 license

  • 2921 performance after HSEC/K9 license

  • 3945e with HSEC/K9 as spoke performance.

For the local switch breakout there are several options: 

SM-X-ES3-24-P  - Based on a 3560X switch - "SM-X-ES3-24-P: EtherSwitch SM L3 + PoEPlus + MACSec + 24 10/100/1000"
SM-ES3G-24-P—24 - Based on a 3560e Switch - "SM-ES3G-24-P: EtherSwitch SM L3 + POE + 24 10/100/1000"
SM-ES3-24-P—23  - Based on the 3560       - "SM-ES3G-24-P: EtherSwitch SM L2 + POE + 24 10/100/1000" 
SM-ES2-24-P     - L2 only 2960sm based    - "SM-ES2-24-P: EtherSwitch SM L2 + PoE + 23 10/100 + 1 10/100/1000"

There are other switch modules, but these are the most popular. In our case we're running the SM-ES2-24-P as we don't require layer 3 on the switch but do require POE. These are managed on their own IP and boot their own IOS. They have two virtual 1g interfaces which interconnect with the router via the backplane and trunk to the switch. This way a Vlan73 on the router will correspond to vlan 73 on the switch.

General management

We want to have some basic security and monitoring.

  • SNMPv3
  • SSH Key auth
  • NMS
  • ACL's on lines.
  • population of DNS to support troubleshooting

This is reflected in the configuration below, but further setup is left to the implementation here.

Cisco has a good document on secure configuration practices that you should review.

Team CYMRU's Secure BGP Template is a worth a review too.

Operational thoughts/gotchas

We've noted the following operational issues or notes for improvement

  • ISPEC is used as a nat translation device, primarialy
  • only one spoke will work per tunnel behind a single IP, even through nat.
  • this will only work over IPv4. If a remote site has IPv6 (not common in the USA) but no IPv4 this will not work. It would be easy to setup ipv6 as a transport for the remote sites. It is very common outside the US for people to have broken IPv4 via CGN but working public unfiltered IPv6, so it may be a better option there.

We may be well served to configure an inbound VRF on the hub router only for inbound VPN clients to connect to. Any traffic would then be on a different port out to the HamWAN.

  • IPSEC + Xauth could be added or even SSL VPN for end user access.

Configs

Hub

The is the standard Hub Config we're running with annotations on the config. 

service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
#set the hostname to what you want
hostname Tampa-VPN
!
boot-start-marker
# this is the latest code from cisco as of 7-APR-2022
boot system flash:c3900e-universalk9-mz.SPA.157-3.M8.bin
boot-end-marker
!
!
logging persistent
# enable password here
enable secret 5 $1$Rr7R$3h5Yz1xtc8Ne/eY1EHODw.
!
#enable the new model for auth
aaa new-model
!
!
#tell the AAA to use the local user list for auth
aaa authentication login default local
aaa authentication login vpnclient local
aaa authorization console
aaa authorization config-commands
aaa authorization exec default local
aaa authorization network localgroups local
!
aaa session-id common
!
#log user failures
login on-failure log
login on-success log
# don't look up names in DNS
no ip domain lookup
# the domain name you need, this is needed for SSH
ip domain name tampa.flscg.org
#enable CEF
ip cef
#enable IPv6 routing
ipv6 unicast-routing
ipv6 cef
!
#enable CLNS for ISIS
clns routing
!
!
!
!
key chain ISIS_HAMWAN
 key 1
  key-string ISIS-PAssword
cts logging verbose
!
# this next sets up config archiving on the router.  Don't do this on anything where you can't remove the disk
# First you need to mkdir flash0:/cfgs if it's not there.
archive
 path flash0:/cfgs/config
 maximum 14
 write-memory
 time-period 3600
#usernames and passwords.  Note that RANCID only uses an SSH key.
username bryan privilege 15 secret 5 $1$gryM$uDSpJxJrvdTLynieY/E.V/
username dd privilege 15 secret 5 $1$/v6S$0cHfkn//80wOOUF8Eg/Iy0
username rancid privilege 15
!
# now we get into the crypto
# this first sets up a IKEv2 Proposal named IKEv2-VPN with AES256 and SHA512 
crypto ikev2 proposal IKEv2-VPN
 encryption aes-cbc-256
 integrity sha512
 group 19
!
# This makes a IKEv2 Policy and ties the proposal into the policy.
crypto ikev2 policy IKEv2-VPN-POLICY
 proposal IKEv2-VPN
!
# This sets the PSK for the remote sites
crypto ikev2 keyring HamWAN
 peer REMOTE-ROUTERS
  description REMOTE-ROUTERS
  # the address is all since we can't control the IP they come in from.  Only if the PSK matches will they be permitted
  address 0.0.0.0 0.0.0.0
  pre-shared-key <key here>
 !

# Now we make a Profile IKEv2-VPN-PROFILE and tie keyring HamWAN to it.
crypto ikev2 profile IKEv2-VPN-PROFILE
 # this must be the Gi0/0 interface address, loopbacks will not work
 match address local 44.98.249.197
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local HamWAN
!
#now attache the ike2 provite to the ipsec profile HamWAN-DMVPN
crypto ipsec profile HamWAN-DMVPN
 set ikev2-profile IKEv2-VPN-PROFILE
!
# Now into the interfaces
# loop back0 is used for logging and all sourced IP's from the router.  Note that v6 will be configured here too
interface Loopback0
 ip address 44.98.249.162 255.255.255.255
 #it must patricpate in ISIS
 ip router isis
 ipv6 address 2607:F3F0:2:400F::1/128
 ipv6 enable
 #needed for the ospfv3 process
 ipv6 ospf 1 area 1
 #needed for ISIS
 clns router isis
!
#this is the mac-daddy where all the magic happens, Tu73
interface Tunnel73
 #the subnet chosen
 ip address 44.98.249.97 255.255.255.240
 no ip redirects
 # select a smaller MTU since we're running encap.
 ip mtu 1400
 #this is the auth "key" for NHRP
 ip nhrp authentication HamWAN
 #the NHRP Network ID for 
 ip nhrp network-id 73
 #defining the next hop server as the Gi0/0 interface
 ip nhrp nhs 44.98.249.97
 # There is 74 bytes of IPv4 + GRE + IPSEC total, so this makes the max segment for tcp 1360.
 ip tcp adjust-mss 1360
 # Not 100% sure this is needed as we only use OSPFv3.
 ip ospf network broadcast
 # cuz it's cisco and sh cdp neig is nice
 cdp enable
 # This is the IPv6 Address, I've cut everything from 2607:f3f0:0002:4000::/52  subnet
 # note how we encode the ipv4 address in the device.  You could use slaac too. 
 ipv6 address 2607:F3F0:2:4000:44:98:249:97/64
 # needed to enable IPv6
 ipv6 enable
 # same as v4
 ipv6 mtu 1400
 # IPv6 is 20 bytes larger than v4
 ipv6 tcp adjust-mss 1340
 # same as v4
 ipv6 nhrp authentication HamWAN
 ipv6 nhrp network-id 73
 # this configures OSPFv3
 ospfv3 1 network broadcast
 #This ensures the HUB is the Designated Router, higher priority wins
 ospfv3 1 priority 255
 # below we setup the areas for the address families in ospfv3
 ospfv3 1 ipv4 area 1
 ospfv3 1 ipv6 area 1
 #Set the Tunnel Source 
 tunnel source GigabitEthernet0/0
 # this is letting it know it's a dmvpn and that it should use NHRP to resolve next hop
 tunnel mode gre multipoint
 # this really isn't needed and adds some over head, but I like to leave it as some basic auth
 # if you test without ipsec, it's nice to have
 tunnel key 7373
 # This enables IPSEC on the interface. 
 tunnel protection ipsec profile HamWAN-DMVPN ikev2-profile IKEv2-VPN-PROFILE
!
interface GigabitEthernet0/0
 # the /31 north bound to the juniper core router
 ip address 44.98.249.197 255.255.255.254
 # we're enabling ipv4 isis on the interface
 ip router isis
 duplex auto
 speed auto
 # v6 address to the core, note the IP used here is not from the /52 subnet.
 ipv6 address 2607:F3F0:2:1005::2/64
 # enable IPv6 AF in the ISIS on this interface
 ipv6 router isis
 # set the circuit as ISIS level 2 only
 isis circuit-type level-2-only
 # now ISIS auth on cisco is a bit fucked
 # cisco configures the hello and the PDU auth diferently
 # this command sets the hello auth, PDU is configured in the isis
 isis authentication mode md5
 isis authentication key-chain ISIS_HAMWAN
!
router ospfv3 1
 #Give it a router ID of loopback 0
 router-id 44.98.249.162
 # base the ID of the snmp-if-index for reporting
 interface-id snmp-if-index
 # limit the queue depth to 1500 TLV's to prevent overload of signaling
 queue-depth update 1500
 # update the reference bandwidth 
 auto-cost reference-bandwidth 1000
 !
 # below is where each address family is configured
 address-family ipv4 unicast
  # we don't run this on interfaces by default
  passive-interface default
  # define the interfaces OSPFv3 runs on
  no passive-interface Loopback0
  no passive-interface Tunnel73
  # send a default IPv4 Route
  default-information originate always
 exit-address-family
 !
 address-family ipv6 unicast
  # this is all the same as IPv4
  passive-interface default
  no passive-interface Tunnel73
  no passive-interface Loopback0
  default-information originate always
 exit-address-family
!
router isis
 # set the area 49.0001 and the NSAP address of the node
 net 49.0001.0440.9824.9162.00
 # this tells ISIS to authenticat all PDU's, not just hellos
 # https://netquirkengineering.files.wordpress.com/2020/07/is-is-md5-authentication.pdf
 authentication mode md5
 authentication key-chain ISIS_HAMWAN
 # use the wide metric style, it's what juniper uses by default
 metric-style wide
 # Log changes
 log-adjacency-changes
 redistribute connected
 # this takes anything the router learns from OSPFv3 (v4)
 redistribute ospfv3 1
 !
 # this is where ipv6 gets configured in ISIS
 address-family ipv6
  # we're going to use both AF's in the same process
  multi-topology
  redistribute connected
  # this takes anything the router learns from OSPFv3 (v6) including connected routes
  redistribute ospf 1 include-connected
 exit-address-family
!
# don't need support for the Sun Network Disk Protocol 
no ip forward-protocol nd
!
# nope, not serving up http
no ip http server
no ip http secure-server
!
# the below is how to setup ssh on the router. 
# the crypto key generate rsa modulus 4096 must be run once
!crypto key generate rsa modulus 4096
ip ssh time-out 60
ip ssh authentication-retries 2
ip ssh version 2
# the below is config to add the keys.  Note that a user name must be configured too.
# note that IOS has a 250 char limit on the cli, so we break them up.
ip ssh pubkey-chain
username bryan
key-string  
AAAAB3NzaC1yc2EAAAADAQABAAABAQC3ruIvMI+gYdCvOr3S5GDi4J93W3+KfePp
dK8WdLzsGyq4/fl9EnI284NVxEiNPCupxen6yJ4yruu6J+TXCaBW77m2MNyV4qo5V4qxLxGU
le3sq1AfmD6vFyMY6XTSY9+JI6Mu022uxixlILWiqTvKh31HkOM8Ui1Fb7wdjqMYEUa3snwX
SDvQKUq0ioeEvy2EdhIinGLDG8EGSR2hmqoXu6D0cfe3/zQ1kvT4lii8j5cWjA9++Ac5tHr5
rKrAUzj0+4fPKsL5bTeT5uC0e8puWNXhgn27ecv4Bx9D+KtoTaCAx//+5Q7EPTeY3ehuyO2i
K3uqH/kP4cdnCtZlm2L3
exit
exit
exit
ip ssh pubkey-chain
username rancid
key-string  
AAAAB3NzaC1yc2EAAAADAQABAAABAQDZSbQaXcziig2UlwR5cs3ihvRIGRtl8+Cc
MsgXHu3WVOt0RAt8m8egiS07BN684FazM611Y4jAN1XafYWdRNHR+8dOm77dZqsoErMynQo0
2x4nUAQUaZfRZmStYmYS0dj2wqueuKUEub29dAMfWC/rdHMI7Y7+CLXRD1W1j50SmGsKBYH2
ZWTT/UGAIQL25dYsj+rWFUk/V+Kf/oDJEBd46MpL8zVLlmL4Ft3HaFjoFasdHOf1vGvE8gAO
VuQMtCRExvdTWTH3pZEg0aNBBwCkgJaxmJw+JNbYuPvd2CiKWxxJZvInmNC3U2T7Yy/1SsA1
QZvAEG4ShtbfDWYdZx8v
exit
exit
exit
!
# setup a SNMP ACL.
ip access-list standard snmp-acl
 permit 47.206.239.202
 permit 199.47.174.149
 permit 44.98.0.0 0.0.255.255
!
# now one for the VTY's
ip access-list extended VTY
 permit tcp 199.47.174.0 0.0.0.255 any
 permit tcp 44.98.0.0 0.0.255.255 any
 permit tcp host 47.206.239.202 any
 permit tcp host 96.254.123.27 any
 deny   ip any any log-input
!
# some basic logging to the hamwan log server
logging facility user
logging source-interface Loopback0
logging host 44.98.254.1
ipv6 ioam timestamp
!
!
# ok this below is how we configure SNMPv3 with auth.
# group HamWAN, allow it to read the v1default mibs and apply the ACL "snmp-acl"
snmp-server group HamWAN v3 priv read v1default access snmp-acl
# configure a user HamWANv3, with view 'HamWAN' and use sha and aes128.  
# Note that none of the snmp utils support 192 or 256.
snmp-server user  HamWANv3 HamWAN v3 auth sha HamWAN-SNMPv3-PA55 priv aes 128 HamWAN-SNMPv3-PA55
snmp-server ifindex persist
snmp-server trap-source Loopback0
snmp-server source-interface informs Loopback0
!
line vty 0 4
 exec-timeout 180 0
 # enable the snmp acl
 access-class VTY in
 transport input all
line vty 5 15
 access-class VTY in
 transport input all
!
scheduler allocate 20000 1000
# configure this router for NTP and have it function as an NTP SERVER
ntp source Loopback0
ntp master
ntp update-calendar
ntp server 45.79.214.107
ntp server 138.236.128.36
ntp server 162.248.241.94
ntp server 172.104.193.207
!
end

Core Juniper Config

This is a config snippit of what we needed to add to our existing config.

Note the ISO MTU under the vlan 26 interface. CISCO and JUNIPER will not talk if this isn't set. 

interfaces {
    ge-0/0/46 {
        unit 0 {
            family ethernet-switching {
                port-mode access;
                vlan {
                    members CISCO-VPN;
                }
            }
        }
    }
    vlan {
        unit 26 {
            family inet {
                address 44.98.249.196/31;
            }
            family iso {
              # holy shit this is important.  CISCO and Juniper will not talk unless the MTU is set
		mtu 1492; 
            }
            family inet6 {
                address 2607:f3f0:0002:1005::1/64;
            }
        }
    }
}
protocols {
    isis {
        interface vlan.26;
    }
}
vlans {
    CISCO-VPN {
        vlan-id 26;
        interface {
            ge-0/0/46.0;
        }
        l3-interface vlan.26;
    }
}

Spoke

# HUB
# Tun 73 - 44.98.249.98/29, 2607:F3F0:2:4000:44:98:249:97/64
# routed subnets  44.98.249.112/29 and 2607:F3F0:2:4010::/64
# Vlan73 - 44.98.249.113/29, 2607:F3F0:2:4010::1/64
# Switch Module - Vlan 73 - 44.98.249.114/29
# I'll not be commenting here on what's the same as the hub.

service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
#Hostname
hostname LKLDFLRXRT0 
!
boot-start-marker
# this is the latest stable code from Cisco as of 7-APR-2022
boot system flash:c2900-universalk9-mz.SPA.157-3.M8.bin
boot-end-marker
!
# This is the improtant VRF defined, and since it's both IPv4 and V6 we have to 
# enable both.  'ip vrf' will not work for this
vrf definition HamWAN
 description HamWAN VRF
 !
 address-family ipv4
 exit-address-family
 !
 address-family ipv6
 exit-address-family
!
enable secret 4 AbuNhklLMiNLBIJkVTLQE5Nrkq2yZUtUJsb.Jp9iAzs
!
aaa new-model
!
aaa authentication login default local
aaa authentication login vpnclient local
aaa authorization console
aaa authorization config-commands
aaa authorization exec default local
aaa authorization network localgroups local
!
aaa session-id common
!
#exclude the Vlan interfaces on the router and on the switch
ip dhcp excluded-address 44.98.249.113 44.98.249.114
!
# setup a dhcp pool for the routed subnet
ip dhcp pool HAMWAN
 vrf HamWAN
 network 44.98.249.112 255.255.255.248
 # this is the site domain name, it should match DNS
 domain-name lkld.flscg.org
 default-router 44.98.249.113
 dns-server 1.1.1.1
 lease 0 0 15
!
no ip domain lookup
#again, should match the dns
ip domain name lkld.flscg.org
ip cef
login on-failure log
login on-success log
ipv6 unicast-routing
ipv6 multicast-routing
ipv6 cef
!
multilink bundle-name authenticated
!
!
!
!
key chain ISIS_HAMWAN
 key 1
  key-string 7 143F13063B25046B08213D3E315333170E1157197A524D55433E035A5317475A50142D6925477E5208100E0149185D4F077F69491F0A05171D0916
cts logging verbose
!
# this should be added by default, but it's how you run the switch in slot 1
hw-module sm 1
!
!
!
archive
 path flash0:/cfgs/config
 maximum 14
 write-memory
 time-period 3600
!
username bryan privilege 15 secret 5 $1$gryM$uDSpJxJrvdTLynieY/E.V/
username dd privilege 15 secret 5 $1$f2AV$7sJDI32RDPaF9bgQv6qdh0
username rancid privilege 15
!
!
crypto ikev2 proposal IKEv2-VPN
 encryption aes-cbc-256
 integrity sha512
 group 19
!
crypto ikev2 policy IKEv2-VPN-POLICY
 proposal IKEv2-VPN
!
crypto ikev2 keyring HamWAN
 peer REMOTE-ROUTERS
  description REMOTE-ROUTERS
  address 44.98.249.197
  pre-shared-key PSK
 !
!
crypto ikev2 profile IKEv2-VPN-PROFILE
 match identity remote address 44.98.249.197 255.255.255.255
 authentication remote pre-share
 authentication local pre-share
 keyring local HamWAN
!
# setup the VLAN 73 for HamWAN, this is where the routed subnets live
vlan 73
 name HamWAN
!
crypto ipsec profile HamWAN-DMVPN
 set ikev2-profile IKEv2-VPN-PROFILE
!
# tunnel 73, this must be in the VRF and have an IP in the same subnet as the HUB
interface Tunnel73
 # put this in the HamWAN VRF
 vrf forwarding HamWAN
 #IP of the tunnel
 ip address 44.98.249.98 255.255.255.240
 ip mtu 1400
 ip nhrp authentication HamWAN
 ip nhrp network-id 73
 ip nhrp nhs 44.98.249.97 nbma 44.98.249.197
 ip tcp adjust-mss 1360
 cdp enable
 # again, for the v6 interface
 ipv6 address 2607:F3F0:2:4000:44:98:249:98/64
 ipv6 enable
 ipv6 mtu 1400
 ipv6 tcp adjust-mss 1340
 ipv6 nhrp authentication HamWAN
 ipv6 nhrp network-id 73
 ipv6 nhrp nhs 2607:F3F0:2:4000:44:98:249:97 nbma 44.98.249.197
 ospfv3 1 network broadcast
 # this sets the priority to 0, meaning this will not become a DR or BDR
 ospfv3 1 priority 0
 ospfv3 1 ipv4 area 1
 ospfv3 1 ipv6 area 1
 # what ever source you want.
 tunnel source GigabitEthernet0/1
 tunnel destination 44.98.249.197
 tunnel key 7373
 tunnel protection ipsec profile HamWAN-DMVPN ikev2-profile IKEv2-VPN-PROFILE
!
# this is the outside interface set to DHCP
interface GigabitEthernet0/1
 ip dhcp client default-router distance 10
 ip address dhcp
 duplex auto
 speed auto
!
# this is a special interface used to manage the service module
interface GigabitEthernet1/0
 ip address 10.0.0.1 255.255.255.252
!
# This is the trunk interface to the service module 
interface GigabitEthernet1/1
 description LINK TO SWITCH
 switchport mode trunk
 no ip address
!
interface Vlan1
 no ip address
!
# the Vlan used for the hamwan clients.  The subnets should live here
interface Vlan73
 #put it in the VRF
 vrf forwarding HamWAN
 # give it some IP's
 ip address 44.98.249.113 255.255.255.248
 ipv6 address 2607:F3F0:2:4010::1/64
 ipv6 enable
 #enable OSPFv3 for both families
 ospfv3 1 network broadcast
 ospfv3 1 ipv4 area 1
 ospfv3 1 ipv6 area 1
!
# ospfv3 config
router ospfv3 1
 # router ID should be the tunnel interface, no loopback here
 router-id 44.98.249.98
 interface-id snmp-if-index
 queue-depth update 1500
 auto-cost reference-bandwidth 1000
 !
 # now this is basically the same as the hub, but the OSPFv3 is running in the 
 # HamWAN VRF, not the global table.  
 address-family ipv4 unicast vrf HamWAN
  redistribute connected
  passive-interface default
  no passive-interface Tunnel73
  no passive-interface Vlan73
  router-id 44.98.249.98
 exit-address-family
 !
 address-family ipv6 unicast vrf HamWAN
  passive-interface default
  no passive-interface Tunnel73
  no passive-interface Vlan73
  redistribute connected
  router-id 44.98.249.98
 exit-address-family
!
no ip forward-protocol nd
no ip http server
no ip http secure-server
!
# the below is how to setup ssh on the router. 
# the crypto key generate rsa modulus 4096 must be run once
!crypto key generate rsa modulus 4096
ip ssh time-out 60
# shource is the tunnel interface. 
ip ssh source-interface Tunnel73
ip ssh authentication-retries 2
ip ssh version 2
# the below is config to add the keys.  Note that a user name must be configured too.
# note that IOS has a 250 char limit on the cli, so we break them up.
ip ssh pubkey-chain
username bryan
key-string  
AAAAB3NzaC1yc2EAAAADAQABAAABAQC3ruIvMI+gYdCvOr3S5GDi4J93W3+KfePp
dK8WdLzsGyq4/fl9EnI284NVxEiNPCupxen6yJ4yruu6J+TXCaBW77m2MNyV4qo5V4qxLxGU
le3sq1AfmD6vFyMY6XTSY9+JI6Mu022uxixlILWiqTvKh31HkOM8Ui1Fb7wdjqMYEUa3snwX
SDvQKUq0ioeEvy2EdhIinGLDG8EGSR2hmqoXu6D0cfe3/zQ1kvT4lii8j5cWjA9++Ac5tHr5
rKrAUzj0+4fPKsL5bTeT5uC0e8puWNXhgn27ecv4Bx9D+KtoTaCAx//+5Q7EPTeY3ehuyO2i
K3uqH/kP4cdnCtZlm2L3
exit
exit
exit
ip ssh pubkey-chain
username rancid
key-string  
AAAAB3NzaC1yc2EAAAADAQABAAABAQDZSbQaXcziig2UlwR5cs3ihvRIGRtl8+Cc
MsgXHu3WVOt0RAt8m8egiS07BN684FazM611Y4jAN1XafYWdRNHR+8dOm77dZqsoErMynQo0
2x4nUAQUaZfRZmStYmYS0dj2wqueuKUEub29dAMfWC/rdHMI7Y7+CLXRD1W1j50SmGsKBYH2
ZWTT/UGAIQL25dYsj+rWFUk/V+Kf/oDJEBd46MpL8zVLlmL4Ft3HaFjoFasdHOf1vGvE8gAO
VuQMtCRExvdTWTH3pZEg0aNBBwCkgJaxmJw+JNbYuPvd2CiKWxxJZvInmNC3U2T7Yy/1SsA1
QZvAEG4ShtbfDWYdZx8v
exit
exit
exit
# enable SCP
ip scp server enable
!
# note on this, you have at allow the 10 space or you can't get to the 
# console of the switch
ip access-list standard snmp-acl
 permit 47.206.239.202
 permit 199.47.174.149
 permit 44.98.0.0 0.0.255.255
 permit 10.0.0.0 0.0.0.255
!
ip access-list extended VTY
 permit ip 199.47.174.0 0.0.0.255 any
 permit ip 44.98.0.0 0.0.255.255 any
 permit ip host 47.206.239.202 any
 permit ip host 96.254.123.27 any
 # same thing here for the 10/8 space and accessing the switch
 permit ip 10.0.0.0 0.0.0.255 any
 deny   ip any any log-input
!
#  Give the logging info, but use the tun73 and the HamWAN VRF
logging source-interface Tunnel73 vrf HamWAN
logging host 44.98.254.1 vrf HamWAN
ipv6 ioam timestamp
!
# ok this below is how we configure SNMPv3 with auth.
# group HamWAN, allow it to read the v1default mibs and apply the ACL "snmp-acl"
snmp-server group HamWAN v3 priv read v1default access snmp-acl
# configure a user HamWANv3, with view 'HamWAN' and use sha and aes128.  
# Note that none of the snmp utils support 192 or 256.
snmp-server user  HamWANv3 HamWAN v3 auth sha HamWAN-SNMPv3-PA55 priv aes 128 HamWAN-SNMPv3-PA55
snmp-server ifindex persist
!
line con 0
 speed 115200
line aux 0
line 2
 no activation-character
 no exec
 transport preferred none
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
# so this the line for the console of the switch.  It's listening on port 2067 and you need the VTY ACL on it was well.  Note the VRF-also, if it's not there, you'll be locked out for management.  
line 67
 access-class VTY in vrf-also
 no activation-character
 no exec
 transport preferred none
 transport input all
 transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
 stopbits 1
 flowcontrol software
line vty 0 4
 access-class VTY in vrf-also
 exec-timeout 180 0
 transport input all
line vty 5 15
 access-class VTY in vrf-also
 exec-timeout 180 0
 transport input all
!
scheduler allocate 20000 1000
# use the hub as an ntp server. 
ntp server 44.98.249.162
!
end

Spoke Switch

# Spoke Switch
# this is simple, it's a 2960g L2 switch.  

no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname LKLDFLRXSW0
!
enable secret 5 $1$EJed$cqOt4TVA/BQV2kQkBSyiw/
!
username bryan privilege 15 secret 5 $1$gryM$uDSpJxJrvdTLynieY/E.V/
username rancid privilege 15
aaa new-model
!
aaa authentication login default local
aaa authentication login vpnclient local
aaa authorization console
aaa authorization config-commands
aaa authorization exec default local
aaa authorization network localgroups local
!
aaa session-id common
system mtu routing 1500
authentication mac-move permit
!
no ip domain-lookup
ip domain-name lkld.flscg.org
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
ip ssh time-out 60
ip ssh authentication-retries 2
ip ssh version 2
ip ssh pubkey-chain
  username bryan
   key-hash ssh-rsa FD61A6D6FFD817666D6E420C5E7029CB
  username rancid
   key-hash ssh-rsa 83FC42333036E83C116A8AAD1CAFFF15
ip scp server enable
!
!
!
!
interface FastEthernet0/2
 shut
!
interface FastEthernet0/3
 shut
!
interface FastEthernet0/4
 shut
!
interface FastEthernet0/5
 shut
!
interface FastEthernet0/6
 shut
!
interface FastEthernet0/7
 shut
!
interface FastEthernet0/8
 shut
!
interface FastEthernet0/9
 shut
!
interface FastEthernet0/10
 shut
!
interface FastEthernet0/11
 shut
!
interface FastEthernet0/12
 shut
!
interface FastEthernet0/13
 shut
!
interface FastEthernet0/14
 shut
!
interface FastEthernet0/15
 shut
!
interface FastEthernet0/16
 shut
!
interface FastEthernet0/17
 shut
!
interface FastEthernet0/18
 shut
!
interface FastEthernet0/19
 shut
!
interface FastEthernet0/20
 shut
!
interface FastEthernet0/21
 switchport access vlan 73
 spanning-tree portfast
!
interface FastEthernet0/22
 switchport access vlan 73
 spanning-tree portfast
!
interface FastEthernet0/23
 switchport access vlan 73
 spanning-tree portfast
!
interface FastEthernet0/24
 switchport access vlan 73
 spanning-tree portfast
!
# Note that this switch only has one exposed gige port on it, the rest are fastE
interface GigabitEthernet0/1
 switchport access vlan 73
 spanning-tree portfast
!
#these are the two special links to the backplane of the router.  They must be trunks
interface GigabitEthernet0/25
 switchport mode trunk
!
interface GigabitEthernet0/26
 switchport mode trunk
!
#Shutdown Vlan1
interface Vlan1
 no ip address
 no ip route-cache
 shutdown
!
# this will be the management IP of the device.  There's no reouting on this box, and no need to worry about the VRF like on the router.
interface Vlan73
 ip address 44.98.249.114 255.255.255.248
 no ip route-cache
!
ip default-gateway 44.98.249.113
no ip http server
no ip http secure-server
!
ip access-list standard snmp-acl
 permit 47.206.239.202
 permit 199.47.174.149
 permit 44.98.0.0 0.0.255.255
 permit 10.0.0.0 0.0.0.255
!
ip access-list extended VTY
 permit tcp 199.47.174.0 0.0.0.255 any
 permit tcp 44.98.0.0 0.0.255.255 any
 permit tcp host 47.206.239.202 any
 permit tcp host 96.254.123.27 any
 permit tcp 10.0.0.0 0.0.0.255 any
 deny   ip any any log-input
logging source-interface Vlan73
logging host 44.98.254.1
snmp-server group HamWAN v3 priv access snmp-acl
snmp-server view SNMPv3-VIEW mib-2 included
snmp-server view SNMPv3-VIEW cisco included
snmp ifmib ifindex persist
!
# no smart install
no vstack
!
line con 0
 speed 115200
 flowcontrol software
line vty 0 4
 access-class VTY in
 exec-timeout 180 0
 transport input all
line vty 5 15
 access-class VTY in
 exec-timeout 180 0
 transport input all
!
ntp source Vlan73
ntp server 44.98.249.162 prefer
end
Retrieved from "https://wiki.w9cr.net/index.php?title=HamWAN_Remote_Site&oldid=7345"