Server Provisioning

This is intended to be a reference for setting up a VM or Server for AllStatLink.

Server Overview

Basic Requirements

AllStarLink has standardized on Ubuntu 16.04 LTS for it's servers.

The minimum configuration of any server will be 2 cores of 2 GHz or faster, 4 GiB of ram and 40 GiB of Disk.

All new servers shall support IPv6, or have it available from the hosting provider.

Reverse IP's shall be delegated via a CNAME to $NAME.PTR.allstarlink.org, where $NAME is the name of the server.

All servers shall be partitioned to use / as the only partition unless a specific configuration is required.

Install guide

When provisioning a new server

• check the VM is setup (cpu/mem/disk) as it should be, if not contact the provider

  cat /proc/cpuinfo |grep processor
  processor       : 0
  processor       : 1
  processor       : 2
  processor       : 3

• on the server install python (apt-get install python). This is needed for the ansibile provisioning
• setup the server in the infrastructure configs and push the users and keys to it.

Force IPv4 for Apt

echo "Acquire::ForceIPv4 true;" >/etc/apt/apt.conf.d/99force-ipv4

Mandatory Software

All servers require this software

apt-get install ntp ntpdate vim screen net-tools strongswan fail2ban snmp haveged libacl1-dev python3-dev libssl-dev gcc g++ fio pbzip2 ncdu sudo traceroute bash-completion ifupdown inetutils-ping dnsutils lldpd iptables less rsyslog netcat snmp snmpd libsnmp-dev ncat telnet libacl1-dev python3-dev libssl-dev gcc g++ python3-llfuse pkg-config cron libwww-perl ipset udev

Mandatory Configs

Ubuntu 18 Config

Ubuntu 18 uses the net netplan config. It's gay as fuck.

First you need to disable the resolved service:

sudo systemctl disable systemd-resolved.service
sudo systemctl stop systemd-resolved
rm /etc/resolv.conf 
sudo touch /etc/cloud/cloud-init.disabled
sudo apt-get purge cloud-init

echo "nameserver 1.1.1.1" > /etc/resolv.conf
apt-get install ifupdown

Configure /etc/network/interfaces

systemctl unmask networking
systemctl enable networking
systemctl restart networking

systemctl stop systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
systemctl disable systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
systemctl mask systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
apt-get --assume-yes purge nplan netplan.io

Network Config

• The network should be configured to use /etc/network/interfaces, and add DNS and the firewall to it and search in the allstarlink.org domain

# The primary network interface
auto eth0
iface eth0 inet6 static
   address 9805:0900:0340:1000::2600/64
   autoconf 0
   accept_ra 2	
iface eth0 inet static
    address 44.103.0.49
    netmask 255.255.255.0
    network 44.103.0.0
    broadcast 44.103.0.255
    gateway 44.103.0.1 
    dns-nameservers 44.103.0.4 1.1.1.1
    dns-search allstarlink.org
    up /etc/network/firewall.sh

• There is typically only one network interface, and it will be named dynamically. We must setup this using udev to be persistent

   root@server# ifconfig |grep HWaddr
   eth0      Link encap:Ethernet  HWaddr 52:54:00:73:86:06  

Now take this HWaddr and put it in the config file

echo 'SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:73:86:06", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="eth0"' >>/etc/udev/rules.d/70-persistent-net.rules

• configure screen to use the scroll back buffer

echo -e "#Bryan Config for scroll back buffer\ntermcapinfo xterm|xterms|xs|rxvt ti@:te@" >>/etc/screenrc

• configure bash completion for interactive shells

   vim /etc/bash.bashrc
   uncomment the stuff below 
   # enable bash completion in interactive shells 

• set the host name

   echo "domain.allstarlink.org" >/etc/hostname

• set the default editor

   update-alternatives --config editor 
   Then select #3 vim.basic

• setup a firewall as /etc/network/firewall.sh and chmod +x it. You'll need to edit this based on the machine. Note the stuff in tampa uses a firewall on the HV too.

   #!/bin/bash
   
   INET_IF=eth0
   
   #Flush and zero all tables
   modprobe ip_tables
   modprobe ipt_limit
   modprobe iptable_mangle
   modprobe ipt_state
   modprobe ipt_LOG
   modprobe iptable_filter
   
   iptables -F INPUT
   iptables -F FORWARD
   iptables -t nat -F POSTROUTING
   iptables -t nat -F PREROUTING
   
   #init the log-and-drop chain
   iptables -F log-and-drop
   iptables -X log-and-drop
   iptables -N log-and-drop
   #init log-and-reject
   iptables -F log-and-reject
   iptables -X log-and-reject
   iptables -N log-and-reject
   
   echo "all tables flushed and dropped"
   # Specific chain used for logging packets before blocking them
   iptables -A log-and-drop -j LOG --log-prefix "[IPTables] Drop "
   iptables -A log-and-drop -j DROP
   
   # Specific chain used for logging packets before blocking them
   iptables -A log-and-reject -j LOG --log-prefix "[IPTables] Reject "
   iptables -A log-and-reject -j REJECT
   echo "logging chains setup"
   
   # The packets having the TCP flags activated are dropped
   # and so for the ones with no flag at all (often used with Nmap scans)
   iptables -A FORWARD -p tcp --tcp-flags ALL ALL -j log-and-drop
   iptables -A FORWARD -p tcp --tcp-flags ALL NONE -j log-and-drop
   
   #Global blocks
   #iptables -t filter -A INPUT -j DROP -s 119.118.232.185/24
   
   # allow IPSEC from other boxes
   IPSECsrc='199.47.174.150,44.98.254.151,44.103.0.48,44.103.0.49,44.98.254.145,44.72.21.13,44.72.21.12'
   #Technically the next two are not needed as we have the policy
   iptables -A INPUT -i $INET_IF -p 50 -j ACCEPT --src "$IPSECsrc"
   iptables -A INPUT -i $INET_IF -p 51 -j ACCEPT --src "$IPSECsrc"
   iptables -A INPUT -i $INET_IF -p udp --dport 500 -j ACCEPT --src "$IPSECsrc"
   # this is needed to allow all ipsec packets when it's host to host
   iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc" 
   
   # allow all ssh in
   iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 22
   
   #allow http and https
   #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 80
   #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 443
   # allow asterisk 4569
   #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 4569
   # allow DNS
   #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 53
   #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 53
   
   echo "end of services"
   # allow ping at 2 per sec
    iptables -t filter -A INPUT -j ACCEPT --in-interface $INET_IF --protocol icmp --icmp-type echo-request --match limit --limit 4/s --limit-burst 3
    iptables -t filter -A INPUT -j log-and-drop  --in-interface $INET_IF --protocol icmp --icmp-type echo-request
   # allow responces to local initated connections
   #iptables -A INPUT -i  $INET_IF --match state --state NEW,INVALID -j log-and-drop
   #iptables -A FORWARD -i $INET_IF  --match state --state NEW,INVALID -j log-and-drop
   iptables -t filter -A INPUT -j ACCEPT --match state --state RELATED,ESTABLISHED
   # Set rp_filter to 2
   for i in `find /proc/sys/net/ipv*/conf -name rp_filter`
   do
           echo "2" >$i
   done
   # setup a default deny rule for outside traffic
   iptables -t filter -A INPUT --in-interface $INET_IF -j log-and-drop

• setup fail2ban for ssh and have it null route offenders. edit ignoreip as needed

   vi /etc/fail2ban/jail.conf
   ignoreip = 127.0.0.1/8 199.47.172.0/22 44.98.254.0/24 44.72.21.0/24 44.103.0.0/24
   bantime  = 3600
   # A host is banned if it has generated "maxretry" during the last "findtime"
   # seconds.
   findtime  = 3600
   # "maxretry" is the number of failures before a host get banned.
   maxretry = 2
   banaction = route

• Set the TimeZone to UTC

   sudo timedatectl set-timezone UTC

• Set the server up in forward and reverse DNS
  • for reverse have the provider do a CNAME in their reverse file pointing to $DOMAIN.PTR.allstarlink.org. In the allstarlink.org DNS zone add an entry

   example:
   stats             IN      PTR     stats.allstarlink.org.

This will do a lookup on 130.254.98.44.in-addr.arpa. and return a CNAME pointing to stats.PTR.allstarlink.org, which has a PTR record pointing to stats.allstarlink.org.

Configure IPSEC

AllStarLink servers use strong crypto using host to host IPSEC between them for protection of services. This is configured only between servers that need it, as we don't have dynamic tunneling enabled, and each server needs a config for each tunnel. This can quickly add up to lots of configrations.

This example will show two servers, 1 and 2 with IP 44.1.1.1 and 44.2.2.2 respectively.

Server 1

We need to provision the ipsec tools to know about the connections and configure a pre shared key (PSK). Note the left server is always the local server.

/etc/ipsec.conf

conn one-to-two
     authby=secret
     #auto=start enabled the tunnel to come up even if there is not traffic for it.  
     auto=start
     keyexchange=ike
     left=4.1.1.1
     right=4.2.2.2
     leftikeport=500
     rightikeport=500   
     type=transport
     esp=aes128gcm16!
     dpddelay=5
     dpdtimeout=20
     dpdaction=restart

vim /etc/ipsec.secrets
44.1.1.1 44.2.2.2 : PSK "This is the AllStarLink PSK"

Then do an 'ipsec restart' on the server.

Server 2

/etc/ipsec.conf

   conn two-to-one
           authby=secret
           #auto=start enabled the tunnel to come up even if there is not traffic for it.  
           auto=start
           keyexchange=ike
           left=4.2.2.2
           right=4.1.1.1
           leftikeport=500
           rightikeport=500   
           type=transport
           esp=aes128gcm16!
           dpddelay=5
           dpdtimeout=20
           dpdaction=restart

   vim /etc/ipsec.secrets
   44.2.2.2 44.1.1.1 : PSK "This is the AllStarLink PSK"

Then do an 'ipsec restart' on the server.

Verify IPsec

The 'ipsec' command is used to verify the tunnel is up between the servers

   root@server# ipsec status
   two-to-one[839]: ESTABLISHED 98 minutes ago, 44.1.1.1[44.1.1.1]...44.2.2.2[44.2.2.2]
   two-to-one{13209}:  INSTALLED, TRANSPORT, reqid 695, ESP SPIs: c824e4db_i c1e4bf5c_o
   two-to-one{13209}:   44.1.1.1/32 === 44.2.2.2/32

If they are not up, check /var/log/syslog and restart ipsec on both servers. Some times a server can get in a bad status if there is a mis-config. Also it's worth noting that IPSEC is processed by iptables once it's decrypted, the iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc" line in the firewall allows all IPsec packets once decrypted to bypass the firewall. This is able to prevent traffic between unencrypted services on the servers (e.g. mysql will not connect if the ipsec is down).

Configure Postfix

Postfix is installed to forward mail for root to a smtp host.

apt-get install postfix mailutils

This will run an installer with a curses interface and you must select Satallite System. Check the System mail name is the hostname of the server, and the SMTP relay host is morty.keekles.org. Root and postmaster mail should be rootmail@allstarlink.org.

Should you need to reconfigure this use:

dpkg-reconfigure postfix

other aliases are setup in /etc/aliases. You must run newaliases after this is updated for them to take effect.

Verification

It's important to verify the server provisiong before being put into production.

Items to check

• reboot the server/vm, do all services start properly?
• Is the IP address configured on the server on eth0?
• Is the hostname set?
• Is it configured in DNS both forward and reverse?
• Is the firewall active (try netcat on a non-permitted port)
• IPSEC is active ipsec status?
• Does Screen work in an xterm with scroll back?
• Is the time set via ntp ntptime and is the timezone set to UTC?
• Is fail2ban working? Make a couple test connections and see if the IP is null routed ip route show

You may need to check your other services on this server now.

Network Monitoring

So couple things we need SNMP installed and provisioned in observium. For linux hosts try to do this over the private interface and also install the unix agent.

Prerequisites

First add the host name to the /etc/hosts on eyes

This way we're not using DNS for anything

UDP 161 need to be permitted ony from eyes, same with the observium TCP port 36602.

SNMP config

SNMPv3 config

• Install SNMP

   sudo apt-get -y install snmp snmpd libsnmp-dev

• Stop the snmpd service so we can add a user

   sudo service snmpd stop

• Add a SNMPv3 user:

   sudo net-snmp-config --create-snmpv3-user -ro -A AuthPassword -a MD5 -x AES privUser
   
   sudo net-snmp-config --create-snmpv3-user -ro -A keeklespasswd -a MD5 -x AES KeeklesSNMP

• copy the /etc/snmp/snmpd.conf from eyes to the subject server

   scp /etc/snmp/snmpd.conf bryan@p25hub.w9cr.net:/home/bryan
   mv /home/bryan/snmpd.conf /etc/snmp/snmpd.conf

• edit the /etc/snmp/snmpd.conf

    vim /etc/snmp/snmpd.conf
    Update the syslocation and such

• restart snmpd

   service snmpd start

• Test SNMP

   # TEST
   snmpwalk -v3 -u KeeklesSNMP -l authNoPriv -a MD5 -A keeklespasswd 127.0.0.1

Install Observium agent

Installation

  export SERVER=<hostname>

• on target server:

   sudo apt-get install snmpd php perl curl xinetd snmp libsnmp-dev libwww-perl 
   #only needed for postfix
   apt-get -y install rrdtool mailgraph
   dpkg-reconfigure mailgraph

• install the current distro program on the target:

   sudo curl -o /usr/local/bin/distro https://gitlab.com/observium/distroscript/raw/master/distro
   sudo chmod +x /usr/local/bin/distro

• From eyes

   scp /opt/observium/scripts/observium_agent_xinetd $SERVER:/etc/xinetd.d/observium_agent_xinetd
   scp /opt/observium/scripts/observium_agent $SERVER:/usr/bin/observium_agent
   ssh $SERVER mkdir -p /usr/lib/observium_agent
   ssh $SERVER mkdir -p /usr/lib/observium_agent/scripts-available
   ssh $SERVER mkdir -p /usr/lib/observium_agent/scripts-enabled
   scp /opt/observium/scripts/agent-local/* $SERVER:/usr/lib/observium_agent/scripts-available

• on target enable the various allowed items

   ln -s /usr/lib/observium_agent/scripts-available/os /usr/lib/observium_agent/scripts-enabled
   #ln -s /usr/lib/observium_agent/scripts-available/zimbra /usr/lib/observium_agent/scripts-enabled
   ln -s /usr/lib/observium_agent/scripts-available/dpkg /usr/lib/observium_agent/scripts-enabled
   ln -s /usr/lib/observium_agent/scripts-available/ntpd /usr/lib/observium_agent/scripts-enabled
   ln -s /usr/lib/observium_agent/scripts-available/virt-what /usr/lib/observium_agent/scripts-enabled 
   #ln -s /usr/lib/observium_agent/scripts-available/postfix_mailgraph /usr/lib/observium_agent/scripts-enabled

• Edit /etc/xinetd.d/observium_agent_xinetd so the Observium server is allowed to connect. You can do this by substituting 127.0.0.1, or place your IP after it, separated by a space. Make sure to restart xinetd afterwards so the configuration file is read.

   sudo service xinetd restart

• Test from eyes

   telnet $SERVER 36602
   snmpwalk -v3 -u KeeklesSNMP -l authNoPriv -a MD5 -A keeklespasswd $SERVER