Server Provisioning
This is intended to be a reference for setting up a VM or Server for AllStatLink.
Contents
Server Overview
Basic Requirements
AllStarLink has standardized on Ubuntu 16.04 LTS for it's servers.
The minimum configuration of any server will be 2 cores of 2 GHz or faster, 4 GiB of ram and 40 GiB of Disk.
All new servers shall support IPv6, or have it available from the hosting provider.
Reverse IP's shall be delegated via a CNAME to $NAME.PTR.allstarlink.org, where $NAME is the name of the server.
All servers shall be partitioned to use / as the only partition unless a specific configuration is required.
Install guide
When provisioning a new server
- check the VM is setup (cpu/mem/disk) as it should be, if not contact the provider
cat /proc/cpuinfo |grep processor processor : 0 processor : 1 processor : 2 processor : 3
- on the server install python (apt-get install python). This is needed for the ansibile provisioning
- setup the server in the infrastructure configs and push the users and keys to it.
Force IPv4 for Apt
echo "Acquire::ForceIPv4 true;" >/etc/apt/apt.conf.d/99force-ipv4
Mandatory Software
All servers require this software
apt-get install ntp ntpdate vim screen net-tools strongswan fail2ban snmp haveged libacl1-dev python3-dev libssl-dev gcc g++ fio pbzip2 ncdu sudo traceroute bash-completion ifupdown inetutils-ping dnsutils lldpd iptables less rsyslog netcat snmp snmpd libsnmp-dev ncat telnet libacl1-dev python3-dev libssl-dev gcc g++ python3-llfuse pkg-config cron libwww-perl ipset udev
Mandatory Configs
Ubuntu 18 Config
Ubuntu 18 uses the net netplan config. It's gay as fuck.
First you need to disable the resolved service:
sudo systemctl disable systemd-resolved.service sudo systemctl stop systemd-resolved rm /etc/resolv.conf sudo touch /etc/cloud/cloud-init.disabled sudo apt-get purge cloud-init
echo "nameserver 1.1.1.1" > /etc/resolv.conf apt-get install ifupdown
Configure /etc/network/interfaces
systemctl unmask networking systemctl enable networking systemctl restart networking
systemctl stop systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online systemctl disable systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online systemctl mask systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online apt-get --assume-yes purge nplan netplan.io
Network Config
- The network should be configured to use /etc/network/interfaces, and add DNS and the firewall to it and search in the allstarlink.org domain
# The primary network interface auto eth0 iface eth0 inet6 static address 9805:0900:0340:1000::2600/64 autoconf 0 accept_ra 2 iface eth0 inet static address 44.103.0.49 netmask 255.255.255.0 network 44.103.0.0 broadcast 44.103.0.255 gateway 44.103.0.1 dns-nameservers 44.103.0.4 1.1.1.1 dns-search allstarlink.org up /etc/network/firewall.sh
- There is typically only one network interface, and it will be named dynamically. We must setup this using udev to be persistent
root@server# ifconfig |grep HWaddr eth0 Link encap:Ethernet HWaddr 52:54:00:73:86:06
Now take this HWaddr and put it in the config file
echo 'SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:73:86:06", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="eth0"' >>/etc/udev/rules.d/70-persistent-net.rules
- configure screen to use the scroll back buffer
echo -e "#Bryan Config for scroll back buffer\ntermcapinfo xterm|xterms|xs|rxvt ti@:te@" >>/etc/screenrc
- configure bash completion for interactive shells
vim /etc/bash.bashrc uncomment the stuff below # enable bash completion in interactive shells
- set the host name
echo "domain.allstarlink.org" >/etc/hostname
- set the default editor
update-alternatives --config editor Then select #3 vim.basic
- setup a firewall as /etc/network/firewall.sh and chmod +x it. You'll need to edit this based on the machine. Note the stuff in tampa uses a firewall on the HV too.
#!/bin/bash INET_IF=eth0 #Flush and zero all tables modprobe ip_tables modprobe ipt_limit modprobe iptable_mangle modprobe ipt_state modprobe ipt_LOG modprobe iptable_filter iptables -F INPUT iptables -F FORWARD iptables -t nat -F POSTROUTING iptables -t nat -F PREROUTING #init the log-and-drop chain iptables -F log-and-drop iptables -X log-and-drop iptables -N log-and-drop #init log-and-reject iptables -F log-and-reject iptables -X log-and-reject iptables -N log-and-reject echo "all tables flushed and dropped" # Specific chain used for logging packets before blocking them iptables -A log-and-drop -j LOG --log-prefix "[IPTables] Drop " iptables -A log-and-drop -j DROP # Specific chain used for logging packets before blocking them iptables -A log-and-reject -j LOG --log-prefix "[IPTables] Reject " iptables -A log-and-reject -j REJECT echo "logging chains setup" # The packets having the TCP flags activated are dropped # and so for the ones with no flag at all (often used with Nmap scans) iptables -A FORWARD -p tcp --tcp-flags ALL ALL -j log-and-drop iptables -A FORWARD -p tcp --tcp-flags ALL NONE -j log-and-drop #Global blocks #iptables -t filter -A INPUT -j DROP -s 119.118.232.185/24 # allow IPSEC from other boxes IPSECsrc='199.47.174.150,44.98.254.151,44.103.0.48,44.103.0.49,44.98.254.145,44.72.21.13,44.72.21.12' #Technically the next two are not needed as we have the policy iptables -A INPUT -i $INET_IF -p 50 -j ACCEPT --src "$IPSECsrc" iptables -A INPUT -i $INET_IF -p 51 -j ACCEPT --src "$IPSECsrc" iptables -A INPUT -i $INET_IF -p udp --dport 500 -j ACCEPT --src "$IPSECsrc" # this is needed to allow all ipsec packets when it's host to host iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc" # allow all ssh in iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 22 #allow http and https #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 80 #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 443 # allow asterisk 4569 #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 4569 # allow DNS #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 53 #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 53 echo "end of services" # allow ping at 2 per sec iptables -t filter -A INPUT -j ACCEPT --in-interface $INET_IF --protocol icmp --icmp-type echo-request --match limit --limit 4/s --limit-burst 3 iptables -t filter -A INPUT -j log-and-drop --in-interface $INET_IF --protocol icmp --icmp-type echo-request # allow responces to local initated connections #iptables -A INPUT -i $INET_IF --match state --state NEW,INVALID -j log-and-drop #iptables -A FORWARD -i $INET_IF --match state --state NEW,INVALID -j log-and-drop iptables -t filter -A INPUT -j ACCEPT --match state --state RELATED,ESTABLISHED # Set rp_filter to 2 for i in `find /proc/sys/net/ipv*/conf -name rp_filter` do echo "2" >$i done # setup a default deny rule for outside traffic iptables -t filter -A INPUT --in-interface $INET_IF -j log-and-drop
- setup fail2ban for ssh and have it null route offenders. edit ignoreip as needed
vi /etc/fail2ban/jail.conf ignoreip = 127.0.0.1/8 199.47.172.0/22 44.98.254.0/24 44.72.21.0/24 44.103.0.0/24 bantime = 3600 # A host is banned if it has generated "maxretry" during the last "findtime" # seconds. findtime = 3600 # "maxretry" is the number of failures before a host get banned. maxretry = 2 banaction = route
- Set the TimeZone to UTC
sudo timedatectl set-timezone UTC
- Set the server up in forward and reverse DNS
- for reverse have the provider do a CNAME in their reverse file pointing to $DOMAIN.PTR.allstarlink.org. In the allstarlink.org DNS zone add an entry
example: stats IN PTR stats.allstarlink.org.
This will do a lookup on 130.254.98.44.in-addr.arpa. and return a CNAME pointing to stats.PTR.allstarlink.org, which has a PTR record pointing to stats.allstarlink.org.
Configure IPSEC
AllStarLink servers use strong crypto using host to host IPSEC between them for protection of services. This is configured only between servers that need it, as we don't have dynamic tunneling enabled, and each server needs a config for each tunnel. This can quickly add up to lots of configrations.
This example will show two servers, 1 and 2 with IP 44.1.1.1 and 44.2.2.2 respectively.
Server 1
We need to provision the ipsec tools to know about the connections and configure a pre shared key (PSK). Note the left server is always the local server.
/etc/ipsec.conf
conn one-to-two authby=secret #auto=start enabled the tunnel to come up even if there is not traffic for it. auto=start keyexchange=ike left=4.1.1.1 right=4.2.2.2 leftikeport=500 rightikeport=500 type=transport esp=aes128gcm16! dpddelay=5 dpdtimeout=20 dpdaction=restart
vim /etc/ipsec.secrets 44.1.1.1 44.2.2.2 : PSK "This is the AllStarLink PSK"
Then do an 'ipsec restart' on the server.
Server 2
/etc/ipsec.conf
conn two-to-one authby=secret #auto=start enabled the tunnel to come up even if there is not traffic for it. auto=start keyexchange=ike left=4.2.2.2 right=4.1.1.1 leftikeport=500 rightikeport=500 type=transport esp=aes128gcm16! dpddelay=5 dpdtimeout=20 dpdaction=restart
vim /etc/ipsec.secrets 44.2.2.2 44.1.1.1 : PSK "This is the AllStarLink PSK"
Then do an 'ipsec restart' on the server.
Verify IPsec
The 'ipsec' command is used to verify the tunnel is up between the servers
root@server# ipsec status two-to-one[839]: ESTABLISHED 98 minutes ago, 44.1.1.1[44.1.1.1]...44.2.2.2[44.2.2.2] two-to-one{13209}: INSTALLED, TRANSPORT, reqid 695, ESP SPIs: c824e4db_i c1e4bf5c_o two-to-one{13209}: 44.1.1.1/32 === 44.2.2.2/32
If they are not up, check /var/log/syslog and restart ipsec on both servers. Some times a server can get in a bad status if there is a mis-config. Also it's worth noting that IPSEC is processed by iptables once it's decrypted, the iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc"
line in the firewall allows all IPsec packets once decrypted to bypass the firewall. This is able to prevent traffic between unencrypted services on the servers (e.g. mysql will not connect if the ipsec is down).
Configure Postfix
Postfix is installed to forward mail for root to a smtp host.
apt-get install postfix mailutils
This will run an installer with a curses interface and you must select Satallite System. Check the System mail name is the hostname of the server, and the SMTP relay host is morty.keekles.org. Root and postmaster mail should be rootmail@allstarlink.org.
Should you need to reconfigure this use:
dpkg-reconfigure postfix
other aliases are setup in /etc/aliases. You must run newaliases after this is updated for them to take effect.
Verification
It's important to verify the server provisiong before being put into production.
Items to check
- reboot the server/vm, do all services start properly?
- Is the IP address configured on the server on eth0?
- Is the hostname set?
- Is it configured in DNS both forward and reverse?
- Is the firewall active (try netcat on a non-permitted port)
- IPSEC is active
ipsec status
? - Does Screen work in an xterm with scroll back?
- Is the time set via ntp
ntptime
and is the timezone set to UTC? - Is fail2ban working? Make a couple test connections and see if the IP is null routed
ip route show
You may need to check your other services on this server now.
Network Monitoring
So couple things we need SNMP installed and provisioned in observium. For linux hosts try to do this over the private interface and also install the unix agent.
Prerequisites
First add the host name to the /etc/hosts on eyes
This way we're not using DNS for anything
UDP 161 need to be permitted ony from eyes, same with the observium TCP port 36602.
SNMP config
SNMPv3 config
- Install SNMP
sudo apt-get -y install snmp snmpd libsnmp-dev
- Stop the snmpd service so we can add a user
sudo service snmpd stop
- Add a SNMPv3 user:
sudo net-snmp-config --create-snmpv3-user -ro -A AuthPassword -a MD5 -x AES privUser sudo net-snmp-config --create-snmpv3-user -ro -A keeklespasswd -a MD5 -x AES KeeklesSNMP
- copy the /etc/snmp/snmpd.conf from eyes to the subject server
scp /etc/snmp/snmpd.conf bryan@p25hub.w9cr.net:/home/bryan mv /home/bryan/snmpd.conf /etc/snmp/snmpd.conf
- edit the /etc/snmp/snmpd.conf
vim /etc/snmp/snmpd.conf Update the syslocation and such
- restart snmpd
service snmpd start
- Test SNMP
# TEST snmpwalk -v3 -u KeeklesSNMP -l authNoPriv -a MD5 -A keeklespasswd 127.0.0.1
Install Observium agent
Installation
export SERVER=<hostname>
- on target server:
sudo apt-get install snmpd php perl curl xinetd snmp libsnmp-dev libwww-perl #only needed for postfix apt-get -y install rrdtool mailgraph dpkg-reconfigure mailgraph
- install the current distro program on the target:
sudo curl -o /usr/local/bin/distro https://gitlab.com/observium/distroscript/raw/master/distro sudo chmod +x /usr/local/bin/distro
- From eyes
scp /opt/observium/scripts/observium_agent_xinetd $SERVER:/etc/xinetd.d/observium_agent_xinetd scp /opt/observium/scripts/observium_agent $SERVER:/usr/bin/observium_agent ssh $SERVER mkdir -p /usr/lib/observium_agent ssh $SERVER mkdir -p /usr/lib/observium_agent/scripts-available ssh $SERVER mkdir -p /usr/lib/observium_agent/scripts-enabled scp /opt/observium/scripts/agent-local/* $SERVER:/usr/lib/observium_agent/scripts-available
- on target enable the various allowed items
ln -s /usr/lib/observium_agent/scripts-available/os /usr/lib/observium_agent/scripts-enabled #ln -s /usr/lib/observium_agent/scripts-available/zimbra /usr/lib/observium_agent/scripts-enabled ln -s /usr/lib/observium_agent/scripts-available/dpkg /usr/lib/observium_agent/scripts-enabled ln -s /usr/lib/observium_agent/scripts-available/ntpd /usr/lib/observium_agent/scripts-enabled ln -s /usr/lib/observium_agent/scripts-available/virt-what /usr/lib/observium_agent/scripts-enabled #ln -s /usr/lib/observium_agent/scripts-available/postfix_mailgraph /usr/lib/observium_agent/scripts-enabled
- Edit /etc/xinetd.d/observium_agent_xinetd so the Observium server is allowed to connect. You can do this by substituting 127.0.0.1, or place your IP after it, separated by a space. Make sure to restart xinetd afterwards so the configuration file is read.
sudo service xinetd restart
- Test from eyes
telnet $SERVER 36602 snmpwalk -v3 -u KeeklesSNMP -l authNoPriv -a MD5 -A keeklespasswd $SERVER