Difference between revisions of "ARRL Ransomware Attack"
| Line 11: | Line 11: | ||
[[File:ARRL Systems Service Disruption 2024-05-31 at 11.38.49 PM.png|thumb|left|500px|ARRL Systems Service Disruption 2024-05-31 at 11.38.49 PM]] | [[File:ARRL Systems Service Disruption 2024-05-31 at 11.38.49 PM.png|thumb|left|500px|ARRL Systems Service Disruption 2024-05-31 at 11.38.49 PM]] | ||
<br clear=all> | <br clear=all> | ||
| + | |||
| + | == [https://groups.io/g/SNJEMCOM/topic/update_on_arrl_outgage/106371089 2023-05-30 Updates from Mike Ritz, W7VO, ARRL 2nd VP]== | ||
| + | |||
| + | Based on his description of the email server below, it would appear my conclusions about them hosting exchange on prem were correct. | ||
| + | |||
| + | <blockquote> | ||
| + | This was reported by Chris NW6V | ||
| + | |||
| + | Mike Ritz, W7VO, ARRL 2nd Vice President, was at the WVDXC club meeting last night, having just attended an ARRL board meeting to discuss the outage. Mike reported those meetings are a weekly event until the matter is resolved. | ||
| + | |||
| + | Everything that was running on INTERNAL servers is down until further notice. That includes their VoIP phone system, their .org email addresses, and front ends for things like LOTW. Everything running on external servers - cloud servers etc. - including LOTW data, is believed unaffected. But, such data will not be available until the internal matters are resolved. Thus, "joe@arrl.org" doesn't work - because that was on their internal mail server. But "joe@arrl.net" does - because the relays didn't run internally. I checked, and NW6V@arrl.net works. | ||
| + | |||
| + | Efforts to restore the internal systems is proceeding full-time. No time-line can be given. The nature of the problem cannot be discussed. | ||
| + | |||
| + | I believe Mike said - with air asterisks around his words - "We have been advised to say nothing." He responded similarly when asked if "the Feds" were investigating this. | ||
| + | |||
| + | Being that the ARRL is connected to Homeland Security through its disaster response functions, and that personal data (no credit cards) for many relatively important persons are stored in the systems (business, military, science, etc.) such an investigation could very well involve the FBI and Homeland Security. | ||
| + | |||
| + | I (Chris NW6V) was an IT Director in mental healthcare for many years, so security was a big part of my responsibility. As a professional looking at it from the outside, this has all the earmarks of a hack - of sufficient severity that it needed to be reported as a CRIME. At which point, IT is required to lock everything down - every computer and device involved becomes EVIDENCE - until a full investigation by forensics experts - da cops - is conducted. Getting everything back up is NOT job 1. Once the "crime scene" is clear (yellow tape down), THEN the job of recovery can begin. If some kind of hack had wormed into the ARRL system, recovery of local system by restoring backups becomes problematic - it may be difficult to verify that backups contain no trace of the hack. In which case, recovery and restoration of services would be slow and very painful. | ||
| + | |||
| + | This fits what we know about the situation to a "T." | ||
| + | |||
| + | 73 Chris NW6V | ||
| + | </blockquote> | ||
=Systems affected= | =Systems affected= | ||
Revision as of 23:51, 31 May 2024
Just some notes on the Logbook of The Word (LoTW) cyber incident that happened during the Hamvention (May 15) 2024. [1]
Contents
updates
- Former Board member Ria Jairam, N2RJ confirms it was a ransomware attack on reddit: I was told it was a ransomware attack.
2024-05-31
2023-05-30 Updates from Mike Ritz, W7VO, ARRL 2nd VP
Based on his description of the email server below, it would appear my conclusions about them hosting exchange on prem were correct.
This was reported by Chris NW6V
Mike Ritz, W7VO, ARRL 2nd Vice President, was at the WVDXC club meeting last night, having just attended an ARRL board meeting to discuss the outage. Mike reported those meetings are a weekly event until the matter is resolved.
Everything that was running on INTERNAL servers is down until further notice. That includes their VoIP phone system, their .org email addresses, and front ends for things like LOTW. Everything running on external servers - cloud servers etc. - including LOTW data, is believed unaffected. But, such data will not be available until the internal matters are resolved. Thus, "joe@arrl.org" doesn't work - because that was on their internal mail server. But "joe@arrl.net" does - because the relays didn't run internally. I checked, and NW6V@arrl.net works.
Efforts to restore the internal systems is proceeding full-time. No time-line can be given. The nature of the problem cannot be discussed.
I believe Mike said - with air asterisks around his words - "We have been advised to say nothing." He responded similarly when asked if "the Feds" were investigating this.
Being that the ARRL is connected to Homeland Security through its disaster response functions, and that personal data (no credit cards) for many relatively important persons are stored in the systems (business, military, science, etc.) such an investigation could very well involve the FBI and Homeland Security.
I (Chris NW6V) was an IT Director in mental healthcare for many years, so security was a big part of my responsibility. As a professional looking at it from the outside, this has all the earmarks of a hack - of sufficient severity that it needed to be reported as a CRIME. At which point, IT is required to lock everything down - every computer and device involved becomes EVIDENCE - until a full investigation by forensics experts - da cops - is conducted. Getting everything back up is NOT job 1. Once the "crime scene" is clear (yellow tape down), THEN the job of recovery can begin. If some kind of hack had wormed into the ARRL system, recovery of local system by restoring backups becomes problematic - it may be difficult to verify that backups contain no trace of the hack. In which case, recovery and restoration of services would be slow and very painful.
This fits what we know about the situation to a "T."
73 Chris NW6V
Systems affected
- Logbook of The World, lotw.arrl.org
- ARRL Learning center, learn.arrl.org
- VEC processing systems
- W1AW bulletins
- Newsletter email service (reflector.arrl.org)
- email forwarding @arrl.net service was not effected, but config is unable to be pushed/updated
- ARRL Store - offline
- Phone systems are completely offline - online as of 2024-05-31
ARRL response
The ARRL didn't acknowledge this until about a day after it happened, mostly due that everyone was at the Dayton Hamvention. Per the logs on status.lotw.arrl.org the first outage was on May 14, 2024 at 1417[2]. It then came back and went hard down May 15, 2024 at 0500 AM.
I've saved the screenshot of their outage page here.
Now it would be the worst time for anything to go down is when everyone is traveling, and especially for an small non-profit org like the ARRL. Based on the timing, it's likely this was not a random coincidence. Who ever got in managed to take several of their systems offline simultaneously and picked the time to do it. Typically incidents begin months ahead of time now so that even if you have backups, they get in and are part of the backups.
I speculate they are cryptolocked and their active data has been locked. Restoring from backup is hard as you can't trust the backups, if they work. This is much harder in a windoze environment like the ARRL runs as well, since backups are harder and windoze is often hard to patch or left upgraded as old software must be maintained. Add in that it's a small non-profit, and it is a massive problem.
Background
The ARRL hosts most of these things on site at headquarters. We know this as they have 104.207.196.0/27 from Crown Castle delivered to them via fiber.
# start NetRange: 104.207.196.0 - 104.207.196.31 CIDR: 104.207.196.0/27 NetName: LTF--104-207-196-0-27 NetHandle: NET-104-207-196-0-1 Parent: CROWNCASTLE-104-207-192-0 (NET-104-207-192-0-1) NetType: Reassigned OriginAS: AS46887 Customer: ARRL (C07477604) RegDate: 2019-12-24 Updated: 2019-12-24 Comment: 1 Ref: https://rdap.arin.net/registry/ip/104.207.196.0 CustName: ARRL Address: 225 Main Street City: Newington StateProv: CT PostalCode: 06111-1494 Country: US RegDate: 2019-12-24 Updated: 2019-12-24 Ref: https://rdap.arin.net/registry/entity/C07477604
Based on some history we know that the ARRL uses 104.207.196.28 for outbound connections from their desktops/lan (a likely NAT device).
Looking at other things in this subnet, we know the following:
104.207.196.1 - a cisco, likely the providers box 104.207.196.2 - a Palo Alto (PTR is fw1.arrl.org) 104.207.196.4 - p1k.arrl.org (Online DXCC) 104.207.196.5 - reflector.arrl.org mailman 2.1.18 104.207.196.10 - irelp.arrl.org, some sort of Raspberry Pi device reporting Apache/2.4.38 (Raspbian) 104.207.196.11 - PTR dstar.arrl.org (not active) 104.207.196.12 - PTR beagle.arrl.org (not active) 104.207.196.14 - PTR subversion.arrl.org (not active) 104.207.196.17 - PTR lotw-test.arrl.org 104.207.196.18 - lotw.arrl.org Linux 104.207.196.19 - learn.arrl.org (but the PTR is exchange.arrl.org) a windoze box with Microsoft-IIS/10.0 win 10 104.207.196.20 - PTR centennial-qp.arrl.org (not active) 104.207.196.21 - reflectornew.arrl.org (not active) 104.207.196.22 - PTR npota.arrl.org. (not active) 104.207.196.23 - PTR igc.arrl.org. (not active) but was Apache 2.4.6 on centos 104.207.196.24 - exchange365.arrl.org. main windoze mail exchanger / outlook 104.207.196.27 - 104.207.196.27.lightower.net some box with RDP open on it. 104.207.196.29 - ARRL FTP service ftp1.arrl.org 104.207.196.30 - PTR fw2.arrl.org
Taken from here
We can reasonably assume the provider is handing off the entire /27 from their router, meaning there is no firewall or other device between the internet and the devices plugged in, and between the devices themselves. This last part is important as they likely can get into one device and then use it to attack the others via a local switch, as it's unlikely each device has it's own firewall port. Now they could be doing a layer2 firewall, and lets hope they are, but a firewall is just one part of this. There are several OS's there which should never be exposed to the internet and likely some are straddling the internal and external networks. This then becomes a jump point that can bypass their main NAT firewall.
History
Someone forwarded an email with a link to the Mexican goatse.cx mirror goatse.mx to the ARRL. The person at the league who received this email, blindly clicked on the link.
Below are the logs from apache on this mirror showing the 104.207.196.28 as the requesting IP.
104.207.196.28 - - [03/Apr/2024:08:05:53 -0500] "GET /hello.jpg HTTP/1.1" 200 21678 "https://goatse.mx/?hamsradio" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
So yes, we know the ARRL staff is prone to opening random links (and who the heck doesn't know goatse?)..
Mailman
On the 1st of april 2024 I was signed up for a mailing list with no notice by the ARRL. This was for the technical specialists and I think I had signed up for that over 10 years prior. Some how my email was still on file somewhere and added to a mailing list. Ok, I figured it out and then finally they posted what they want to do, but it took almost a day, and it was all boomer hams bitching about being unable to unsubscribe due to the mailing list being set to need admin approval to unsubscribe!
I noticed this was handled on via reflector.arrl.org, 104.207.196.5, and it was running a version of mailman which had know vulnerabilities in it. I emailed the admin at that time:
To: arrl-lab-collaborators-owner@reflector.arrl.org From: Bryan Fields <Bryan@bryanfields.net> Subject: you need to upgrade mailman now. Message-ID: <593f79d5-4870-f635-89c7-64024a8ece76@bryanfields.net> Date: Tue, 2 Apr 2024 09:39:53 -0400 User-Agent: Mutt/1.12.0 (2019-05-25) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit 2.1.18 is ancient and has a bunch of vulnerabilities. Example: https://bugs.launchpad.net/mailman/+bug/1952384 -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net
I ended up leaving this list, but revised the headers on the emails I have in light of the current issues. It would appear that this all went through the local mail exchange server exchange.arrl.org at 104.207.196.24. What's bad here is this is leaking the internal IP's of the local LAN at the ARRL, 10.1.128.190, and is running an old version of exchange 15.1.2242.12 from 2021. This could have been an attack vector and allowed an attacker to get into local desktops and perhaps find ssh keys or access to the other servers.
Received: from DS7PR05CA0030.namprd05.prod.outlook.com (2603:10b6:5:3b9::35) by CH0PR16MB5340.namprd16.prod.outlook.com (2603:10b6:610:192::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Mon, 1 Apr 2024 18:00:52 +0000 Received: from DS2PEPF00003443.namprd04.prod.outlook.com (2603:10b6:5:3b9:cafe::70) by DS7PR05CA0030.outlook.office365.com (2603:10b6:5:3b9::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.24 via Frontend Transport; Mon, 1 Apr 2024 18:00:52 +0000 Received: from exchange.arrl.org (104.207.196.24) by DS2PEPF00003443.mail.protection.outlook.com (10.167.17.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7452.22 via Frontend Transport; Mon, 1 Apr 2024 18:00:51 +0000 Received: from EXCHANGE365.ARRLHQ.ORG (10.1.128.190) by EXCHANGE365.ARRLHQ.ORG (10.1.128.190) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Mon, 1 Apr 2024 14:00:20 -0400 Received: from reflector.arrl.org (104.207.196.5) by EXCHANGE365.ARRLHQ.ORG (10.1.128.190) with Microsoft SMTP Server id 15.1.2242.12 via Frontend Transport; Mon, 1 Apr 2024 14:00:20 -0400 Received: from localhost.localdomain (localhost [127.0.0.1]) by reflector.arrl.org (Postfix) with ESMTP id 0195C1A0275 for <bryan@bryanfields.net>; Mon, 1 Apr 2024 14:00:20 -0400 (EDT) X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003443:EE_|CH0PR16MB5340:EE_ X-MS-Office365-Filtering-Correlation-Id: af0fef25-f944-4eb6-4002-08dc5275aaf2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CeqR/woHeH045VfwqIBZjGB6mHwOa3P7KaafscrtCiLd+iINXZbxKkP7g334TlyGJFEHq7E7+By1vNysNuhwCUtL5CLxnEteaY1Gd2gZOfhTngBZe4nJMx+/wD46kF9YgZ2qcEyo8mZinPl5Oj1AZ+mvduvho21Kn7WQrF7jRVCA/hUJ9/6v2Ues9T/KO45herKQJ7CsI3fXuGwFLEXDLjh8mpSaYyL6Nw/2mfWjuYLHWGWL10BZDZWdmDdKhXhJ3eysWE6E1WLQgKoD43fUu3+EdStmLasnlDNF6LBae+iK7sCkvzD8JBkvbLuB5LjAw6f+ZGAD1pBeOLRKag3qG4wtOq2zawoEYH0OJjq6yOzh78RkqM2HY39Rq6tXA4hJhAJ43CwIVaa9SFL5Wa2sHjwSxKx1cbAJHGAKGbPTCQ0mnHOfbNLdQvIbJb0y5+0dGJebrRl19OX3iUQ7fKz6IWw2miJYMK8n+3BO+WgOxnhS/vXrn3m01yiVuNTLkMjbsMMLTIkWFfTcm77aafQGN++nYGBRncL+xbzyo4LOo1iJcUB9LN9y7OZx12Miv7qXPqYKXSEc4T+QF0lJTh7XMzU0kgdlDmxeer0i+6bTDSJIvHEMYM2YMjVWL/gCpby9WgxSMNFEErXlPIMazx8lXTqHPInYWH8WQMVAzDK2OPlXnlRwGQ3RHBD3OshJ/Il6ipoujfTSr+P9ropPgjBaZQ== X-Forefront-Antispam-Report: CIP:104.207.196.24;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:exchange.arrl.org;PTR:exchange365.arrl.org;CAT:NONE;SFS:(13230031)(376005)(82310400014)(36860700004)(61400799018);DIR:OUT;SFP:1102; X-OriginatorOrg: reflector.arrl.org X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Apr 2024 18:00:51.1328 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: af0fef25-f944-4eb6-4002-08dc5275aaf2 X-MS-Exchange-CrossTenant-Id: baa21d0f-7301-40fe-bdd6-03d443a8c218 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=baa21d0f-7301-40fe-bdd6-03d443a8c218;Ip=[104.207.196.24];Helo=[exchange.arrl.org] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003443.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR16MB5340
Thoughts
The ARRL really needs to step up their game on the IT security but more importantly us members need a good amount of updates on this. Be honest and forthright about what's happened. While it's bad, it's the fault of the blackhats who did this. If there's a ransom to be paid, the members need to be informed of this, even if covered by insurance.
The league must present a root cause report of this so we can learn from it. Anything less will be viewed as a coverup by the members.
- ↑ https://www.arrl.org/news/arrl-systems-service-disruption
- ↑ all times are Eastern, GMT-4
