Difference between revisions of "ARRL Ransomware Attack"
| (6 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
[[Category:ARRL]] | [[Category:ARRL]] | ||
Just some notes on the Logbook of The Word (LoTW) <s>cyber incident</s> [https://en.wikipedia.org/wiki/Ransomware ransomware attack] that happened during the Hamvention (May 15) 2024. <ref>https://www.arrl.org/news/arrl-systems-service-disruption</ref> | Just some notes on the Logbook of The Word (LoTW) <s>cyber incident</s> [https://en.wikipedia.org/wiki/Ransomware ransomware attack] that happened during the Hamvention (May 15) 2024. <ref>https://www.arrl.org/news/arrl-systems-service-disruption</ref> | ||
| + | |||
| + | |||
| + | '''The ARRL confirmed it paid a million dollar ransom to the criminals behind this.''' | ||
=Systems affected= | =Systems affected= | ||
| − | * Logbook of The World, lotw.arrl.org | + | * Logbook of The World, [https://lotw.arrl.org lotw.arrl.org] - Online as of 2024-07-02 |
* [https://www.arrl.org/learning-center-help ARRL Learning center], learn.arrl.org | * [https://www.arrl.org/learning-center-help ARRL Learning center], learn.arrl.org | ||
* VEC processing systems - Per reddit<ref>https://old.reddit.com/r/amateurradio/comments/1d80hy0/update_arrl_services_disruption/</ref> these are coming back online and processing is starting as of 2024-06-04 | * VEC processing systems - Per reddit<ref>https://old.reddit.com/r/amateurradio/comments/1d80hy0/update_arrl_services_disruption/</ref> these are coming back online and processing is starting as of 2024-06-04 | ||
* W1AW bulletins | * W1AW bulletins | ||
| − | * Newsletter email service (reflector.arrl.org) | + | * Newsletter email service ([http://reflector.arrl.org reflector.arrl.org]) - still down as of 2024-08-22 |
* email forwarding @arrl.net service was not effected, but config is unable to be pushed/updated | * email forwarding @arrl.net service was not effected, but config is unable to be pushed/updated | ||
| − | * ARRL Store - | + | * ARRL Store - online as of 2024-06-06 |
* Phone systems are completely offline - online as of 2024-05-31 | * Phone systems are completely offline - online as of 2024-05-31 | ||
= Updates = | = Updates = | ||
| + | |||
| + | == 2024-08-21 == | ||
| + | |||
| + | '''The League confirmed they paid a ransom.''' | ||
| + | |||
| + | Fullstop. | ||
| + | |||
| + | This is an egregious misuse of member funds, and goes on to fund more of these attacks. These payments directly finance criminals and fund state level actors. | ||
| + | It doesn't matter if insurance paid a portion of this, as it will affect insurance premiums in the future, and set precedence that we will pay. | ||
| + | |||
| + | One of the other things I must take issue with is the League referring to services as "member benefits". This passive language changes a service we members have into something that's a nice-to-have, and be lucky they bestow it upon us. | ||
| + | |||
| + | <blockquote> | ||
| + | August 21, 2024 | ||
| + | |||
| + | ARRL IT Security Incident - Report to Members | ||
| + | |||
| + | Sometime in early May 2024, ARRL’s systems network was compromised by threat actors (TAs) using information they had purchased on the dark web. The TAs accessed headquarters on-site systems and most cloud-based systems. They used a wide variety of payloads affecting everything from desktops and laptops to Windows-based and Linux-based servers. Despite the wide variety of target configurations, the TAs seemed to have a payload that would host and execute encryption or deletion of network-based IT assets, as well as launch demands for a ransom payment, for every system. | ||
| + | |||
| + | This serious incident was an act of organized crime. The highly coordinated and executed attack took place during the early morning hours of May 15. That morning, as staff arrived, it was immediately apparent that ARRL had become the victim of an extensive and sophisticated ransomware attack. The FBI categorized the attack as “unique” as they had not seen this level of sophistication among the many other attacks, they have experience with. Within 3 hours a crisis management team had been constructed of ARRL management, an outside vendor with extensive resources and experience in the ransomware recovery space, attorneys experienced with managing the legal aspects of the attack including interfacing with the authorities, and our insurance carrier. The authorities were contacted immediately as was the ARRL President. | ||
| + | |||
| + | The ransom demands by the TAs, in exchange for access to their decryption tools, were exorbitant. It was clear they didn’t know, and didn’t care, that they had attacked a small 501(c)(3) organization with limited resources. Their ransom demands were dramatically weakened by the fact that they did not have access to any compromising data. It was also clear that they believed ARRL had extensive insurance coverage that would cover a multi-million-dollar ransom payment. After days of tense negotiation and brinkmanship, '''ARRL agreed to pay a $1 million ransom.''' That payment, along with the cost of restoration, has been largely covered by our insurance policy. | ||
| + | |||
| + | From the start of the incident, the ARRL board met weekly using a continuing special board meeting for full progress reports and to offer assistance. In the first few meetings there were significant details to cover, and the board was thoughtfully engaged, asked important questions, and was fully supportive of the team at HQ to keep the restoration efforts moving. Member updates were posted to a single page on the website and were posted across the internet in many forums and groups. ARRL worked closely with professionals deeply experienced in ransomware matters on every post. It is important to understand that the TAs had ARRL under a magnifying glass while we were negotiating. Based on the expert advice we were being given, we could not publicly communicate anything informative, useful, or potentially antagonistic to the TAs during this time frame. | ||
| + | |||
| + | Today, most systems have been restored or are waiting for interfaces to come back online to interconnect them. While we have been in restoration mode, we have also been working to simplify the infrastructure to the extent possible. We anticipate that it may take another month or two to complete restoration under the new infrastructure guidelines and new standards. | ||
| + | |||
| + | Most ARRL member benefits remained operational during the attack. One that wasn’t was Logbook of The World (LoTW), which is one of our most popular member benefits. LoTW data was not impacted by the attack and once the environment was ready to again permit public access to ARRL network-based servers, we returned LoTW into service. The fact that LoTW took less than 4 days to get through a backlog that at times exceeded over 60,000 logs was outstanding. | ||
| + | |||
| + | The board at the ARRL Second Board Meeting in July voted to approve a new committee, the Information Technology Advisory Committee. This will be comprised of ARRL staff, board members with demonstrated experience in IT, and additional members from the IT industry who are currently employed as subject matter experts in a few areas. They will help analyze and advise on future steps to take with ARRL IT within the financial means available to the organization. | ||
| + | |||
| + | We thank you for your patience as we navigated our way through this. The emails of moral support and offers of IT expertise were well received by the team. Although we are not entirely out of the woods yet and are still working to restore minor servers that serve internal needs (such as various email services like bulk mail and some internal reflectors), we are happy with the progress that has been made and for the incredible dedication of staff and consultants who continue to work together to bring this incident to a successful conclusion. | ||
| + | |||
| + | Copyright © 2024 American Radio Relay League, Incorporated. Use and distribution of this publication, or any portion thereof, is permitted for non-commercial or educational purposes, with attribution. All other purposes require written permission. | ||
| + | </blockquote> | ||
| + | |||
| + | |||
| + | == 2024-07-23 ARRL Board meeting == | ||
| + | |||
| + | This was an update on the board meeting, and the league is establishing a board committee to be responsible for IT systems. As the ARRL has doubled down on board secrecy we'll never know if this is effective. It's interesting that the CEO (Dave Minster) went along with this as it's a clear conflict with powers under the CEO. It would give a scapegoat should any IT failures happen again. | ||
| + | |||
| + | <blockquote> | ||
| + | • The Board requested the President establish the ARRL Information Technology Advisory Committee (ITAC), acknowledging the importance of providing information technology-based services to ARRL members, and to improve and protect ARRL systems. The ITAC will engage experts in the field of IT to help oversee the development, operations, and evolution of the ARRL IT system infrastructure and applications. | ||
| + | • The Board reassigned the Logbook of The World® (LoTW®) Maintenance Committee to the newly established Information Technology Advisory Committee (ITAC). ITAC will support the Maintenance Committee’s responsibilities with LoTW® development, including technical changes and future upgrades to the system. | ||
| + | </blockquote> | ||
| + | |||
| + | ==2024-07-03 == | ||
| + | |||
| + | [http://www.arrl.org/news/arrl-vec-services-update-during-systems-disruption They now have a page updating on the VEC status.] | ||
| + | |||
| + | <blockquote> | ||
| + | We are currently unable to create International Amateur Radio Permits, License Class Certificates, and Volunteer Examiner (VE) badges, certificates, and stickers. New ARRL VE applications and renewals are unable to be processed at this time. | ||
| + | </blockquote> | ||
| + | |||
| + | == 2024-07-01 == | ||
| + | |||
| + | Update from the ARRL. LoTW is back in service. Other systems remain down. | ||
| + | |||
| + | <pre> | ||
| + | July 1, 2024 | ||
| + | |||
| + | MEMBER BULLETIN | ||
| + | |||
| + | July 1, 2024 | ||
| + | |||
| + | Dear ARRL member, | ||
| + | |||
| + | Effective 12:00pm ET / 16:00 UTC we will be returning Logbook of The World® (LoTW®) to service. | ||
| + | |||
| + | As work progressed on the network, some users encountered LoTW opening briefly during which some 6600 logs were uploaded. The logs were not processed until this weekend as we tested that the interfaces to LoTW were functioning properly. | ||
| + | We are taking steps to help manage what will likely be a huge influx of logs. We are requesting that if you have large uploads, perhaps from contests or from a DXpedition, please wait a week or two before uploading to give LoTW a chance to catch up. We have also implemented a process to reject logs with excessive duplicates. Please do not upload your entire log to “ensure” your contacts are in LoTW as they will be rejected. Lastly, please do not call ARRL Headquarters to report issues you are having with LoTW. You can contact support at LoTW-help@arrl.org [ mailto:LoTW-help@arrl.org ] . | ||
| + | |||
| + | Through the end of the year, you may experience planned times when LoTW will be unavailable. We have been using this time to evaluate operational and infrastructure improvements we would like to make to LoTW. Those times will be announced. | ||
| + | We appreciate your patience as we worked through the challenges keeping LoTW from returning to service. We know the importance of LoTW to our members, and to the tens of thousands of LoTW users who are not ARRL members. LoTW, just behind QST, is our second most popular ARRL benefit. | ||
| + | |||
| + | Copyright © 2024 American Radio Relay League, Incorporated. Use and distribution of this publication, or any portion thereof, is permitted for non-commercial or educational purposes, with attribution. All other purposes require written permission. | ||
| + | </pre> | ||
| + | |||
| + | == 2024-06-14 == | ||
| + | |||
| + | The ARRL now states the https://learn.arrl.org is online and accessible. What's new is it's behind cloudflare, so we can't really tell if they ditched windoze or not. comparing it to the prior site, it still looks the same, so likely not. There is simply no good reason to use windoze with IIS. | ||
| + | |||
== 2024-06-07 == | == 2024-06-07 == | ||
Latest revision as of 11:42, 8 September 2024
Just some notes on the Logbook of The Word (LoTW) cyber incident ransomware attack that happened during the Hamvention (May 15) 2024. [1]
The ARRL confirmed it paid a million dollar ransom to the criminals behind this.
Contents
- 1 Systems affected
- 2 Updates
- 2.1 2024-08-21
- 2.2 2024-07-23 ARRL Board meeting
- 2.3 2024-07-03
- 2.4 2024-07-01
- 2.5 2024-06-14
- 2.6 2024-06-07
- 2.7 2024-06-06
- 2.8 2024-06-04
- 2.9 * Former Board member Ria Jairam, N2RJ confirms it was a ransomware attack on reddit: I was told it was a ransomware attack.
- 2.10 2024-05-31
- 2.11 2023-05-30 Updates from Mike Ritz, W7VO, ARRL 2nd VP
- 3 ARRL response
- 4 Background
- 5 Thoughts
Systems affected
- Logbook of The World, lotw.arrl.org - Online as of 2024-07-02
- ARRL Learning center, learn.arrl.org
- VEC processing systems - Per reddit[2] these are coming back online and processing is starting as of 2024-06-04
- W1AW bulletins
- Newsletter email service (reflector.arrl.org) - still down as of 2024-08-22
- email forwarding @arrl.net service was not effected, but config is unable to be pushed/updated
- ARRL Store - online as of 2024-06-06
- Phone systems are completely offline - online as of 2024-05-31
Updates
2024-08-21
The League confirmed they paid a ransom.
Fullstop.
This is an egregious misuse of member funds, and goes on to fund more of these attacks. These payments directly finance criminals and fund state level actors. It doesn't matter if insurance paid a portion of this, as it will affect insurance premiums in the future, and set precedence that we will pay.
One of the other things I must take issue with is the League referring to services as "member benefits". This passive language changes a service we members have into something that's a nice-to-have, and be lucky they bestow it upon us.
August 21, 2024
ARRL IT Security Incident - Report to Members
Sometime in early May 2024, ARRL’s systems network was compromised by threat actors (TAs) using information they had purchased on the dark web. The TAs accessed headquarters on-site systems and most cloud-based systems. They used a wide variety of payloads affecting everything from desktops and laptops to Windows-based and Linux-based servers. Despite the wide variety of target configurations, the TAs seemed to have a payload that would host and execute encryption or deletion of network-based IT assets, as well as launch demands for a ransom payment, for every system.
This serious incident was an act of organized crime. The highly coordinated and executed attack took place during the early morning hours of May 15. That morning, as staff arrived, it was immediately apparent that ARRL had become the victim of an extensive and sophisticated ransomware attack. The FBI categorized the attack as “unique” as they had not seen this level of sophistication among the many other attacks, they have experience with. Within 3 hours a crisis management team had been constructed of ARRL management, an outside vendor with extensive resources and experience in the ransomware recovery space, attorneys experienced with managing the legal aspects of the attack including interfacing with the authorities, and our insurance carrier. The authorities were contacted immediately as was the ARRL President.
The ransom demands by the TAs, in exchange for access to their decryption tools, were exorbitant. It was clear they didn’t know, and didn’t care, that they had attacked a small 501(c)(3) organization with limited resources. Their ransom demands were dramatically weakened by the fact that they did not have access to any compromising data. It was also clear that they believed ARRL had extensive insurance coverage that would cover a multi-million-dollar ransom payment. After days of tense negotiation and brinkmanship, ARRL agreed to pay a $1 million ransom. That payment, along with the cost of restoration, has been largely covered by our insurance policy.
From the start of the incident, the ARRL board met weekly using a continuing special board meeting for full progress reports and to offer assistance. In the first few meetings there were significant details to cover, and the board was thoughtfully engaged, asked important questions, and was fully supportive of the team at HQ to keep the restoration efforts moving. Member updates were posted to a single page on the website and were posted across the internet in many forums and groups. ARRL worked closely with professionals deeply experienced in ransomware matters on every post. It is important to understand that the TAs had ARRL under a magnifying glass while we were negotiating. Based on the expert advice we were being given, we could not publicly communicate anything informative, useful, or potentially antagonistic to the TAs during this time frame.
Today, most systems have been restored or are waiting for interfaces to come back online to interconnect them. While we have been in restoration mode, we have also been working to simplify the infrastructure to the extent possible. We anticipate that it may take another month or two to complete restoration under the new infrastructure guidelines and new standards.
Most ARRL member benefits remained operational during the attack. One that wasn’t was Logbook of The World (LoTW), which is one of our most popular member benefits. LoTW data was not impacted by the attack and once the environment was ready to again permit public access to ARRL network-based servers, we returned LoTW into service. The fact that LoTW took less than 4 days to get through a backlog that at times exceeded over 60,000 logs was outstanding.
The board at the ARRL Second Board Meeting in July voted to approve a new committee, the Information Technology Advisory Committee. This will be comprised of ARRL staff, board members with demonstrated experience in IT, and additional members from the IT industry who are currently employed as subject matter experts in a few areas. They will help analyze and advise on future steps to take with ARRL IT within the financial means available to the organization.
We thank you for your patience as we navigated our way through this. The emails of moral support and offers of IT expertise were well received by the team. Although we are not entirely out of the woods yet and are still working to restore minor servers that serve internal needs (such as various email services like bulk mail and some internal reflectors), we are happy with the progress that has been made and for the incredible dedication of staff and consultants who continue to work together to bring this incident to a successful conclusion.
Copyright © 2024 American Radio Relay League, Incorporated. Use and distribution of this publication, or any portion thereof, is permitted for non-commercial or educational purposes, with attribution. All other purposes require written permission.
2024-07-23 ARRL Board meeting
This was an update on the board meeting, and the league is establishing a board committee to be responsible for IT systems. As the ARRL has doubled down on board secrecy we'll never know if this is effective. It's interesting that the CEO (Dave Minster) went along with this as it's a clear conflict with powers under the CEO. It would give a scapegoat should any IT failures happen again.
• The Board requested the President establish the ARRL Information Technology Advisory Committee (ITAC), acknowledging the importance of providing information technology-based services to ARRL members, and to improve and protect ARRL systems. The ITAC will engage experts in the field of IT to help oversee the development, operations, and evolution of the ARRL IT system infrastructure and applications. • The Board reassigned the Logbook of The World® (LoTW®) Maintenance Committee to the newly established Information Technology Advisory Committee (ITAC). ITAC will support the Maintenance Committee’s responsibilities with LoTW® development, including technical changes and future upgrades to the system.
2024-07-03
They now have a page updating on the VEC status.
We are currently unable to create International Amateur Radio Permits, License Class Certificates, and Volunteer Examiner (VE) badges, certificates, and stickers. New ARRL VE applications and renewals are unable to be processed at this time.
2024-07-01
Update from the ARRL. LoTW is back in service. Other systems remain down.
July 1, 2024 MEMBER BULLETIN July 1, 2024 Dear ARRL member, Effective 12:00pm ET / 16:00 UTC we will be returning Logbook of The World® (LoTW®) to service. As work progressed on the network, some users encountered LoTW opening briefly during which some 6600 logs were uploaded. The logs were not processed until this weekend as we tested that the interfaces to LoTW were functioning properly. We are taking steps to help manage what will likely be a huge influx of logs. We are requesting that if you have large uploads, perhaps from contests or from a DXpedition, please wait a week or two before uploading to give LoTW a chance to catch up. We have also implemented a process to reject logs with excessive duplicates. Please do not upload your entire log to “ensure” your contacts are in LoTW as they will be rejected. Lastly, please do not call ARRL Headquarters to report issues you are having with LoTW. You can contact support at LoTW-help@arrl.org [ mailto:LoTW-help@arrl.org ] . Through the end of the year, you may experience planned times when LoTW will be unavailable. We have been using this time to evaluate operational and infrastructure improvements we would like to make to LoTW. Those times will be announced. We appreciate your patience as we worked through the challenges keeping LoTW from returning to service. We know the importance of LoTW to our members, and to the tens of thousands of LoTW users who are not ARRL members. LoTW, just behind QST, is our second most popular ARRL benefit. Copyright © 2024 American Radio Relay League, Incorporated. Use and distribution of this publication, or any portion thereof, is permitted for non-commercial or educational purposes, with attribution. All other purposes require written permission.
2024-06-14
The ARRL now states the https://learn.arrl.org is online and accessible. What's new is it's behind cloudflare, so we can't really tell if they ditched windoze or not. comparing it to the prior site, it still looks the same, so likely not. There is simply no good reason to use windoze with IIS.
2024-06-07
The following was posted by AA6YQ regarding the architecture of the logbook and DXCC systems. Apparently they use a FoxPro DB, which isn't a name I've heard in years.
The LoTW Server utilizes an SAP database engine.
Note that LoTW does not manage DXCC award credits, VUCC award credits, or WAS award credits. Those are each managed by a dedicated system with which LoTW interoperates.
The DXCC system employs a FoxPro database. The ARRL has twice attempted to re-implement the DXCC system since 2016; both attempts failed.
de AA6YQ
2024-06-06
The email below was sent to members. Note the many odd characters are in the source message, they were not added by me. This makes me question the industry experts they have retained in this matter.
The statement is vague and no mention if ARRL funds are being used for payment of any expenses/ransom related to this. Even if the insurance is paying, it's a given we will likely be dropped for renewal and see premiums increase, perhaps to unaffordable levels.
June 6, 2024 MEMBER BULLETIN  June 6, 2024  Dear ARRL member,  We are writing to inform you of a serious incident that has impacted ARRL’s operations. On or around May 12, 2024, ARRL was the victim of a sophisticated network attack by a malicious international cyber group. We immediately involved the FBI and engaged with third party experts to investigate.  On May 16, on the eve of the ARRL National Convention in Ohio, we posted a notice on our website to inform members about the incident. Since then, we have made substantial progress to mitigate the impact of this attack on our organization. We have been posting regular updates, including the status of restored services. Please refer to our dedicated news post at https://www.arrl.org/news/arrl-systems-service-disruption [ https://www.arrl.org/news/arrl-systems-service-disruption ] .  We are aware that certain members believe we should be openly communicating everything associated with this incident. We are working with industry experts, including cyber crime attorneys and the authorities, who have directed us to be conservative and cautious with our communications while restoring the ARRL network.  Many of our services and programs have been impacted by this attack. We have been diligently assessing each system to ascertain the extent of compromise. For example, while the Logbook of The World® server and related user data were unaffected, we have taken the precautionary measure of keeping the service offline until we can ensure the security and integrity of our networks. Similarly, access to Online DXCC is unavailable, although individual award data remains secure.  Fortunately, some of our key systems, such as the ARRL website and our association membership system, were unaffected. Despite the severity of the attack, no personal information was compromised. Additionally, ARRL does not store credit card information anywhere on our systems, and we do not collect social security numbers.  Our Directors have heard from some members who are concerned about managing their membership renewals. Members can renew online at www.arrl.org/join [ http://www.arrl.org/join ] or by phoning ARRL.  We understand the frustration and inconvenience this incident has caused, and we sincerely appreciate your patience and understanding as our dedicated staff and partners continue to work tirelessly to restore affected systems and services.  Thank you for your ongoing support. Copyright © 2024 American Radio Relay League, Incorporated. Use and distribution of this publication, or any portion thereof, is permitted for non-commercial or educational purposes, with attribution. All other purposes require written permission. ARRL Black Diamond Logo [http://www.arrl.org] ARRL 225 Main Street, Newington, CT 06111-1400 www.arrl.org [ http://www.arrl.org ] · 860-594-0200 ARRL Facebook [https://www.facebook.com/ARRL.org] ARRL Instagram [https://www.instagram.com/arrlhq/] ARRL Twitter [https://arrl.informz.net/InformzDataService/OnlineVersion/Individual?mailingInstanceId=3055839&subscriberId=520503687] ARRL YouTube [https://www.youtube.com/user/ARRLHQ] ARRL LinkedIn [https://www.linkedin.com/company/american-radio-relay-league] ARRL The National Association for Amateur Radio® Unsubscribe [ http://arrl.informz.net/arrl/pages/Copy_of_Copy_of_Unsubscribe_Test_GF?_zs=Tbhxl&_zmi=rMeF ]
2024-06-04
On or around May 12, 2024, ARRL was the victim of a sophisticated network attack by a malicious international cyber group. ARRL immediately involved the FBI and engaged with third party experts to investigate.
This serious incident was extensive and categorized by the FBI as “unique,” compromising network devices, servers, cloud-based systems, and PCs.
ARRL management quickly established an incident response team. This has led to an extensive effort to contain and remediate the networks, restore servers, and staff are beginning the testing of applications and interfaces to ensure proper operation.
Thank you for your patience and understanding as our staff continue to work through this with an outstanding team of experts to restore full functionality to our systems and services.
We will continue to update members as advised and to the extent we are able.
No mention about the payment of any ransom or how this is being paid for, ie insurance. It looks like this was exchange being pwnd and everything else from there.
Do you think the League will migrate off of windoze and exchange or go right back to it? (lol, of course they will throw good money after bad and go right back to it.)
* Former Board member Ria Jairam, N2RJ confirms it was a ransomware attack on reddit: I was told it was a ransomware attack.
2024-05-31
2023-05-30 Updates from Mike Ritz, W7VO, ARRL 2nd VP
Based on his description of the email server below, it would appear my conclusions about them hosting exchange on prem were correct.
This was reported by Chris NW6V
Mike Ritz, W7VO, ARRL 2nd Vice President, was at the WVDXC club meeting last night, having just attended an ARRL board meeting to discuss the outage. Mike reported those meetings are a weekly event until the matter is resolved.
Everything that was running on INTERNAL servers is down until further notice. That includes their VoIP phone system, their .org email addresses, and front ends for things like LOTW. Everything running on external servers - cloud servers etc. - including LOTW data, is believed unaffected. But, such data will not be available until the internal matters are resolved. Thus, "joe@arrl.org" doesn't work - because that was on their internal mail server. But "joe@arrl.net" does - because the relays didn't run internally. I checked, and NW6V@arrl.net works.
Efforts to restore the internal systems is proceeding full-time. No time-line can be given. The nature of the problem cannot be discussed.
I believe Mike said - with air asterisks around his words - "We have been advised to say nothing." He responded similarly when asked if "the Feds" were investigating this.
Being that the ARRL is connected to Homeland Security through its disaster response functions, and that personal data (no credit cards) for many relatively important persons are stored in the systems (business, military, science, etc.) such an investigation could very well involve the FBI and Homeland Security.
I (Chris NW6V) was an IT Director in mental healthcare for many years, so security was a big part of my responsibility. As a professional looking at it from the outside, this has all the earmarks of a hack - of sufficient severity that it needed to be reported as a CRIME. At which point, IT is required to lock everything down - every computer and device involved becomes EVIDENCE - until a full investigation by forensics experts - da cops - is conducted. Getting everything back up is NOT job 1. Once the "crime scene" is clear (yellow tape down), THEN the job of recovery can begin. If some kind of hack had wormed into the ARRL system, recovery of local system by restoring backups becomes problematic - it may be difficult to verify that backups contain no trace of the hack. In which case, recovery and restoration of services would be slow and very painful.
This fits what we know about the situation to a "T."
73 Chris NW6V
ARRL response
The ARRL didn't acknowledge this until about a day after it happened, mostly due that everyone was at the Dayton Hamvention. Per the logs on status.lotw.arrl.org the first outage was on May 14, 2024 at 1417[3]. It then came back and went hard down May 15, 2024 at 0500 AM.
I've saved the screenshot of their outage page here.
Now it would be the worst time for anything to go down is when everyone is traveling, and especially for an small non-profit org like the ARRL. Based on the timing, it's likely this was not a random coincidence. Who ever got in managed to take several of their systems offline simultaneously and picked the time to do it. Typically incidents begin months ahead of time now so that even if you have backups, they get in and are part of the backups.
I speculate they are cryptolocked and their active data has been locked. Restoring from backup is hard as you can't trust the backups, if they work. This is much harder in a windoze environment like the ARRL runs as well, since backups are harder and windoze is often hard to patch or left upgraded as old software must be maintained. Add in that it's a small non-profit, and it is a massive problem.
Background
The ARRL hosts most of these things on site at headquarters. We know this as they have 104.207.196.0/27 from Crown Castle delivered to them via fiber.
# start NetRange: 104.207.196.0 - 104.207.196.31 CIDR: 104.207.196.0/27 NetName: LTF--104-207-196-0-27 NetHandle: NET-104-207-196-0-1 Parent: CROWNCASTLE-104-207-192-0 (NET-104-207-192-0-1) NetType: Reassigned OriginAS: AS46887 Customer: ARRL (C07477604) RegDate: 2019-12-24 Updated: 2019-12-24 Comment: 1 Ref: https://rdap.arin.net/registry/ip/104.207.196.0 CustName: ARRL Address: 225 Main Street City: Newington StateProv: CT PostalCode: 06111-1494 Country: US RegDate: 2019-12-24 Updated: 2019-12-24 Ref: https://rdap.arin.net/registry/entity/C07477604
Based on some history we know that the ARRL uses 104.207.196.28 for outbound connections from their desktops/lan (a likely NAT device).
Looking at other things in this subnet, we know the following:
104.207.196.1 - a cisco, likely the providers box 104.207.196.2 - a Palo Alto (PTR is fw1.arrl.org) 104.207.196.4 - p1k.arrl.org (Online DXCC) 104.207.196.5 - reflector.arrl.org mailman 2.1.18 104.207.196.10 - irelp.arrl.org, some sort of Raspberry Pi device reporting Apache/2.4.38 (Raspbian) 104.207.196.11 - PTR dstar.arrl.org (not active) 104.207.196.12 - PTR beagle.arrl.org (not active) 104.207.196.14 - PTR subversion.arrl.org (not active) 104.207.196.17 - PTR lotw-test.arrl.org 104.207.196.18 - lotw.arrl.org Linux 104.207.196.19 - learn.arrl.org (but the PTR is exchange.arrl.org) a windoze box with Microsoft-IIS/10.0 win 10 104.207.196.20 - PTR centennial-qp.arrl.org (not active) 104.207.196.21 - reflectornew.arrl.org (not active) 104.207.196.22 - PTR npota.arrl.org. (not active) 104.207.196.23 - PTR igc.arrl.org. (not active) but was Apache 2.4.6 on centos 104.207.196.24 - exchange365.arrl.org. main windoze mail exchanger / outlook 104.207.196.27 - 104.207.196.27.lightower.net some box with RDP open on it. 104.207.196.29 - ARRL FTP service ftp1.arrl.org 104.207.196.30 - PTR fw2.arrl.org
Taken from here
We can reasonably assume the provider is handing off the entire /27 from their router, meaning there is no firewall or other device between the internet and the devices plugged in, and between the devices themselves. This last part is important as they likely can get into one device and then use it to attack the others via a local switch, as it's unlikely each device has it's own firewall port. Now they could be doing a layer2 firewall, and lets hope they are, but a firewall is just one part of this. There are several OS's there which should never be exposed to the internet and likely some are straddling the internal and external networks. This then becomes a jump point that can bypass their main NAT firewall.
History
Someone forwarded an email with a link to the Mexican goatse.cx mirror goatse.mx to the ARRL. The person at the league who received this email, blindly clicked on the link.
Below are the logs from apache on this mirror showing the 104.207.196.28 as the requesting IP.
104.207.196.28 - - [03/Apr/2024:08:05:53 -0500] "GET /hello.jpg HTTP/1.1" 200 21678 "https://goatse.mx/?hamsradio" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"
So yes, we know the ARRL staff is prone to opening random links (and who the heck doesn't know goatse?)..
Mailman
On the 1st of april 2024 I was signed up for a mailing list with no notice by the ARRL. This was for the technical specialists and I think I had signed up for that over 10 years prior. Some how my email was still on file somewhere and added to a mailing list. Ok, I figured it out and then finally they posted what they want to do, but it took almost a day, and it was all boomer hams bitching about being unable to unsubscribe due to the mailing list being set to need admin approval to unsubscribe!
I noticed this was handled on via reflector.arrl.org, 104.207.196.5, and it was running a version of mailman which had know vulnerabilities in it. I emailed the admin at that time:
To: arrl-lab-collaborators-owner@reflector.arrl.org From: Bryan Fields <Bryan@bryanfields.net> Subject: you need to upgrade mailman now. Message-ID: <593f79d5-4870-f635-89c7-64024a8ece76@bryanfields.net> Date: Tue, 2 Apr 2024 09:39:53 -0400 User-Agent: Mutt/1.12.0 (2019-05-25) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit 2.1.18 is ancient and has a bunch of vulnerabilities. Example: https://bugs.launchpad.net/mailman/+bug/1952384 -- Bryan Fields 727-409-1194 - Voice http://bryanfields.net
I ended up leaving this list, but revised the headers on the emails I have in light of the current issues. It would appear that this all went through the local mail exchange server exchange.arrl.org at 104.207.196.24. What's bad here is this is leaking the internal IP's of the local LAN at the ARRL, 10.1.128.190, and is running an old version of exchange 15.1.2242.12 from 2021. This could have been an attack vector and allowed an attacker to get into local desktops and perhaps find ssh keys or access to the other servers.
Received: from DS7PR05CA0030.namprd05.prod.outlook.com (2603:10b6:5:3b9::35) by CH0PR16MB5340.namprd16.prod.outlook.com (2603:10b6:610:192::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7409.46; Mon, 1 Apr 2024 18:00:52 +0000 Received: from DS2PEPF00003443.namprd04.prod.outlook.com (2603:10b6:5:3b9:cafe::70) by DS7PR05CA0030.outlook.office365.com (2603:10b6:5:3b9::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7452.24 via Frontend Transport; Mon, 1 Apr 2024 18:00:52 +0000 Received: from exchange.arrl.org (104.207.196.24) by DS2PEPF00003443.mail.protection.outlook.com (10.167.17.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7452.22 via Frontend Transport; Mon, 1 Apr 2024 18:00:51 +0000 Received: from EXCHANGE365.ARRLHQ.ORG (10.1.128.190) by EXCHANGE365.ARRLHQ.ORG (10.1.128.190) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Mon, 1 Apr 2024 14:00:20 -0400 Received: from reflector.arrl.org (104.207.196.5) by EXCHANGE365.ARRLHQ.ORG (10.1.128.190) with Microsoft SMTP Server id 15.1.2242.12 via Frontend Transport; Mon, 1 Apr 2024 14:00:20 -0400 Received: from localhost.localdomain (localhost [127.0.0.1]) by reflector.arrl.org (Postfix) with ESMTP id 0195C1A0275 for <bryan@bryanfields.net>; Mon, 1 Apr 2024 14:00:20 -0400 (EDT) X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DS2PEPF00003443:EE_|CH0PR16MB5340:EE_ X-MS-Office365-Filtering-Correlation-Id: af0fef25-f944-4eb6-4002-08dc5275aaf2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CeqR/woHeH045VfwqIBZjGB6mHwOa3P7KaafscrtCiLd+iINXZbxKkP7g334TlyGJFEHq7E7+By1vNysNuhwCUtL5CLxnEteaY1Gd2gZOfhTngBZe4nJMx+/wD46kF9YgZ2qcEyo8mZinPl5Oj1AZ+mvduvho21Kn7WQrF7jRVCA/hUJ9/6v2Ues9T/KO45herKQJ7CsI3fXuGwFLEXDLjh8mpSaYyL6Nw/2mfWjuYLHWGWL10BZDZWdmDdKhXhJ3eysWE6E1WLQgKoD43fUu3+EdStmLasnlDNF6LBae+iK7sCkvzD8JBkvbLuB5LjAw6f+ZGAD1pBeOLRKag3qG4wtOq2zawoEYH0OJjq6yOzh78RkqM2HY39Rq6tXA4hJhAJ43CwIVaa9SFL5Wa2sHjwSxKx1cbAJHGAKGbPTCQ0mnHOfbNLdQvIbJb0y5+0dGJebrRl19OX3iUQ7fKz6IWw2miJYMK8n+3BO+WgOxnhS/vXrn3m01yiVuNTLkMjbsMMLTIkWFfTcm77aafQGN++nYGBRncL+xbzyo4LOo1iJcUB9LN9y7OZx12Miv7qXPqYKXSEc4T+QF0lJTh7XMzU0kgdlDmxeer0i+6bTDSJIvHEMYM2YMjVWL/gCpby9WgxSMNFEErXlPIMazx8lXTqHPInYWH8WQMVAzDK2OPlXnlRwGQ3RHBD3OshJ/Il6ipoujfTSr+P9ropPgjBaZQ== X-Forefront-Antispam-Report: CIP:104.207.196.24;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:exchange.arrl.org;PTR:exchange365.arrl.org;CAT:NONE;SFS:(13230031)(376005)(82310400014)(36860700004)(61400799018);DIR:OUT;SFP:1102; X-OriginatorOrg: reflector.arrl.org X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Apr 2024 18:00:51.1328 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: af0fef25-f944-4eb6-4002-08dc5275aaf2 X-MS-Exchange-CrossTenant-Id: baa21d0f-7301-40fe-bdd6-03d443a8c218 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=baa21d0f-7301-40fe-bdd6-03d443a8c218;Ip=[104.207.196.24];Helo=[exchange.arrl.org] X-MS-Exchange-CrossTenant-AuthSource: DS2PEPF00003443.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR16MB5340
Thoughts
The ARRL really needs to step up their game on the IT security but more importantly us members need a good amount of updates on this. Be honest and forthright about what's happened. While it's bad, it's the fault of the blackhats who did this. If there's a ransom to be paid, the members need to be informed of this, even if covered by insurance.
The league must present a root cause report of this so we can learn from it. Anything less will be viewed as a coverup by the members.
