BeyondTrust Discovers Breach of Okta Support Unit

Summary

On October 2^nd, 2023, the BeyondTrust security teams detected an identity-centric attack on an in-house Okta administrator account. We immediately detected and remediated the attack through our own Identity Security tools, resulting in no impact or exposure to BeyondTrust’s infrastructure or to our customers. The incident was the result of Okta’s support system being compromised which allowed an attacker to access sensitive files uploaded by their customers.

The incident began when BeyondTrust security teams detected an attacker trying to access an in-house Okta administrator account using a valid session cookie stolen from Okta’s support system. Custom policy controls blocked the attacker's initial activity, but limitations in Okta's security model allowed them to perform a few confined actions. BeyondTrust’s own Identity Security Insights tool alerted the team of the attack, and they were able to block all access and verify that that attacker did not gain access to any systems.

The initial incident response indicated a possible compromise at Okta of either someone on their support team or someone in position to access customer support-related data. We raised our concerns of a breach to Okta on October 2^nd. Having received no acknowledgement from Okta of a possible breach, we persisted with escalations within Okta until October 19^th when Okta security leadership notified us that they had indeed experienced a breach and we were one of their affected customers.

Okta have now issued this statement confirming the breach that we detected nearly three weeks ago. Again, while there was no exposure to BeyondTrust or our customers, we are sharing details of the attack to educate other Okta users and infosec professionals. For BeyondTrust customers who leverage our Identity Security Insights product, we have also outlined the various detections that would alert you to this type of attack and recommendations to better control your attack surface and limit the possibility and impact of Okta-focused attacks.

Timeline Overview

• October 2, 2023 – Detected and remediated identity centric attack on an in-house Okta administrator account and alerted Okta
• October 3, 2023 – Asked Okta support to escalate to Okta security team given initial forensics pointing to a compromise within Okta support organization
• October 11, 2023 and October 13, 2023 – Held Zoom sessions with Okta security team to explain why we believed they might be compromised
• October 19, 2023 – Okta security leadership confirmed they had an internal breach, and BeyondTrust was one of their affected customers.

Attack Details

On October 2^nd, 2023, an Okta support agent requested a BeyondTrust Okta administrator generate a HAR file to assist in resolving an ongoing support issue the administrator was working on. HAR files are HTTP archives that can be generated by a web browser to log interactions with a website, in this case used for debugging an issue with the site. The administrator complied with the request and generated a HAR file containing an API request and a session cookie which was uploaded to the Okta support portal.

The Okta administrator’s account was protected with FIDO2 authentication, and policies within BeyondTrust’s Okta only allowed access to the admin console from managed devices with Okta Verify installed.

Within 30 minutes of the administrator uploading the file to Okta’s support portal an attacker used the session cookie from this support ticket, attempting to perform actions in the BeyondTrust Okta environment. BeyondTrust’s custom policies around admin console access initially blocked them, but they pivoted to using admin API actions authenticated with the stolen session cookie. API actions cannot be protected by policies in the same way as actual admin console access. Using the API, they created a backdoor user account using a naming convention like existing service accounts.

Our own instance of BeyondTrust’s Identity Security Insights, and tailored detections from our security teams, alerted us to several aspects of the intrusion. We immediately disabled the backdoor user account and revoked the attacker’s access before the account could be used and preventing any further actions. We saw no evidence of other irregular activity across all other privileged Okta users in Identity Security Insights, no evidence of other suspicious Okta accounts being created, and no evidence of any unusual activity in the targeted user’s account before this incident.

Detailed Attack Timeline

Below is the detailed timeline of events:

October 2, 2023

• A BeyondTrust Okta administrator uploads a browser recording (HAR file) at the request of Okta support related to ongoing troubleshooting of a non-security related support issue.
• Within 30 minutes of the support file upload there was an attempt to access the BeyondTrust Okta admin console as the BeyondTrust Okta administrator using an IP address in Malaysia linked to anonymizing proxy/VPN services.
• Okta events are logged from this Malaysian IP however there were no prior authentication events or activity from this user in this location as we would normally expect.
• Attacker was authenticated but access to the Okta console was denied due to a non-default Okta security policy configuration enforced by BeyondTrust security teams:
  • Default deny access and only allow access if specific criteria is met.
  • Attacker denied console access due to policy requirement of requiring Okta Verify on a managed device
• Attacker attempts to generate a password health report using the underlying API of the Okta admin console.
• Attacker attempts to gain access to main Okta dashboard but receives a policy challenge
  • Note: It is important for Okta customers to enhance security policies through settings such as prompting admin users for MFA at every sign in. While this was within an existing session the attacker hijacked, Okta still views dashboard access as a new sign in, and prompts for MFA.
• Attacker uses Okta official API to create a fake service account named “svc_network_backup” in an attempt to make it look like existing service accounts.
  • Note: Session cookies can be used to authenticate to official Okta API and in many cases these lack the policy restrictions that apply to the interactive admin console.
• Attacker acted quickly but our detections and responses were immediate, disabling the account and mitigating any potential exposure.
• BeyondTrust initiated an incident response process, immediately isolating and forensically investigating all systems and accounts associated with the administrator.
• The investigation did not discover any indication of compromise however it did uncover the HAR file that had been generated for the support case. This was notable as these are only created in exceptional circumstances, in this instance for troubleshooting a support case.
• BeyondTrust contacted Okta support to inform them of our concerns while we continued to investigate.

October 3, 2023

• Further investigation ruled out the possibility of the compromise originated from a BeyondTrust system leading us to conclude the Okta support system was likely compromised.
• Requested Okta support to escalate to their information security team given our concern that Okta was likely compromised, and other Okta customers might be exposed. No known compromise or ongoing security incident was communicated by Okta.

October 11, 2023

• Okta Support Zoom meeting with a member of their information security team where we shared our findings and requested additional log data from Okta related to support case data access.
• Okta committed to providing the requested logs and working with us. No known compromise or ongoing security incident was communicated by Okta.

October 13, 2023

• Okta support logs were received but contained several discrepancies. We requested more detailed logs relating to the discrepancies and reiterated our concerns that there was a high likelihood of compromise within Okta support and that we were likely not the only customer impacted. No known compromise or ongoing security incident was communicated by Okta.

October 19, 2023

• Call with Okta Security Leadership who notified us that there was a breach at Okta and we were one of the customers exposed during that breach.

October 20, 2023

• Coinciding with Okta’s public announcement we are publishing this blog with detailed information including indicators of compromise to provide information to the security community and protect mutual customers.

BeyondTrust would like to thank Okta for working with us to protect mutual customers. We appreciate their transparency in reporting this breach, notifying affected customers, and highlighting further investigative steps.

Identity Security Insights Detections Specific to this Discovery

The following are detections and recommendations available within BeyondTrust’s Identity Security Insights solution would have triggered for Insights customers if they were targeted by the techniques used in this Okta attack.

• Okta session hijacking: Attackers steal Okta session cookies and use them to access Okta from infrastructure they control, allowing them to bypass most MFA and security controls related to authentication. This detection looks for suspicious sessions appearing without an authentication event that are consistent with session hijacking.

• Okta user performed administrative action using a proxy: This attacker, and other Okta-focused attackers like Scattered Spider often use proxies to login as privileged users and perform sensitive administrative actions, but legitimate users rarely do.

• Okta admin privileges were granted to a user: Attackers often attempt to escalate privilege, or grant privilege to backdoor accounts. This information-level detection highlights all Okta admin assignments. These assignments are typically rare and usually occur within an established process.

• Okta password health report generated: This report is generated rarely in most environments we monitor. This information-level detection highlights when that happens in case the activity is suspicious.

• Okta user with some level of admin access uses MFA vulnerable to SIM swapping: Our incident response process was significantly faster because the admin user used FIDO2 for MFA, allowing us to rule out attacker-in-the-middle phishing as the mechanism for the token theft. Posture recommendations for privileged users give identity security professionals incremental changes they can make to better protect these crucial accounts.

If you are currently using Insights, please review your findings for any applicable detections based on the details below and feel free to reach out to BeyondTrust Support for help reviewing your own environment’s potential exposure from Okta’s breach. If you are not currently an Identity Security Insights customer but would like to leverage our free trial to assess your environment, please contact us.

Indicators of Compromise

• Access to Okta admin functions through proxy (isproxy: true in Okta log events)
• Access to Okta from IPs 202[.]59.10.100 or 23[.].105.182.19
• Access to Okta, especially Okta admin functions, from VPS/hosting providers. (Especially: VPS Malaysia, LeaseWeb.)
• Access to Okta with this user agent for an outdated version of chrome for MacOs: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.3538.77 Safari/537.36
• Okta account created via REST API with name svc_network_backup, or another name mimicking existing, legitimate accounts.
• Activity against endpoints like /reports/password-health/async_csv_download_schedule?, which are typically used from Okta Admin Console UI only, without any corresponding admin console login
• Okta activity for a user without any clear indication that the user authenticated (e.g. a user.session.start event for that user from a similar geographic area)
• Admin console login attempts that are denied by policy without a subsequent successful login to admin console from the same user within an hour

Other Notes

• Okta have recently updated their KB articles relating to the creation and sanitization of HAR files, we recommend reviewing these.
• We did not see any failed attempts to use the stolen session after its expiration, but this may be because actions attempted with expired sessions do not appear in Okta logs.

Recommended Posture Improvements

• Add policy controls in Okta to restrict access to admin console.
• Consider adjusting Okta global session policy to issue an MFA challenge at every sign on, which will prevent attackers with a stolen cookie from accessing main dashboard.
• Limit length of Okta sessions and take other steps to reduce window during which a stolen cookie can be used.
• Be aware that admin API actions authenticated via session cookie are only covered by the Global Session Policy, which is often less restrictive than other policies.
• Be aware that session hijacking allows attackers to bypass MFA.
• Require strong hardware MFA for all Okta admins to prevent token hijacking via attacker-in-the-middle phishing.

Closing Thoughts

Modern identity-based attacks can be complex, and as this attack shows, can originate from environments outside your own. Good specific policies and internal controls are necessary to limit things like how HAR files are shared. Defense in depth is important though. The failure of a single control or process should not result in breach. Here, multiple layers of controls -- e.g. okta sign on controls, identity security monitoring, and so on, prevented a breach.

Further Identity Security Resources and References

Identity Attack & Defense: Lessons in Okta Security

We recently released a blog more generally around Okta security and recent threat actor activity.

https://www.beyondtrust.com/blog/entry/lessons-in-okta-security

Ep. 40 of The Adventures of Alice & Bob Podcast: the Breach of Okta's Support Unit

Hot off the mic, tune into Ep. 40 of The Adventures of Alice & Bob Podcast as James sits down with BeyondTrust CTO Marc Maiffret to discuss how BeyondTrust discovered a breach of Okta’s Support Unit, escalated concerns, and gathered the necessary evidence to spur Okta into action.

Listen here: Ep. 40 - Breached! BeyondTrust Discovers Breach of Okta Support Unit // Marc Maiffret

BeyondTrust's security research team is hiring. Learn more about open positions here.