A Guide to Using Longitudinal Data Analysis for Improved Identity Threat Detection

With organizations managing a sprawling number of identities—both human and non-human—across on-prem, cloud, and SaaS environments, today’s attack surface has expanded significantly. Each of these identities presents an opportunity for a threat actor to exploit and gain access to sensitive system data and resources. As a result, identity security has become critical in securing today’s digital environment. One useful approach for detecting threats in the identity space is Longitudinal Data Analysis (LDA), the analysis of changes in user behavior across time. By analyzing temporal patterns of access and privilege changes, security teams can uncover risks that are not evident using strictly point-in-time analysis.

Depending on the field, Longitudinal Data Analysis may be called Panel Data Analysis, Event History Analysis, or Repeated Measures Analysis. For the purpose of this blog, we will refer to this approach as Longitudinal Data Analysis.

LDA represents a critical approach in identifying suspicious changes in the configuration and/or behavior of an account, IP address, session, or other entity over time. Rather than examining only the current state of an entity, such as a user’s level of privilege, LDA employs a variety of methods to combine past state with current state to identify anomalous activity or configurations. This form of analysis can identify unusual behavior that would otherwise go unidentified if only the current state was analyzed. Escalation of privilege, or privilege jumps, for example, may indicate account compromise, making detection essential in securing an organization’s IT infrastructure. LDA and its associated methods can be leveraged by security teams to monitor and identify unusual privilege changes, helping organizations mitigate the risks posed by account vulnerabilities.

This blog marks the first in a series that explores applications of LDA in identity security. In this initial blog, we will focus on how LDA can be leveraged to detect anomalous privilege changes, a common security risk in large organizations.

At its core, LDA is a technique for analyzing change over time. It is a variation on time series analysis that, rather than focusing on a single entity over time, is used for the analysis and modeling of many entities across time. A defining feature of LDA is the collection of repeated measurements per entity, allowing for the study of change over time. This approach is well-suited to the identity security setting, where many identities and their behaviors/configurations need to be observed over time to detect when suspicious changes in behavior occur. For instance, these repeated measurements could take the form of a daily record of privilege level, number of failed authentications, and number of IP addresses per account. LDA utilizes various approaches to data modeling and structuring to allow for the temporal analysis of changes.

Examples of these temporal changes include:

• Lag and Cumulative Lag Variables: These variables represent previous states or events. They are often used to identify when current behavior deviates significantly from past behavior.

• (Start / Stop) Time: This represents the structuring of data into time intervals during which an event or state is active, such as when an account is in a low-privileged or high-privileged state.

• Interval Data: This represents the structuring of data into (possibly equally spaced) time intervals where the state of each entity is recorded at various points in time.

• Generalized Estimating Equations / Mixed Models: Statistical methods are used to account for dependencies within repeated measures or clustered data, such as the modeling of the expected privilege of each account within an organization. These methods are particularly useful as they can assume specific correlation structures (e.g. scenarios where an account’s current state is correlated with previous states). Later blogs in this series will describe applications of these models in greater detail.

• Hidden Markov Models: An unsupervised statistical method for identifying an entity’s latent states at a given time—such as normal/anomalous—given its observed behavior.

Applying LDA requires a systematic approach to data collection, analysis, and anomaly detection:

1. Data Collection

Collecting data over time is the foundation of LDA. In the context of identity security, this includes logging account activity, privilege levels, access requests, and other interactions with the identity management system.

To enable LDA, it is critical that both the current state and previous states are recorded. For example, if the goal is to identify unusual privilege changes, it is insufficient to record only an account’s current state. At a minimum, we would require the user’s current privilege level and previous privilege level, although more effective analysis can be performed with a more complete collection of state changes.

2. Data Structuring

As we’ve established above, Longitudinal Data Analysis is time-based—that means the data needs to be properly structured to preserve timestamps, or temporal markers, to allow it to ensure events are comparable and identify trends, anomalies, or behavioral shifts can be made visible.

There are several approaches to structuring data in LDA:

• Repeated Measurements - Most commonly, data is structured such that each entity is observed at various points in time. Measurements at each time period are then recorded in separate rows. Representing each change of state in a separate row is the most comprehensive approach, but it can be computationally expensive.

• Interval or Snapshot-based - An alternative approach that utilizes interval or snapshot-based data, where observations are taken at specific time periods. This approach is less costly, although it may lack the necessary precision for certain analyses.

• Wide Format - The use of additional columns in the form of lag and cumulative lag variables is another method that allows combining information on the current state and previous state without necessitating multiple records per entity.

In LDA for identity security, we derive features that capture critical information about user behavior, access patterns, and privilege changes over time. Feature engineering transforms raw logs into meaningful signals that help detect anomalies, predict risk, and uncover hidden threats. This step is essential to enable models to learn temporal dependencies and behavioral baselines.

Some key types of features include:

• Inactivity Periods: The length of time since an account was last active.

• Account Context: Information regarding the intent of the account, such as whether it is a service account or an account intended for daily use by a person.

• Privilege Level Changes: Any increases or decreases in an account's permissions.

• Patterns of Access Requests: Frequency and type of access attempts over time, including contextual information regarding the location, timing, and method of access requests.

• Risk Flags: Indicators such as the number of failed login attempts or unusual times of access.

• Resources Accessed: Types and sensitivity levels of resources accessed by an account, highlighting any changes in access patterns that may signal risk.
  
  

4. Anomaly Detection

Through LDA, unusual patterns of account activity, changes in privilege, and dormant periods can be visualized and analyzed. By setting thresholds for unusual activity—such as unexpected privilege escalation (privilege jumps), erratic login behavior, or inactivity—security teams can effectively flag accounts or actions that warrant further investigation.

This detection process can leverage different analytical methods, including:

• Rules-Based Detection: A traditional approach that uses predefined logic or thresholds to flag anomalies. For example, a rule might trigger an alert if a user logs in outside of business hours or if their privilege level changes twice within an hour. While this method is simple and transparent, it may miss subtle or novel threats that fall outside the defined parameters.

• Supervised Anomaly Detection: This technique uses labeled historical data to train models that can distinguish between normal and abnormal behavior. It requires examples of both benign and malicious activity. Once trained, the model can identify similar patterns in new data. This approach is powerful when quality labels are available but may struggle with detecting previously unseen attack techniques.

• Unsupervised Anomaly Detection: Ideal for detecting unknown or emerging threats, this method analyzes data without prior labels. It identifies deviations from learned behavioral baselines or statistical norms. Techniques like clustering, density estimation, or autoencoders can uncover subtle anomalies that might not be apparent through rules or prior examples.

By combining these approaches within a longitudinal analysis framework, identity security teams can move beyond static thresholds and gain a more adaptive, context-aware view of risk that is capable of detecting both known and novel identity-based threats.

For any service account showing a significant increase in privilege, an alert is generated for the security team to investigate further. The frequency of alerts can be modified by tuning the thresholds used to identify service accounts and relevant privilege changes.

Additional Recommendations

LDA not only helps in identifying unusual privilege changes, but also empowers security teams to improve overall security practices:

• Automate Alerts for Anomalies - By setting up automated alerts, security teams can be notified in real-time when service accounts show privilege jumps or other suspicious activities.

• Regularly Review Privileged Accounts - Periodic reviews of privileged accounts can prevent security risks. Accounts that are inactive for long periods should either be deactivated or have their privileges reduced. The same concepts used here in identifying privilege change could also be used to identify dormant accounts that have suddenly become active.

• Establish Thresholds for Anomalies - Defining threshold levels for privilege jumps and dormancy periods, adjusting as needed based on organizational policy, enables fine-tuning and the reduction of false positives/negatives.