For many years, security practitioners have used the proverbial expression “searching for the needle in a haystack” to describe their struggles in identifying threats within an organization’s data. However, the advancements in cloud technologies, such as modern cloud data lakes like Snowflake, have transformed the way SecOps teams operate, enabling them to become more data-driven and eliminate data silos. With this transformation comes the ability to perform faster searches for Indicators of Compromise (IOCs) and speed up threat investigations by 10x.
Leveraging the Power of Snowflake
With massive amounts of telemetry data generated by cloud applications, having disparate data silos across an organization can create bottlenecks for security analytics to perform threat investigation and searching for IOCs. However, by leveraging new platforms, new products have transformed SecOps teams to be more data-driven. By applying an open data model and stitching all the data with normalized fields, SecOps teams can enable all downstream analytics with zero data engineering, zero operations, and full data democratization.
Using the power of Snowflake, new solutions aggregate relevant search indexes into a new index that stiches all relevant data sources into a new search index using an open data model, like OCSF. This enables security teams to perform faster lookup of IOCs (i.e., file hashes, users, entities) for threat investigators within a security data lake over 12+ month periods.
Unpacking the Benefits of an Open Data Model
To unpack the benefits of an open data model, we need to look at security analytics, which rely on highly correlated data to yield new insights into the data. With a unified schema that normalizes field names (e.g., SRC_IP) and field values (Success/Failure) across all the disparate sources, the data model empowers analysts to find answers faster without any of the data wrangling in legacy systems. Furthermore, the data model needs to be OCSF compliant, enabling customers to rapidly onboard new data sources that comply with this new format.
Achieving Performance Search Index
Elysium Analytics has built an aggregation pipeline inside Snowflake to roll up the data from relevant sources, enabling the power of Snowflake SOS to query over 12 plus months of data within seconds. Merely turning on Snowflake SOS for source IP will result in poor performance, hitting all views and tables across the entire corpus of the dataset. Instead, building an aggregated table with deduping heuristics is much more high-performance. The aggregation pipeline process indexes to deduplicate repetitive messages and recursive alerts, enriches the records with missing details (user account, internal IP, last owner, etc.), and provides the following benefits:
- Semantic data model to unify the data in terms of field names and values across the sources
- Aggregate index for long-range queries over 12+ month periods
- Deduping the data to remove noise with repetitive and recursive messages
- Fast performance with Snowflake SOS
- Enrichment of the data with missing details
- Rapid Adoption of New Data Platform Technologies
Most security products today require the security analysts to rehydrate data from cold storage to query historical data. This creates large delays in response times during threat investigations. Elysium Analytics has solved this issue by rapidly adopting new data platform technologies to speed up long forensic searches from days into seconds, enabling our customers to respond faster to emerging threats.
In conclusion, the advancements in cloud technologies have transformed the way SecOps teams operate, enabling them to perform faster searches for IOCs and speed up threat investigations by 10x. By leveraging Snowflake and adopting an open data model, Elysium Analytics has developed solutions that enable security teams to aggregate relevant search. Please contact Elysium Analytics to schedule a demo!
