Migrate ArcSight detection rules to Microsoft Sentinel
This article describes how to identify, compare, and migrate your ArcSight detection rules to Microsoft Sentinel analytics rules.
Identify and migrate rules
Microsoft Sentinel uses machine learning analytics to create high-fidelity and actionable incidents, and some of your existing detections may be redundant in Microsoft Sentinel. Therefore, don't migrate all of your detection and analytics rules blindly. Review these considerations as you identify your existing detection rules.
- Make sure to select use cases that justify rule migration, considering business priority and efficiency.
- Check that you understand Microsoft Sentinel rule types.
- Check that you understand the rule terminology.
- Review any rules that haven't triggered any alerts in the past 6-12 months, and determine whether they're still relevant.
- Eliminate low-level threats or alerts that you routinely ignore.
- Use existing functionality, and check whether Microsoft Sentinel’s built-in analytics rules might address your current use cases. Because Microsoft Sentinel uses machine learning analytics to produce high-fidelity and actionable incidents, it’s likely that some of your existing detections won’t be required anymore.
- Confirm connected data sources and review your data connection methods. Revisit data collection conversations to ensure data depth and breadth across the use cases you plan to detect.
- Explore community resources such as the SOC Prime Threat Detection Marketplace to check whether your rules are available.
- Consider whether an online query converter such as Uncoder.io might work for your rules.
- If rules aren’t available or can’t be converted, they need to be created manually, using a KQL query. Review the rules mapping to create new queries.
Learn more about best practices for migrating detection rules.
To migrate your analytics rules to Microsoft Sentinel:
Verify that you have a testing system in place for each rule you want to migrate.
Prepare a validation process for your migrated rules, including full test scenarios and scripts.
Ensure that your team has useful resources to test your migrated rules.
Confirm that you have any required data sources connected, and review your data connection methods.
Verify whether your detections are available as built-in templates in Microsoft Sentinel:
If the built-in rules are sufficient, use built-in rule templates to create rules for your own workspace.
In Microsoft Sentinel, go to the Configuration > Analytics > Rule templates tab, and create and update each relevant analytics rule.
For more information, see Detect threats out-of-the-box.
If you have detections that aren't covered by Microsoft Sentinel's built-in rules, try an online query converter, such as Uncoder.io to convert your queries to KQL.
Identify the trigger condition and rule action, and then construct and review your KQL query.
If neither the built-in rules nor an online rule converter is sufficient, you'll need to create the rule manually. In such cases, use the following steps to start creating your rule:
Identify the data sources you want to use in your rule. You'll want to create a mapping table between data sources and data tables in Microsoft Sentinel to identify the tables you want to query.
Identify any attributes, fields, or entities in your data that you want to use in your rules.
Identify your rule criteria and logic. At this stage, you may want to use rule templates as samples for how to construct your KQL queries.
Consider filters, correlation rules, active lists, reference sets, watchlists, detection anomalies, aggregations, and so on. You might use references provided by your legacy SIEM to understand how to best map your query syntax.
Identify the trigger condition and rule action, and then construct and review your KQL query. When reviewing your query, consider KQL optimization guidance resources.
Test the rule with each of your relevant use cases. If it doesn't provide expected results, you may want to review the KQL and test it again.
When you're satisfied, you can consider the rule migrated. Create a playbook for your rule action as needed. For more information, see Automate threat response with playbooks in Microsoft Sentinel.
Learn more about analytics rules:
- Create custom analytics rules to detect threats. Use alert grouping to reduce alert fatigue by grouping alerts that occur within a given timeframe.
- Map data fields to entities in Microsoft Sentinel to enable SOC engineers to define entities as part of the evidence to track during an investigation. Entity mapping also makes it possible for SOC analysts to take advantage of an intuitive investigation graph that can help reduce time and effort.
- Investigate incidents with UEBA data, as an example of how to use evidence to surface events, alerts, and any bookmarks associated with a particular incident in the incident preview pane.
- Kusto Query Language (KQL), which you can use to send read-only requests to your Log Analytics database to process data and return results. KQL is also used across other Microsoft services, such as Microsoft Defender for Endpoint and Application Insights.
Compare rule terminology
This table helps you to clarify the concept of a rule in Microsoft Sentinel compared to ArcSight.
ArcSight | Microsoft Sentinel | |
---|---|---|
Rule type | • Filter rule • Join rule • Active list rule • And more |
• Scheduled query • Fusion • Microsoft Security • Machine Learning (ML) Behavior Analytics |
Criteria | Define in rule conditions | Define in KQL |
Trigger condition | • Define in action • Define in aggregation (for event aggregation) |
Threshold: Number of query results |
Action | • Set event field • Send notification • Create new case • Add to active list • And more |
• Create alert or incident • Integrates with Logic Apps |
Map and compare rule samples
Use these samples to compare and map rules from ArcSight to Microsoft Sentinel in various scenarios.
Rule | Description | Sample detection rule (ArcSight) | Sample KQL query | Resources |
---|---|---|---|---|
Filter (AND ) |
A sample rule with AND conditions. The event must match all conditions. |
Filter (AND) example | Filter (AND) example | String filter: • String operators Numerical filter: • Numerical operators Datetime filter: • ago • Datetime • between • now Parsing: • parse • extract • parse_json • parse_csv • parse_path • parse_url |
Filter (OR ) |
A sample rule with OR conditions. The event can match any of the conditions. |
Filter (OR) example | Filter (OR) example | • String operators • in |
Nested filter | A sample rule with nested filtering conditions. The rule includes the MatchesFilter statement, which also includes filtering conditions. |
Nested filter example | Nested filter example | • Sample KQL function • Sample parameter function • join • where |
Active list (lookup) | A sample lookup rule that uses the InActiveList statement. |
Active list (lookup) example | Active list (lookup) example | • A watchlist is the equivalent of the active list feature. Learn more about watchlists. • Other ways to implement lookups |
Correlation (matching) | A sample rule that defines a condition against a set of base events, using the Matching Event statement. |
Correlation (matching) example | Correlation (matching) example | join operator: • join • join with time window • shuffle • Broadcast • Union define statement: • let Aggregation: • make_set • make_list • make_bag • pack |
Correlation (time window) | A sample rule that defines a condition against a set of base events, using the Matching Event statement, and uses the Wait time filter condition. |
Correlation (time window) example | Correlation (time window) example | • join • Microsoft Sentinel rules and join statement |
Filter (AND) example: ArcSight
Here's a sample filter rule with AND
conditions in ArcSight.
Filter (AND) example: KQL
Here's the filter rule with AND
conditions in KQL.
SecurityEvent
| where EventID == 4728
| where SubjectUserName =~ "AutoMatedService"
| where isnotempty(SubjectDomainName)
This rule assumes that the Azure Monitoring Agent (AMA) collects the Windows Security Events. Therefore, the rule uses the Microsoft Sentinel SecurityEvent table.
Consider these best practices:
- To optimize your queries, avoid case-insensitive operators when possible:
=~
. - Use
==
if the value isn't case-sensitive. - Order the filters by starting with the
where
statement, which filters out the most data.
Filter (OR) example: ArcSight
Here's a sample filter rule with OR
conditions in ArcSight.
Filter (OR) example: KQL
Here are a few ways to write the filter rule with OR
conditions in KQL.
As a first option, use the in
statement:
SecurityEvent
| where SubjectUserName in
("Adm1","ServiceAccount1","AutomationServices")
As a second option, use the or
statement:
SecurityEvent
| where SubjectUserName == "Adm1" or
SubjectUserName == "ServiceAccount1" or
SubjectUserName == "AutomationServices"
While both options are identical in performance, we recommend the first option, which is easier to read.
Nested filter example: ArcSight
Here's a sample nested filter rule in ArcSight.
Here's a rule for the /All Filters/Soc Filters/Exclude Valid Users
filter.
Nested filter example: KQL
Here are a few ways to write the filter rule with OR
conditions in KQL.
As a first option, use a direct filter with a where
statement:
SecurityEvent
| where EventID == 4728
| where isnotempty(SubjectDomainName) or
isnotempty(TargetDomainName)
| where SubjectUserName !~ "AutoMatedService"
As a second option, use a KQL function:
Save the following query as a KQL function with the
ExcludeValidUsers
alias.SecurityEvent | where EventID == 4728 | where isnotempty(SubjectDomainName) | where SubjectUserName =~ "AutoMatedService" | project SubjectUserName
Use the following query to filter the
ExcludeValidUsers
alias.SecurityEvent | where EventID == 4728 | where isnotempty(SubjectDomainName) or isnotempty(TargetDomainName) | where SubjectUserName !in (ExcludeValidUsers)
As a third option, use a parameter function:
Create a parameter function with
ExcludeValidUsers
as the name and alias.Define the parameters of the function. For example:
Tbl: (TimeGenerated:datetime, Computer:string, EventID:string, SubjectDomainName:string, TargetDomainName:string, SubjectUserName:string)
The
parameter
function has the following query:Tbl | where SubjectUserName !~ "AutoMatedService"
Run the following query to invoke the parameter function:
let Events = ( SecurityEvent | where EventID == 4728 ); ExcludeValidUsers(Events)
As a fourth option, use the join
function:
let events = (
SecurityEvent
| where EventID == 4728
| where isnotempty(SubjectDomainName)
or isnotempty(TargetDomainName)
);
let ExcludeValidUsers = (
SecurityEvent
| where EventID == 4728
| where isnotempty(SubjectDomainName)
| where SubjectUserName =~ "AutoMatedService"
);
events
| join kind=leftanti ExcludeValidUsers on
$left.SubjectUserName == $right.SubjectUserName
Considerations:
- We recommend that you use a direct filter with a
where
statement (first option) due to its simplicity. For optimized performance, avoid usingjoin
(fourth option). - To optimize your queries, avoid the
=~
and!~
case-insensitive operators when possible. Use the==
and!=
operators if the value isn't case-sensitive.
Active list (lookup) example: ArcSight
Here's an active list (lookup) rule in ArcSight.
Active list (lookup) example: KQL
This rule assumes that the Cyber-Ark Exception Accounts watchlist exists in Microsoft Sentinel with an Account field.
let Activelist=(
_GetWatchlist('Cyber-Ark Exception Accounts')
| project Account );
CommonSecurityLog
| where DestinationUserName in (Activelist)
| where DeviceVendor == "Cyber-Ark"
| where DeviceAction == "Get File Request"
| where DeviceCustomNumber1 != ""
| project DeviceAction, DestinationUserName,
TimeGenerated,SourceHostName,
SourceUserName, DeviceEventClassID
Order the filters by starting with the where
statement that filters out the most data.
Correlation (matching) example: ArcSight
Here's a sample ArcSight rule that defines a condition against a set of base events, using the Matching Event
statement.
Correlation (matching) example: KQL
let event1 =(
SecurityEvent
| where EventID == 4728
);
let event2 =(
SecurityEvent
| where EventID == 4729
);
event1
| join kind=inner event2
on $left.TargetUserName==$right.TargetUserName
Best practices:
- To optimize your query, ensure that the smaller table is on the left side of the
join
function. - If the left side of the table is relatively small (up to 100 K records), add
hint.strategy=broadcast
for better performance.
Correlation (time window) example: ArcSight
Here's a sample ArcSight rule that defines a condition against a set of base events, using the Matching Event
statement, and uses the Wait time
filter condition.
Correlation (time window) example: KQL
let waittime = 10m;
let lookback = 1d;
let event1 = (
SecurityEvent
| where TimeGenerated > ago(waittime+lookback)
| where EventID == 4728
| project event1_time = TimeGenerated,
event1_ID = EventID, event1_Activity= Activity,
event1_Host = Computer, TargetUserName,
event1_UPN=UserPrincipalName,
AccountUsedToAdd = SubjectUserName
);
let event2 = (
SecurityEvent
| where TimeGenerated > ago(waittime)
| where EventID == 4729
| project event2_time = TimeGenerated,
event2_ID = EventID, event2_Activity= Activity,
event2_Host= Computer, TargetUserName,
event2_UPN=UserPrincipalName,
AccountUsedToRemove = SubjectUserName
);
event1
| join kind=inner event2 on TargetUserName
| where event2_time - event1_time < lookback
| where tolong(event2_time - event1_time ) >=0
| project delta_time = event2_time - event1_time,
event1_time, event2_time,
event1_ID,event2_ID,event1_Activity,
event2_Activity, TargetUserName, AccountUsedToAdd,
AccountUsedToRemove,event1_Host,event2_Host,
event1_UPN,event2_UPN
Aggregation example: ArcSight
Here's a sample ArcSight rule with aggregation settings: three matches within 10 minutes.
Aggregation example: KQL
SecurityEvent
| summarize Count = count() by SubjectUserName,
SubjectDomainName
| where Count >3
Next steps
In this article, you learned how to map your migration rules from ArcSight to Microsoft Sentinel.