Streamlining Security Event Collection with Microsoft 365 Defender and Azure Functions

Toby G

12 Mar 2025 • 4 min read

In today's complex cybersecurity landscape, organisations need efficient ways to collect, process, and analyse security events from their Microsoft environments. Microsoft 365 Defender (MDE) offers powerful threat detection capabilities, but extracting and integrating this valuable security data with other systems can be challenging.

In this post, I'll explore a production-ready Azure Function solution that automates the collection and forwarding of Microsoft 365 Defender security events to third-party SIEM solutions via Azure Event Hub. This solution leverages the Microsoft Graph API to efficiently query Advanced Hunting data with enterprise-grade features like rate limiting, error handling, and event deduplication.

The Challenge of Security Event Collection

Security teams face several challenges when working with Microsoft 365 Defender data:

Manual Collection: Without automation, security analysts must manually export data or write custom scripts
API Complexity: Working with Microsoft Graph API requires careful handling of authentication, pagination, and rate limits
Data Volume: Security events generate massive amounts of data that need efficient processing
Integration Gaps: Connecting Microsoft 365 Defender to third-party SIEM solutions often requires custom development

Enter the M365 Defender Event Collector Function

This Azure Function offers a complete solution for automating the collection and forwarding of Microsoft 365 Defender security events. Let's explore its key features and how it works.

Core Features

Automated Event Collection: Collects security events on a configurable schedule
Predefined Queries: Includes ready-to-use queries for antivirus detections, security alerts, and alert evidence
Event Deduplication: Prevents duplicate events from being processed
Rate Limiting Protection: Handles API throttling with exponential backoff
Batch Processing: Efficiently processes events in batches for optimal performance
Event Hub Integration: Seamlessly forwards events to Azure Event Hub for downstream processing

How It Works

The solution follows a straightforward process flow:

Authentication: Securely authenticates with Microsoft Graph API using client credentials flow
Query Execution: Runs predefined Advanced Hunting queries against the M365 Defender data
Pagination Handling: Efficiently processes large result sets with pagination
Rate Limit Management: Implements sophisticated retry logic for API throttling
Event Processing: Deduplicates and formats events for consistency
Forwarding: Sends processed events to Azure Event Hub for integration with SIEM systems

Setting Up the Solution

Prerequisites

Before implementing this solution, you'll need:

An Azure Subscription
Azure Function App (PowerShell runtime)
Microsoft 365 E5 Security Licence (or equivalent with Advanced Hunting access)
Azure AD App Registration with ThreatHunting.Read.All permissions
Azure Event Hub

Configuration Steps

Create App Registration:
- Register a new application in Entra ID (Azure AD)
- Assign the required ThreatHunting.Read.All Microsoft Graph API permission
- Generate a client secret
Create Azure Event Hub:
- Set up a new Event Hub namespace and hub instance
- Configure appropriate throughput units based on expected volume
Deploy the Azure Function:
- Create a PowerShell-based Azure Function
- Configure the timer trigger schedule (default: every 5 minutes)
- Set up Event Hub output binding
Set Environment Variables:
- TenantId: Your Entra ID Directory ID
- ClientId: Your App Registration Client ID
- ClientSecret: Your App Registration Secret

Customising for Your Environment

The solution is designed to be easily customisable for your specific needs:

Modifying Queries

You can edit the $queries array in the script to add, remove, or modify the KQL queries. The default configuration includes:

$queries = @(
    @{ 
        Name = "AntivirusDetection"
        KQL = "DeviceEvents 
            | where ActionType == 'AntivirusDetection' 
            | where Timestamp > ago($lookbackMinutes) 
            | order by Timestamp asc 
            | take $batchSize"
    },
    # Additional queries...
)

Adjusting Collection Frequency

Modify the timer trigger schedule in function.json to change how often events are collected:

Every 5 minutes: 0 */5 * * * * (default)
Every 30 minutes: 0 */30 * * * *
Hourly: 0 0 * * * *
Every 6 hours: 0 0 */6 * * *

Tuning Batch Size

The $batchSize parameter controls how many records are retrieved in each API call. The default is 10,000, but you can adjust this based on your environment's needs (maximum: 15,000).

Advanced Features

Sophisticated Rate Limiting Handling

One of the most impressive aspects of this solution is its handling of API rate limits:

if ($_.Exception.Response.StatusCode -eq 429) {
    # Handle rate limiting with exponential backoff
    $retryAfter = 12 # Default delay if header not present
    if ($_.Exception.Response.Headers["Retry-After"]) {
        $retryAfter = [int]$_.Exception.Response.Headers["Retry-After"]
    }
    Write-Host "Rate limit encountered. Waiting $retryAfter seconds..."
    Start-Sleep -Seconds $retryAfter
    $retryCount++
}

This code intelligently:

Captures HTTP 429 responses
Extracts the recommended wait time from the "Retry-After" header
Implements exponential backoff with a default delay
Retries the request up to 3 times

Event Deduplication

The solution prevents duplicate event processing with an efficient tracking mechanism:

$eventKey = "$($query.Name)_$($_.Id)_$($_.TimeGenerated)"
                    
if (-not $processedEvents.ContainsKey($eventKey)) {
    $processedEvents[$eventKey] = $true
    $outputEvents += @{
        QueryName = $query.Name
        Data = $_
        Timestamp = (Get-Date).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ss.fffffffZ")
    }
}

Monitoring and Troubleshooting

Key Monitoring Points

The function includes detailed logging for monitoring:

Authentication status
Query execution results
Rate limiting encounters
Batch processing statistics
Event forwarding status

Common Issues and Solutions

Authentication Failures
- Verify environment variables are correct
- Check App Registration permissions
- Ensure client secret hasn't expired
Rate Limiting
- Reduce batch size
- Increase timer trigger interval
- Optimise queries for efficiency
Missing Data
- Validate query timeframes
- Confirm permissions
- Review error logs

Integration with SIEM Solutions

The function forwards events to Azure Event Hub in a standardised format:

{
  "QueryName": "AntivirusDetection",
  "Data": {
    // Raw event data from Microsoft 365 Defender
  },
  "Timestamp": "2024-03-21T10:30:00.0000000Z"
}

From Azure Event Hub, you can integrate with various SIEM solutions:

Microsoft Sentinel: Use the built-in Event Hub connector
Splunk: Implement the Splunk Azure Event Hub connector
Elastic Stack: Use Logstash with the Azure Event Hub input plugin
Custom Solutions: Develop custom consumers for the Event Hub

Conclusion

The Microsoft 365 Defender Event Collector Function provides a robust, scalable solution for automating security event collection and integration. By leveraging Azure Functions and Event Hub, it bridges the gap between Microsoft's security ecosystem and third-party SIEM solutions.

This solution not only saves security teams valuable time but also ensures consistent, reliable event collection with enterprise-grade features like rate limiting protection, error handling, and event deduplication.

Whether you're looking to enhance your security monitoring capabilities or streamline your security operations, this Azure Function offers a production-ready approach to working with Microsoft 365 Defender data.

Next Steps

To get started with this solution:

Review the complete code and documentation
Set up the required Azure resources
Deploy and configure the function
Customise queries for your specific security monitoring needs
Integrate with your preferred SIEM solution

Happy security monitoring!