SIEM Log Management

What SIEM Log Management Covers and Why It Matters

• A SIEM (Security Information and Event Management) system is only as useful as the logs it ingests — garbage in, garbage out applies more to SIEM than almost any other security tool.
• Log management encompasses the full pipeline: generation, collection, transport, parsing, normalization, correlation, storage, and retirement.
• The three most common transport mechanisms are syslog (for network devices, Linux hosts, and appliances), Windows Event Forwarding (WEF, for Windows endpoints), and API pulls (for cloud services like AWS CloudTrail, Azure AD, and SaaS platforms).
• MITRE ATT&CK maps log collection and analysis as a detection control spanning multiple techniques — consistent logging is the foundation of T1562.001 (Impair Defenses: Disable or Modify Tools) detection.

Syslog — The Universal Log Transport

Syslog Formats

Syslog comes in two major RFC standards. Most devices support one or the other:

Key difference: RFC 5424 adds structured data elements (SD-ELEMENTS in square brackets), which allows key-value pairs without relying on regex parsing. RFC 3164 is simpler but harder to parse reliably.

Syslog Facilities and Severities

Syslog messages include a facility code (what generated it) and a severity level:

Priority formula: PRI = (facility * 8) + severity. So facility 4 (auth) + severity 3 (error) = PRI 35.

Configuring rsyslog Forwarding

# On the log source — /etc/rsyslog.conf or /etc/rsyslog.d/forward.conf

# Forward all auth messages to SIEM
auth.* @siem.internal.example.com:514
auth.* @@siem.internal.example.com:6514  # TLS

# Forward specific facility/severity combos
*.info;mail.none;authpriv.none;cron.none @siem.internal.example.com:514

# Template for RFC 5424 format
$template ForwardFormat,"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% [example@1 logsource=\"%HOSTNAME%\" facility=\"%syslogfacility-text%\"] %msg%\n"
*.* @@siem.internal.example.com:6514;ForwardFormat

Windows Event Forwarding (WEF)

WEF uses Windows’ built-in WinRM to forward events from endpoints to a central Windows Event Logging & Audit Policy collector server.

WEF Subscription Types

Essential Event IDs to Forward

SPL — Verify WEF Health

index=windows sourcetype=WinEventLog:Security
| stats count, dc(Computer) as Endpoints by date_mday
| eval expected_endpoints = 500
| eval coverage_pct = (Endpoints / expected_endpoints) * 100
| table date_mday, count, Endpoints, coverage_pct
| where coverage_pct < 90

Log Rotation — Protect Against Disk Full and Data Loss

Log rotation prevents disks from filling up and ensures old logs are archived or purged. A failed log rotation can cause a gap in visibility during an incident.

Logrotate Configuration

# /etc/logrotate.d/syslog
/var/log/syslog
/var/log/auth.log
/var/log/kern.log
{
    rotate 7
    daily
    compress
    delaycompress
    missingok
    notifempty
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Parsing and Normalization — Making Logs Queryable

Raw logs are unstructured text. Parsing extracts fields (timestamp, source IP, dest IP, user, action), and normalization maps those fields to a common schema.

Parsing Challenges

Normalization Schema Example (ECS — Elastic Common Schema)

Why normalization matters:

• Correlation rules that work across sources: “detect failed logins” works the same whether the source is Windows Security (4625), Linux auth.log, or VPN logs — because they all map to the same event.code and user.name fields
• Dashboards work across data sources without per-vendor field references
• EDR-to-SIEM field mapping works when endpoints and network logs use the same namespace

Correlation Rules — Turning Logs into Alerts

Correlation rules are the core value of a SIEM — they aggregate and analyze logs to find patterns that indicate threats.

Rule Design Patterns

SPL — Correlation Rule for Impossible Travel

index=* sourcetype=WinEventLog:Security EventCode=4624
| search Account_Name!="SYSTEM" AND Account_Name!="ANONYMOUS LOGON"
| iplocation Client_IP
| streamstats time_window=2h last(Country) as prev_country, last(City) as prev_city by Account_Name
| where prev_country != "" AND Country != prev_country
| eval travel_time = _time - prev_time
| eval travel_speed_kmh = distance(prev_lat, prev_lon, Latitude, Longitude, "k") / (travel_time / 3600)
| where travel_speed_kmh > 900
| eval alert = "IMPOSSIBLE TRAVEL: " . Account_Name . " logged in from " . prev_city . "," . prev_country . " then " . City . "," . Country . " in " . round(travel_time/60, 1) . " min (" . round(travel_speed_kmh, 0) . " km/h)"
| table _time, Account_Name, Workstation_Name, Client_IP, Country, City, prev_country, prev_city, travel_speed_kmh, alert

Log Retention — Storage vs Compliance vs Investigation

Every log has a cost — storage, indexing, query performance. Retention policies balance investigation needs against operational costs.

SPL — Monitor SIEM log ingestion volume:

index=*
| stats sum(b) as total_bytes by source, index
| eval total_gb = round(total_bytes / 1073741824, 2)
| eval total_ingest_gb = round(sum(total_gb), 2)
| table source, index, total_gb
| addcoltotals

Common Log Management Mistakes

Related

• Indicators: IoC, IoA, and TTP — covers the indicators: ioc, ioa, and ttp concepts
• Digital Forensics & Live Response — detection and response for T1562.001 techniques
• Log Sources Overview — covers the log sources overview concepts

Sources

• RFC 3164 — BSD syslog
• RFC 5424 — Syslog Protocol
• Microsoft — Windows Event Forwarding
• Elastic Common Schema (ECS)
• Splunk — Event Timestamp Best Practices
• SANS — Log Management Best Practices