SIEM and alert triage

Why a SIEM

A company of 1,000 people can produce millions of log events a day from hundreds of systems. No person can read that. A SIEM collects the logs, converts them into a common format (normalising), stores them, lets you search them, and runs detection rules that raise alerts when a pattern looks like an attack.

• Collect: agents and connectors pull logs from Windows, Linux, firewalls, cloud and apps.
• Normalise: map different field names to common ones (a source IP is "src_ip" whatever the device called it).
• Correlate: link events across sources, like "many failed logons, then one success, then a new admin account".
• Alert: create a ticket for the SOC with the evidence attached.
• Search and dashboards: let analysts investigate and managers see trends.

Well-known SIEMs include Splunk, Microsoft Sentinel, IBM QRadar, Google Security Operations and Elastic Security. Wazuh is a popular open-source platform for home labs. Each has its own search language, but the ideas are the same.

index=wineventlog EventCode=4625
| stats count by src_ip, user
| where count > 10
| sort - count

SecurityEvent
| where EventID == 4625
| summarize Failures = count() by IpAddress, TargetUserName
| where Failures > 10
| order by Failures desc

A correlation rule: guessing then success

A single failed logon is normal: people mistype. Many failures followed by a success from the same source is much more worrying, because the guessing may have worked. Rules like this combine events to cut noise.

events = [
    ("198.51.100.23", "failure"), ("198.51.100.23", "failure"),
    ("198.51.100.23", "failure"), ("198.51.100.23", "success"),
    ("10.20.5.7", "failure"), ("10.20.5.7", "success"),
]
fails = {}
for ip, result in events:
    if result == "failure":
        fails[ip] = fails.get(ip, 0) + 1
    elif fails.get(ip, 0) >= 3:
        print("ALERT: success after", fails[ip], "failures from", ip)

Triage: the first 15 minutes

Triage means sorting by urgency, like a hospital emergency desk. For each alert, an L1 analyst works through steps like these:

1. Read the alert: what rule fired, on which host and user, when?
2. Gather context: who is this user, what is this machine, is it important, is the source IP internal, a known partner, or known bad (threat intelligence)?
3. Look around: other alerts or unusual events for this user or host in the last 24 hours.
4. Decide: true positive, false positive, or benign true positive (real activity that is allowed, like an approved penetration test).
5. Set severity and act: close with a note, or escalate to L2 with the evidence, following the playbook.

Response-time targets differ between companies; the playbook and service agreement set them.

Writing it down

Every triage ends with a short note: what you checked, what you found, your verdict and why. Another analyst should be able to read it and reach the same conclusion. "Closed, FP" is not a note; "Closed as false positive: source 10.20.5.33 is the vulnerability scanner (asset tag VS-01), scan approved in change ticket CHG-1182" is.

Common mistakes

• Closing alerts by habit because "it is always a false positive".
• Not checking the user's other activity: one alert is often part of a bigger story.
• Escalating with no evidence attached.
• Never feeding back: if a rule is noisy, tell the detection engineers so it gets tuned.