Fundamentals

OSI Model

A working analyst's guide to the OSI model --- what happens at each layer, how traffic maps to layers during investigations, and why you need this mental model when reading packet captures.

View on Graph

What It Is and Why Analysts Need It

The OSI (Open Systems Interconnection) model is a conceptual framework published by the International Organization for Standardization in 1984. It standardizes network communication into seven distinct layers, each with a specific role. When a SOC analyst says “this looks like a L7 attack” or “that’s a L4 anomaly,” these terms come directly from the OSI stack.

Why the OSI model matters day-to-day:

  • Isolation during triage — When an alert fires, the first question is “at which layer is the malicious activity happening?” A DNS tunneling detection (L7 Application) has nothing to do with a MAC flooding attack (L2 Data Link), even though both are network-based.
  • Tool positioning — Every security tool operates at specific layers. A WAF analyses L7 HTTP traffic. An NGFW inspects L3-L4. Network taps see L1-L2. Knowing which tool to deploy starts with knowing which layer the adversary is operating on.
  • Packet analysis context — When you open a PCAP in Wireshark, the OSI model helps you navigate. Layer 2 (Ethernet header), Layer 3 (IP header), Layer 4 (TCP/UDP header), Layer 7 (application payload). Each is a distinct section of every packet.

The Seven Layers — What Happens, What Attacks, What Tools

Layer 1 — Physical

What it does: Transmits raw bit streams over physical medium — copper wire, fiber optic, radio waves. Defines voltage levels, cable types, connector shapes, and data rates.

Protocols and hardware: Ethernet cables (Cat5e, Cat6), fiber optics, hubs, repeaters, modems, wireless access points (physical radio layer).

Attacks at this layer:

  • Physical tampering (cable taps, rogue access points)
  • Electromagnetic interception (TEMPEST)
  • Signal jamming (radio DoS)

Security tools: Physical access controls, Faraday cages, fiber optic tap detection.

Layer 2 — Data Link

What it does: Frames data into packets, handles MAC addressing, manages access to the physical medium. This is where switches and bridges operate. Ethernet frames are created at L2.

Key concepts: MAC addresses, ARP (Address Resolution Protocol), VLANs, Spanning Tree Protocol (STP), PPPoE.

Protocols: Ethernet (IEEE 802.3), Wi-Fi (IEEE 802.11), ARP, VLAN trunking (802.1Q).

Attacks at this layer:

  • ARP spoofing — Attacker sends forged ARP replies to poison the switch’s MAC table, intercepting traffic between two hosts.
  • MAC flooding — Flooding a switch with fake MAC addresses until it fails open (into hub mode), allowing traffic capture.
  • VLAN hopping — Exploiting trunk port misconfigurations to hop between VLANs.

Detection: Monitor for ARP cache mismatches. Tools like arpwatch alert on MAC-to-IP binding changes. Switch port security limits MAC addresses per port.

Security tools: Port security (MAC limiting), Dynamic ARP Inspection (DAI), DHCP Snooping, 802.1X network access control.

Layer 3 — Network

What it does: Routes packets between different networks. This is where IP addressing lives. Every packet has a source and destination IP at L3. Routers make forwarding decisions here.

Key concepts: IP addresses (IPv4 and IPv6), subnetting, routing tables, TTL (Time To Live), ICMP.

Protocols: IP, ICMP, OSPF, BGP, RIP, IPsec (partially).

Attacks at this layer:

  • DDoS volumetric attacks — UDP floods, ICMP floods, amplification attacks (DNS amplification, NTP amplification) targeting bandwidth.
  • IP spoofing — Forging source IP addresses to evade detection or impersonate trusted hosts.
  • Routing hijacks — BGP route injection to reroute traffic (e.g., the 2018 Route 53 hijack).
  • Traceroute enumeration — T1590 (Gather Victim Network Information).

Detection: Monitor for large ICMP echo reply volumes, TTL anomalies, BGP prefix hijacks via RPKI validation. NetFlow/IPFIX analysis identifies traffic volume spikes.

Security tools: Routers with ACLs, NGFW (stateful packet inspection), DDoS mitigation appliances, BGP monitoring with RPKI.

Layer 4 — Transport

What it does: Manages end-to-end communication between applications. TCP provides reliable, ordered delivery with error checking. UDP provides fast, connectionless delivery. This is where ports and session management live.

Key concepts: TCP ports, UDP ports, three-way handshake (SYN → SYN-ACK → ACK), sequence numbers, TCP flags (SYN, ACK, FIN, RST, PSH, URG), window sizing, congestion control.

Protocols: TCP, UDP, SCTP, DCCP.

Attacks at this layer:

  • SYN flood — Sending many SYN packets without completing the handshake, exhausting server connection tables.
  • Port scanning — Half-open scans (SYN scan), FIN scans, Xmas scans to enumerate open ports without completing the full handshake.
  • TCP session hijacking — Predicting sequence numbers to inject data into an established session.

Detection: Monitor for incomplete TCP handshakes (SYN_RCVD connections growing). Analyze TCP flags: a SYN-FIN packet is invalid and indicates scanning. Sequence number anomalies suggest hijacking.

Security tools: Stateful firewalls, IDS/IPS (Snort, Suricata), SYN cookies, TCP stack hardening.

Layer 5 — Session

What it does: Establishes, maintains, and terminates sessions between applications. Manages dialog control (who talks when), checkpointing, and recovery.

Key concepts: Session establishment/teardown, checkpointing, dialog control (half-duplex vs. full-duplex), NetBIOS sessions, SMB sessions, TLS session management (partially at L5).

Protocols: NetBIOS, RPC, SOCKS, PPTP, SIP session management.

Attacks at this layer:

  • Session hijacking — Stealing or predicting session tokens after authentication (especially in web applications).
  • NetBIOS poisoning — LLMNR/NBT-NS poisoning to intercept authentication within a LAN (see T1557.001).
  • RPC manipulation — Exploiting MS-RPC for remote code execution (e.g., EternalBlue, T1210).

Detection: Monitor for unusual RPC activity, session token reuse, abnormal NetBIOS name service traffic.

Security tools: Session management hardening, token binding, SMB signing enforcement.

Layer 6 — Presentation

What it does: Translates data between application and network formats. Handles encryption, compression, encoding, and character set conversion. Ensures data sent by one application is readable by another.

Key concepts: Encryption (TLS handshake negotiation), data compression, character encoding (ASCII, UTF-8), MIME types, SSL/TLS certificate negotiation.

Protocols: SSL/TLS (encryption layer negotiation), Kerberos (authentication exchange encoding), JPEG, GIF, MPEG (data formatting).

Attacks at this layer:

  • SSL stripping — Downgrading HTTPS to HTTP by intercepting the encryption negotiation.
  • Padding oracle attacks — Exploiting encryption padding behavior to decrypt data (e.g., POODLE).
  • Certificate spoofing — Using self-signed or stolen certificates to impersonate services.

Detection: Monitor for TLS version downgrade (especially TLS 1.0 connections in a 1.2+ environment), cipher suite downgrade, Certificate Transparency log mismatches.

Security tools: HSTS enforcement, certificate pinning, TLS inspection (interception proxies), CT monitoring (crt.sh).

Layer 7 — Application

What it does: Provides network services directly to end-user applications. This is where the data the user actually cares about lives — web pages, emails, file transfers, API calls.

Key concepts: HTTP requests/responses, DNS queries, SMTP transactions, REST API calls, WebSocket connections.

Protocols: HTTP/HTTPS, DNS, SMTP, IMAP, POP3, FTP, SSH, Telnet, DHCP, SNMP, SMB, LDAP.

Attacks at this layer:

  • SQL injection (T1190) — Injecting SQL commands through web application input fields.
  • Cross-site scripting (XSS) (T1559) — Injecting malicious scripts into web pages viewed by other users.
  • Phishing (T1566) — Social engineering via email or web pages.
  • DNS tunneling — Encapsulating non-DNS data in DNS queries/responses for data exfiltration or C2.
  • Web application brute force — Credential stuffing against login forms.

Detection: WAF alerts on SQLi/XSS patterns, DNS query volume analysis (especially TXT records), HTTP response code anomalies, user agent string analysis.

Security tools: WAF (ModSecurity, AWS WAF), web proxy/filter, DNS sinkhole, DLP solutions, CDN-based DDoS protection.

Using the OSI Model in Triage — A Worked Example

A SOC analyst sees an alert: “Multiple failed authentication attempts from external IP 203.0.113.55 against mail server 10.0.1.50 on port 443.”

Layer mapping:

  • L7: The attack is against the webmail login page (HTTPS application). This rules out protocol-level attacks.
  • L6: TLS encrypts the traffic — the analyst cannot read the HTTP payload inline but can inspect certificate metadata and TLS handshake parameters.
  • L4: Destination port 443 (HTTPS) tells the analyst it’s hitting the web service, not SMTP or IMAP.
  • L3: Source IP 203.0.113.55 is external — this determines the correct mitigation (block at the edge firewall, not an internal ACL).

Next question: Is this a credential-stuffing botnet (many accounts, few attempts each) or a targeted brute force (one account, many attempts)? This determines whether to block the IP or reset the targeted user’s credentials.

Each layer answer narrows the triage space. That is the power of the OSI model.

Sources