TCP/IP

What Is TCP/IP and Why Packets Matter to Analysts

TCP/IP is not a single protocol — it is a suite of protocols organized into four layers (Link, Internet, Transport, Application) that powers all modern network communication.

For SOC analysts, the critical layer is Transport: TCP (Transmission Control Protocol) and UDP (User Datagram Protocol). TCP provides reliable, ordered, error-checked delivery — every byte is tracked with sequence numbers and every segment is acknowledged. UDP just sends packets with no delivery guarantee.

The reason every analyst must understand TCP at the packet level: malicious traffic hides in protocol behavior, not just payloads. A port scan looks like a specific flag pattern. A C2 beacon looks like a consistent time-between-packets metric. Data exfiltration over DNS looks like abnormally large TXT responses. These patterns are invisible to signature-based detection — but obvious to an analyst who reads packets.

The TCP Header — What Each Field Tells You

Every TCP segment has a header containing these key fields:

TCP Flags — The Core Detection Vocabulary

There are six TCP flags. The flag combination in every packet tells you exactly what phase of communication is occurring.

Abnormal Flag Combinations (Indicators of Malicious Activity)

The Three-Way Handshake

The TCP three-way handshake establishes every connection. Understanding it lets you read network flows like a book.

Normal Handshake Flow

Client                          Server
  |                                |
  |  SYN (SEQ=1000)             |
  |  -------------------------->  |
  |                                |
  |  SYN-ACK (SEQ=2000, ACK=1001) |
  |  <--------------------------  |
  |                                |
  |  ACK (SEQ=1001, ACK=2001)  |
  |  -------------------------->  |
  |                                |
  |  [Data Transfer Begins]       |

Step 1 — SYN: Client sends a TCP segment with the SYN flag set, an initial sequence number (ISN = 1000 in this example), and no payload.

Step 2 — SYN-ACK: Server responds with SYN and ACK flags set. Its own ISN (2000) and acknowledges the client’s sequence number by setting ACK = client_ISN + 1 (1001).

Step 3 — ACK: Client sends an ACK segment acknowledging the server’s sequence number (ACK = 2001). The connection is now ESTABLISHED.

What the analyst sees in a packet capture: Three packets in rapid succession. The first has only SYN. The second has SYN+ACK. The third has only ACK. If you don’t see all three, the connection never completed.

Abnormal Handshake Patterns

Half-open scan (Stealth scan):

SYN → 
SYN-ACK ←  (port is open)
RST →       (client sends reset — never completes the handshake)

This is an Nmap SYN scan (-sS). The scanner sends SYN, and if it receives SYN-ACK (port is open), it immediately sends RST. The server sees a connection attempt that never completed — many services log this, and standard applications don’t.

SYN flood (DoS):

SYN →      (from spoofed IP 1)
SYN →      (from spoofed IP 2)
SYN →      (from spoofed IP 3)
...

Many SYN packets arriving from different source IPs, no SYN-ACK responses from the client. The server’s SYN_RCVD connection table fills up, denying service to legitimate connections.

Sequence Numbers in Practice

Sequence numbers track every byte transmitted. This is critical for two detection use cases:

1. Session Hijacking Detection

If an attacker injects packets into an existing TCP connection, the sequence numbers must match. This is how old-school TCP session hijacking worked — and why modern systems use random ISNs. If you see packets with sequence numbers that don’t fall within the expected window, an injection attack may be in progress. This is how old-school TCP session hijacking worked — and why modern systems use random ISNs.

2. Data Exfiltration Volume Tracking

By tracking the difference between sequence numbers in a flow, an analyst can calculate exactly how many bytes were transferred:

Bytes transferred = Last_SEQ + Payload_Length - First_SEQ

Example: A DNS tunnel with 1400-byte response segments is clearly not normal DNS traffic (standard DNS responses are 50-500 bytes).

How Analysts Use TCP/IP Knowledge for Detection

Beaconing Detection (C2)

C2 beacons follow predictable TCP patterns:

• Consistent interval: The time between connections to the C2 server is regular (± small jitter). e.g., every 60 seconds ± 5 seconds.
• Consistent packet size: Each beacon payload is roughly the same size (e.g., 512 bytes POST body every time).
• Connection duration: Often short-lived — connect, send beacon, receive command, disconnect.

Detection query (Zeek conn.log):

# Consistent intervals and small payloads = beaconing
event zeek_init() {
    local c = 0;
    for (conn in get_conn_log()) {
        if (conn$duration < 1 sec && conn$orig_bytes < 1000)
            print conn$id$resp_h;
    }
}

Port Scan Detection

• Half-open scans: SYN with no subsequent ACK. Multiple such connections to different ports from the same source = scanning.
• FIN scans: Packets with FIN flag (but no SYN) arriving in sequence to different ports.
• Xmas/NULL scans: Unusual flag combinations to test server behavior.

Data Exfiltration Over Unusual Ports

• SSH (port 22) on a high-traffic file server transferring data externally.
• DNS (port 53) tunneling: DNS responses larger than 512 bytes, high TXT query ratio, subdomains longer than typical.
• ICMP tunneling: Data embedded in ICMP echo request payloads (ping tunnels).

TCP vs. UDP — When Each Matters for Detection

Related

• OSI Model — covers the osi model concepts
• DNS — detection and response for T1572, T1568 techniques
• Common Ports and Protocols — covers the common ports and protocols concepts
• Network Security Basics — detection and response for T1040, T1046 techniques

Sources

• RFC 793 — TCP Specification
• Wireshark TCP Analysis
• JA3 TLS Fingerprinting
• SANS — TCP Packet Analysis