TCP/IP for Security Engineers

TL;DR: IP will carry a packet with any source address the sender writes, so the address is a claim, not proof. That makes spoofing possible wherever networks skip ingress filtering (BCP 38), and spoofed UDP turns servers into reflectors and amplifiers. TCP's handshake and 32-bit sequence numbers stop an attacker who cannot see the traffic from injecting into a connection, but only by making guessing expensive, and side channels have repeatedly made it cheaper. Neither layer protects against anyone on the path: they can read, change and reset at will. So availability defences (SYN cookies, filtering, rate limits) live here, and confidentiality and integrity live above, in TLS or an equivalent.

What each layer promises, and what it does not

Most protocol attacks start from a gap in what the layers below promise. Four facts carry most of the weight.

• IP delivers packets on a best-effort basis to a destination address. It does not check that the source address belongs to the sender. Routers forward on the destination.
• UDP adds ports and a checksum, nothing else. There is no handshake, so a server answers whatever address the request claims to come from.
• TCP adds a handshake, sequence numbers, ordering and retransmission. The handshake proves the other side can receive packets at the claimed address, which is why spoofing a full TCP conversation is much harder than spoofing UDP.
• ICMP reports errors and diagnostics. Those messages change how endpoints behave, which makes them both useful and a source of side channels.

None of them provides confidentiality or integrity against an attacker who can see the traffic. The checksums detect accidental corruption, not deliberate change.

Spoofing and ingress filtering

Because the source address is just a field, an attacker can send packets claiming to come from anywhere. They will not see the replies, which go to the real owner of that address, but for many attacks they do not need to.

The defence is at the edge of the network the packets come from. BCP 38 (RFC 2827, May 2000) asks networks to drop packets leaving their customers whose source addresses those customers could not legitimately use, and RFC 3704 extends the idea to networks with several upstream providers. Where it is deployed, spoofing from that network is impossible. Where it is not, the whole internet inherits the problem, which is why spoofed traffic still underpins large denial-of-service attacks. Inside your own network, the same idea appears as anti-spoofing checks on routers and switches: a port or subnet may only send from its own addresses.

The handshake and SYN floods

A TCP connection starts with three packets: the client sends SYN with its initial sequence number, the server replies SYN-ACK with its own, and the client sends ACK. Between the second and third packets the server holds a half-open connection.

A SYN flood sends many SYNs, often from spoofed addresses, so the replies go nowhere and the server's queue of half-open connections fills; legitimate clients are then turned away. RFC 4987 (2007) surveys the defences. The best known, SYN cookies, removes the state: the server encodes what it needs (a time counter, the negotiated maximum segment size and a keyed hash over the connection's addresses and ports) into its own initial sequence number, keeps nothing, and reconstructs the connection only when a valid ACK arrives that echoes the cookie. A spoofed SYN then costs the server a computation and a reply packet, not memory.

SYN cookies handle exhaustion of connection state. They do not help when the flood saturates the link itself; that needs upstream scrubbing or a provider that can absorb the volume, and, at the source, BCP 38.

Sequence numbers and blind injection

Every byte in a TCP connection is numbered in a 32-bit space, about 4.3 billion values, and each side starts at a randomised initial sequence number. A packet is accepted only if its sequence number falls in the receiver's current window. RFC 9293 (August 2022), which consolidated TCP's specification and replaced RFC 793, includes the requirement to generate initial sequence numbers unpredictably.

An off-path attacker, who cannot see the traffic, must guess. An on-path attacker reads the numbers directly and can inject, alter or reset at will; sequence numbers do nothing against them.

Worked example: the arithmetic of a blind reset

Suppose an attacker wants to tear down a long-lived connection, such as a BGP session between two routers, by injecting a reset (RST) packet. They know both addresses and the server port.

RFC 5961 (2010) changed the rules for that reason: an in-window but inexact RST or SYN is answered with a challenge ACK, and only a real peer can respond correctly. Then CVE-2016-5696 showed how arithmetic can be undone. Linux limited challenge ACKs with a single global counter shared by all connections. By watching how many challenge ACKs its own connection received, an off-path attacker could tell whether a spoofed probe to someone else's connection had triggered one, and so learn the port and sequence numbers far faster than guessing. Linux 4.7 fixed it with per-connection limits and randomness. The lesson for interviews: shared rate limits can turn into side channels.

For sessions where a reset is costly, such as routing, the stronger answer is authentication at the TCP layer: the TCP Authentication Option (RFC 5925) adds a keyed MAC to each segment, and the Generalized TTL Security Mechanism (RFC 5082) drops packets that could not have come from a directly connected neighbour.

Reflection and amplification

UDP's lack of a handshake makes it the natural tool for reflection: send a request with the victim's address as the source, and the server sends its reply to the victim. If the reply is much larger than the request, the attack is also amplified. The amplification factor is simply response bytes divided by request bytes. With an illustrative factor of 50, an attacker who can send 1 Gbit/s of spoofed requests directs 50 Gbit/s of replies at the victim, from servers the victim cannot block one by one.

Protocol designers now build the limit in. QUIC, the transport under HTTP/3, does not let a server send more than three times the bytes it has received from an address until that address is validated, and requires a client's first packets to be padded to at least 1,200 bytes, so a small spoofed packet cannot draw a large reply. Operators of UDP services apply the same idea: rate-limit responses per source, avoid answering large queries over UDP to unvalidated sources, and close services that do not need to be public.

Fragmentation and off-path DNS poisoning

Large IP packets may be split into fragments and reassembled at the destination. An off-path attacker who can predict how a reply will fragment can try to substitute a forged later fragment, which carries no new port or DNS transaction ID to guess. RFC 9715 (January 2025) therefore advises DNS over UDP to avoid IP fragmentation altogether, by keeping responses small and falling back to TCP.

SAD DNS (CVE-2020-25705) is the other classic. Off-path DNS cache poisoning normally needs the attacker to guess both the 16-bit transaction ID and the resolver's random source port. The researchers used the global rate limit on ICMP error messages as a side channel to discover which port the resolver was using, removing most of the randomness. It is the same pattern as the challenge-ACK bug: a shared limit leaking per-connection state. See DNS security for DNSSEC and encrypted DNS.

What TCP/IP does not give you

• No confidentiality or integrity against the path. Anyone on the path, a Wi-Fi access point, an ISP, a compromised router, can read and modify traffic and reset connections. See attacker positions on the network.
• No identity. An address says where to deliver, not who is sending. Address-based allowlists are a coarse filter, not authentication.
• NAT is not a firewall. It rewrites addresses; protection comes from the stateful filtering that usually accompanies it, and outbound connections remain open.

The practical conclusion is to authenticate and encrypt every channel that matters, with TLS or mutual TLS, IPsec or WireGuard, and to treat network-layer controls as protection for availability and as defence in depth, not as proof of who is talking.

Misconceptions interviewers listen for

• "TCP sequence numbers are a security feature." They make blind guessing expensive; they do nothing against on-path attackers.
• "UDP services cannot be used against third parties." Spoofed requests turn them into reflectors and amplifiers.
• "NAT hides and protects us." It changes addresses; filtering and authentication do the protecting.
• "An internal IP allowlist authenticates the caller." Addresses can be spoofed or shared; authenticate the channel.

How this shows up in interviews

Expect "what happens in a TCP handshake?", "how does a SYN flood work and how do SYN cookies help?", "why is UDP used for amplification?" and "can someone inject into a TCP connection they cannot see?". Strong answers separate off-path from on-path attackers, give the arithmetic of guessing, name a side channel that undid it, and end by placing real protection in an authenticated layer above.

Say it in 30 seconds: "IP doesn't authenticate source addresses, so spoofing works wherever networks skip BCP 38, and spoofed UDP makes reflection and amplification possible. TCP's handshake and random sequence numbers stop blind injection by making guessing expensive, and RFC 5961's exact matching made resets harder still, but side channels like the shared challenge-ACK counter have undone that arithmetic before. None of it protects against someone on the path, so availability defences like SYN cookies live here, and confidentiality and integrity come from TLS above."