§8.7Network-Layer Security: IPsec and VPNs

Network layer Kurose & Ross pp. 651–658 · ~14 min read

  • virtual private network
  • security association
  • tunnel mode

Where you are

  • Application layer
  • Transport layer
  • Network layer you are here
  • Link layer
  • Physical layer

Securing the network layer means securing everything above it at once — and doing that turns out to require wrapping each datagram inside another one.

Words you will meet

  • IPsec (IP Security) — the IP (Internet Protocol) security protocol suite, providing security at the network layer.

  • Virtual private network — an institution’s private traffic, carried encrypted over the public Internet.

  • Security association — a one-way agreement holding the keys and algorithms for one direction.

  • SPI (Security Parameter Index) — the 32-bit number naming which security association a datagram belongs to.

  • Tunnel mode — wrapping the whole original datagram, header included, inside a new one.

  • Blanket coverage — the property that everything above the network layer is protected without knowing about it.

Why this matters

Section 8.5 secured one application. Section 8.6 secured everything running over TCP (Transmission Control Protocol) . This secures everything, full stop.

When two network-layer entities have confidentiality between them, the sender encrypts the payload of every datagram it sends. That payload might be a TCP segment, a UDP (User Datagram Protocol) segment, an ICMP (Internet Control Message Protocol) message — it does not matter. E-mail, web pages, TCP handshakes and network management traffic are all hidden from anyone sniffing the path.

That is what “blanket coverage” means, and it is why the question at the end of section 8.5 — why bother securing higher layers at all? — has the answer it does. Blanket coverage cannot authenticate a user. It only knows machines.

Why a virtual private network exists

An institution spread over several regions wants its offices to exchange data securely. It could build a private network: its own routers, its own links, its own DNS (Domain Name System) , completely separate from the public Internet.

That works and it is expensive. The institution must buy, install and maintain physical infrastructure.

A virtual private network sends the same inter-office traffic over the public Internet instead — but encrypts it before it enters the Internet.

Figure 8.27 — a virtual private network
1 hops: host 172.16.1.17 → R1 — gateway 200.168.1.100
Headquarters — 172.16.1/24Branch office — 172.16.2/24public InternetIPsec — encrypted before it enters the InternetIPsec, laptop to headquartersordinary IPv4 — not everything is securedhost 172.16.1.17R1 — gateway 200.168.1.100R2 — gateway 193.68.2.23host 172.16.2.48salesperson in a hotelan ordinary web server

Drag any device to rearrange the picture. Hover a link to see its rate, delay and length.

Play the last journey. A gateway emits both secured IPsec datagrams and ordinary IPv4 ones, and something has to decide which is which — that decision is the Security Policy Database, further down this page.

Read this diagram as text
  • host 172.16.1.17 wired link R1 — gateway 200.168.1.100
  • R1 — gateway 200.168.1.100 wired link R2 — gateway 193.68.2.23
  • R2 — gateway 193.68.2.23 wired link host 172.16.2.48
  • salesperson in a hotel wired link R1 — gateway 200.168.1.100
  • R1 — gateway 200.168.1.100 wired link an ordinary web server

Not everything a gateway sends is secured

Play the last journey in the figure. A host at headquarters may want to reach an ordinary web server — Amazon, Google — out on the public Internet.

So the gateway router emits both plain IPv4 (Internet Protocol version 4) datagrams and IPsec datagrams, and something must decide which treatment each one gets. That decision is the Security Policy Database, near the end of this page, and it is easy to forget that it is needed at all.

Two protocols, and the one the book teaches

IPsec offers two protocols. AH (Authentication Header) provides source authentication and data integrity, but no confidentiality. ESP (Encapsulation Security Payload) provides source authentication, data integrity, and confidentiality.

For virtual private networks and other IPsec applications, ESP is much more widely used than AH. The book therefore focuses exclusively on ESP, and says so explicitly, to avoid much of IPsec’s complication.

This site follows the book. AH is real, and it is not covered here; the RFCs and the book’s references are where to find it.

Security associations

Before any secured datagram can be sent, the two entities establish a security association. It is a one-way logical connection — a simplex one, in the book’s word — so protecting traffic in both directions takes two.

Counting the associations in one VPN

Take the institution in the figure: a headquarters, a branch office, and n travelling salespeople, with two-way IPsec traffic on each path.

  • Headquarters gateway ↔ branch gateway: 2 associations, one per direction.
  • Headquarters gateway ↔ each laptop: 2 more, per salesperson.

So the total is 2 + 2n, and the headquarters gateway alone maintains state for all of them.

An IPsec entity stores every association’s state in its Security Association Database, a data structure in the operating system kernel.

What R1 stores for one security association
StateIn the Figure 8.28 example

Cells marked ⓘ have an explanation — click to read it.

A security association is one-way. Everything here describes traffic from R1 to R2 only; the reverse direction is a second association with its own state.

Building the datagram

IPsec has two packet forms: tunnel mode and transport mode. Tunnel mode suits virtual private networks and is more widely deployed, so it is the one covered here — again following the book, which sets transport mode aside deliberately.

R1's recipe, and R2 unwrapping it
1. Append the ESP trailerstep 1 of 6
original IPv4 datagram from 172.16.1.17 to 172.16.2.48original IPv4 datagramESP trailer

Padding to reach a whole number of cipher blocks, the pad length so the far end knows how much to remove, and a next-header field naming what kind of data the payload is.

Six steps out, six steps back. Watch step 5: the new header is added last, and it names the gateways rather than the hosts.

Read all steps as text
  1. 1. Append the ESP trailerPadding to reach a whole number of cipher blocks, the pad length so the far end knows how much to remove, and a next-header field naming what kind of data the payload is.
  2. 2. Encrypt all of thatThe original datagram AND the trailer are encrypted together, using the algorithm and key the security association specifies.
  3. 3. Prepend the ESP headerTwo fields, both sent in the clear: the SPI, which says which security association this belongs to, and a sequence number.
  4. 4. Compute the integrity check over everything so farA message authentication code over the ESP header plus the encrypted unit — computed after encryption, not before.
  5. 5. Prepend a brand new IP headerSource and destination are the two GATEWAYS, not the two hosts. Its protocol field is 50, which is how R2 knows to apply IPsec processing.
  6. R2 unwraps it, in six stepsDestination is R2 itself, so R2 processes it. Protocol 50 means IPsec ESP. Then: use the SPI to find the security association; verify the integrity check; check the sequence number for freshness; decrypt; remove the padding; and forward the original datagram into the branch office.
Figure 8.29 — the tunnel-mode IPsec datagram

Click any box to read what that part of the message is for.

Every field, as text

    Read it outward from the middle. The original datagram — header and all — is buried inside, and only the two outermost pieces are readable on the wire.

    In plain words

    The original datagram is not modified. It is put inside a new datagram, the way a letter goes inside a second envelope.

    The outer envelope is addressed gateway-to-gateway. The inner one still carries the real sender and the real recipient — and nobody on the path can read it.

    The book calls the whole assembly a Tex-Mex delicacy. This site calls it a wrapped datagram, because Mandate B does not permit the joke. The shape it describes is right: layers around layers, unwrapped in reverse.

    Two details in that recipe that are easy to miss

    The integrity check is computed after encryption, over the ESP header and the already-encrypted unit. Encrypt first, then authenticate what you produced — the opposite order from section 8.5’s e-mail design, where Alice signs first and encrypts second.

    The padding field is section 8.2.1 arriving in a packet format. Block ciphers require the message to be an integer multiple of the block length, so meaningless bytes are added to make it fit. A pad-length field says how many to remove. A constraint introduced five sections ago is now a field on the wire.

    Two databases: what, and how

    There is one more piece, and the book raises it as a question.

    When R1 receives an unsecured datagram from a headquarters host, bound somewhere outside, how does it know whether to apply IPsec at all? And if it should, which of the many associations in its database should it use?

    The answer is a second data structure. The Security Policy Database says which datagrams are to be IPsec-processed, as a function of source address, destination address and protocol type. It also says which association to use for each.

    The Security Policy Database says what to do with an arriving datagram. The Security Association Database says how to do it.

    What Trudy learns

    Put Trudy on the path between R1 and R2, without the keys, and ask what she can see.

    She cannot see the original datagram — and the phrase covers more than the data. The protocol number, the source address and the destination address are all inside the encrypted part.

    For every datagram crossing that association, all Trudy knows is that something travelled from 200.168.1.100 to 193.68.2.23. Not which host sent it, not which host receives it, not whether it is TCP or UDP or ICMP.

    The traffic she can still analyse

    Hiding the addresses does not hide that traffic exists. Trudy still sees a stream of datagrams between two gateways, and she can count them, time them and measure their sizes.

    That is a real residual channel, and it is why the book’s claim is carefully worded: she does not know what is being carried, not that nothing is being carried.

    The mental model

    Application-layer security protects one application. Transport-layer security protects one protocol’s users. Network-layer security protects a pair of machines, and everything those machines send between them.

    The price of that generality is the wrapper. The protection covers the whole datagram, header included, so the datagram can no longer route itself. A new header has to be built around it, naming the two gateways instead.

    Everything else on this page follows from that one move. The SPI, because the receiver must identify the association before it can decrypt. The databases, because a gateway also sends unprotected traffic. And the padding, because a block cipher was chosen four sections ago.

    Check yourself

    Check yourself

    0 of 7 answered
    1. 1.What does "blanket coverage" mean for network-layer security?

    2. 2.A VPN has a headquarters, a branch office, and 5 travelling salespeople, with two-way IPsec on every path. How many security associations are there?

    3. 3.Why must the SPI travel in the clear?

    4. 4.predictTrudy sits between R1 and R2 without the keys. Which of these can she read?

    5. 5.What is the difference between the Security Policy Database and the Security Association Database?

    6. 6.Why does a tunnel-mode datagram need a brand new IP header?

    7. 7.The ESP trailer contains a padding field. Where does that requirement come from?

    What to remember

    • Tunnel mode hides the addresses, which is why it needs new ones. The original header is encrypted along with everything else, so Trudy learns only that something travelled between the two gateways. A fresh outer header is what lets the Internet still route it.

    • A security association is one-way, so a VPN needs 2 + 2n of them for a headquarters, a branch office and n travelling salespeople. Each carries its own keys, algorithms and endpoints.

    • Two databases answer two different questions. The Security Policy Database says what to do with a datagram — secure it or send it plain — and the Security Association Database says how.