Securing Data in Transit with IPSec

Network security has many facets, and much emphasis is placed (rightly) on keeping intruders and attackers out of the network via firewalls. However, in today’s business environment, there are also many instances in which sensitive data needs to be protected within the local network from users who have legitimate access to the network – but do not need to have access to the data in question.

The answer in that case is encryption (and encrypting data also provides an extra layer of security against intruders who do manage to get into the network). With Microsoft operating systems prior to Windows 2000, encrypting data required third party software. Now encryption capabilities are built into the OS. These include the Encrypting File System (EFS) and Internet Protocol Security (IPSec).

The type of encryption you need to use depends on the state of the data. File encryption can protect data residing on disk, but does not protect that data when it’s in transit over the network. If you don’t believe this, send an EFS-encrypted file across the network and capture the packets in transit. You’ll see that the data is readable. IPSec is needed to encrypt to protect data from network sniffers.

A Brief History of IPSec

IPSec is an industry standard set of protocols and services based on cryptography, used to encrypt data so that it cannot be read or tampered with during its journey across an IP network. There are a number of RFCs that provide specifications for IPSec and its protocols, as defined by the Internet Engineering Task Force (IETF). Good starting points are RFC 1825 and 2401, which deal with the security architecture for IP ( and IPSec can be used with the current IPv4, and is built into the next generation of IP, IPv6.

What IPSec Does

IPSec is designed to provide authentication (verification of the identity of the sender), integrity (assurance that the data was not changed in transit) and confidentiality (encryption of the data so that it can’t be read by anyone who doesn’t have the correct key).

Because it operates at the network layer of the OSI model (Layer 3), IPSec has an advantage over SSL and other methods that operate at higher layers. Applications must be written to be aware of and use SSL, while applications can be used with IPSec without being written to be aware of it. Thus encryption occurs transparently to the upper layers.

IPSec protects only IP-based traffic; it is of no use to other network layer protocols such as IPX. There are also some types of IP traffic (such as Kerberos) that are not protected by Microsoft’s implementation of IPSec by default. Microsoft calls these exemptions.

IPSec Protocols, Modes and Security Associations

IPSec is not a single protocol; rather, it is made up of two protocols, which can be used separately or together:

  • AH (Authentication Header). As the name implies, AH is used to authenticate the identity of the sender, and to provide integrity of the data to ensure that it hasn’t been modified. It does not encrypt data, and provides no confidentiality. AH signs the entire packet.
  • ESP (Encapsulating Security Payload). ESP can provide confidentiality by encrypting the data itself, along with authentication and integrity. However, ESP generally doesn’t sign the entire packet, only the data.

To protect the IP header as well as the data itself, AH and ESP can be used together.

There are two modes of operation for both AH and ESP:

  • Tunnel mode, which is used to create a virtual private network. Tunnel mode provides gateway to gateway (or server to server) protection.
  • Transport mode, which is used to encrypt data inside a tunnel that is created by L2TP (the layer 2 tunneling protocol). Transport mode provides end-to-end security, all the way from the sending computer to the final destination.

The two computers that are communicating via IPSec establish a security association (SA). This represents the “agreement” between the two about the way the data will be exchanged and protected. Thus both of these computers must support IPSec. IPSec support is built into Windows 2000 (Server and Pro) and XP Pro computers and will be included in Windows Server 2003.

How IPSec Works in Windows

Microsoft and Cisco worked together to develop the implementation of IPSec that is included in Windows 2000 and later operating systems. Cisco’s ISAKMP/IKE is used along with Microsoft’s IPSec driver.

Internet Key Exchange (IKE) negotiates the security associations during two phases: ISAKMP phase (phase 1) and IPSec phase (phase 2). See RFC 2409 for more information about IKE and its components, ISAKMP and Oakley. Another IPSec component, the Policy Agent, distributes IPSec polices that are created by the administrator. The IPSec policies can be stored in Active Directory or in the local configuration policies. The Policy Agent is called IPSec Services in Windows XP.

To use IPSec in Windows 2000/XP, you must define an IPSec policy that specifies the authentication method and IP filters to be used. There are three authentication methods to choose from: Kerberos (the default), certificates, or preshared keys. Preshared keys are not recommended for sensitive environments, because the key is stored as plain text in the database where the IPSec policies are stored, and thus poses a security risk.

How to Configure a Windows 2000 Pro Computer to use IPSec

Setting up your Windows 2000 computer to use IPSec is relatively simple. Remember that both the sending and receiving computers must support IPSec. Also, you must be an administrator to set IPSec policies. The following steps will configure your system to take advantage of IPSec encryption:

  1. Click the Start menu, then select Settings | Network and Dialup Connections.
  2. Right click the connection you want to configure for IPSec communications.
  3. Choose Properties. On the General tab, under Components used by this connection, choose Internet Protocol (TCP/IP). Click the Properties button (see Figure 1).


Figure 1

4. On the TCP/IP properties sheet, click the Advanced button (See Figure 2).


Figure 2

5. Choose the Options tab and select the IP Security optional setting (see Figure 3).


Figure 3

6. Click the Properties button.
7.Check the option button that says Use this IP Security policy to enable IPSec communications (see Figure 4). NOTE: If the options are greyed out and cannot be changed, this usually means the computer belongs to an Active Directory domain and gets its IPSec policies from Active Directory.


Figure 4

There are three predefined IPSec policies you can choose from: Client (respond only), Server (request security) and Secure Server (require security). The Client policy is used if you do not want IPSec to be used unless the server with which you are communicating requests it. The Server policy causes the computer to try to negotiate an IPSec connection, but if the server on the other end doesn’t support IPSec or isn’t configured to use it, your computer will go ahead and communicate via unsecured communications. The Secure Server policy is used if you want the computer to send and accept IPSec secured communications only. If the computer on the other end can’t use IPSec, your computer will reject all traffic from it. This is the most highly secure setting.

IPSec Policies

You may need customized policies to fit your organization’s needs, rather than using the predefined policies. IPSec policies can be created, changed and managed via the IP Security Policy MMC. Create a custom MMC and add the IPSec snap-in. You can also access local IPSec policies using the Local Security Settings tool in the Administrative Tools menu.

Under Security Settings in the left pane, expand the Public Key Policies node and click IP Security Policies. You’ll see the three default predefined policies in the right details pane. Any custom policies you create will also be listed here (see Figure 5).


Figure 5

Creating an IPSec Policy

To define a new policy, right click IP Security Policies on Local Machine in the left console pane. Select Create IP Security Policy (see Figure 6).


Figure 6

The IP Security Policy Wizard will start. The wizard will ask you to provide a name and description for the new policy. Next, you’ll be asked to decide how the policy should respond to requests for secure communications. On the next page, you’ll be asked to set an initial authentication method. Kerberos is the default, or you can select certificates (you’ll have to specify a CA) or preshared key (this is a secret string of characters that must be shared by the two computers that are communicating via IPSec). On the last page of the wizard, click Finish to create the policy. You can edit its properties later by double clicking it in the right console pane or right clicking it and choosing Properties.

You can add and edit rules for the policy by selecting the Rules tab (see Figure 7).


Figure 7

Clicking Add will invoke another wizard, the Security Rule Wizard. The steps of this wizard include the following:

  1. Define whether this rule will cause an IPSec tunnel to be created. IPSec tunneling is used to create a virtual private network link. If you specify that a tunnel will be created, you must provide the IP address of the computer that will serve as the endpoint of the tunnel.
  2. Select the type of network connection to which the rule is to be applied. You can choose from the following: all network connections, local area network (LAN) connections, or remote access connections. (The default setting is all connections).
  3. Select an initial authentication method for the rule (Windows 2000 Kerberos, certificate, or a preshared key).
  4. Choose the type of IP traffic to which the rule will apply. Default choices are: All ICMP Traffic and All IP Traffic.

You can add additional filters by selecting the Add button on the IP Filter List screen. This will invoke the Filter Wizard. Select a filter action for the rule. Default actions you can choose from include:

  • Permit (allows unsecured packets to pass through)
  • Request Security – Optional (negotiates security; will accept unsecured communications, but always responds using IPSec; will allow unsecured communications if the other computer is not IPSec-aware)
  • Require Security (will not allow unsecured communications with non IPSec-aware computers).

In most cases, the predefined filter actions will work, but you can also create custom filter actions. The Filter Action Wizard is used for this purpose. You can choose which IPSec protocol(s) will be used with the action – ESP, AH, or both. You can also specify which integrity and encryption algorithms are to be used and how often a new key is to be generated.

Assigning Policies

Your new policy cannot be used for establishing IPSec connections until it has been assigned. No policies are assigned by default, but it’s easy to assign a policy. Just right click it in the right details pane of the MMC and select Assign from the context menu. If you don’t want it to be used any longer, follow the same procedure and select Unassign