Before we launch into how to establish, test and troubleshoot a connection step by step, here's a brief overview that may allow you to skim ahead to the most useful part of this document for you.
There are several general places where you might have a problem:
This document also contains notes which expand on points made in these sections, and tips for problem reporting. If the other end of your connection is not FreeS/WAN, see also our interoperation document.
Instructions and tips are in our install document. If you encounter a problem, it may be:
Missing library. See this FAQ.
Missing utilities required for compile. See this checklist.
Kernel version incompatibility. See this FAQ.
Another compile problem. Find information in the out.* files, ie. out.kpatch, out.kbuild, created at compile time in the top-level Linux FreeS/WAN directory. Error messages generated by KLIPS during the boot sequence are accessible with the dmesg command
Check the list archives and the List in Brief to see if this is a known issue. If it is not, report it to the bugs list as described in our problem reporting section. In some cases, you may be asked to provide debugging information using gdb; details below.
If your kernel compiles but you fail to install your new FreeS/WAN-enabled kernel, review the sections on installing the patched kernel, and testing to see if install succeeded.
Know your IPSec needs. For most people who begin experimenting with IPSec in the field, your configuration will more or less resemble one of the samples in our configuration document . Other folks may wish to create a testbed network in a lab environment for intensive IPSec testing or proof-of-concept. They may be interested in this description of a sample testbed network.
Draw a network schematic. This will assist others in helping you should you mail the list for help.
For example:
Sunset==========West------------------East
corporate LAN untrusted net
Add IPs to the diagram.
Sunset==========West------------------East
corporate LAN untrusted net
192.168.1.5 N.N.N.235
eth0(external)=N.N.N.222
eth1(internal)=192.168.1.1;
gateway for 192.168.0.0/24
How many tunnels do you need to connect your sites sufficiently for your purposes? Check our configuration document for why you might need multiple tunnels. How would you name each tunnel?
Above, it is possible to make two tunnels: sunset-east and west-east. If East does not need to access any resources on West, and West is not masquerading (see below) we may only want to create sunset-east. Sunset-east is a tunnel between IPSec-enabled machines West and East, created to protect traffic between the net on which Sunset resides, and East. However, it is safest to create all potential tunnels in your configuration. This lowers the risk that you will forget to configure a needed tunnel, and send important data cleartext. As an added safeguard, by default, (at the time of writing) Linux FreeS/WAN prevents you from routing cleartext packets between IPSec gateways which are also linked by a tunnel. Reliance on this behaviour is not a substitute for secure network design.
Take into account any masquerading and Network Address Translation rules on your gateway. If you are masquerading packets from Sunset as they leave West, Linux FreeS/WAN will treat these as though they originated at West, not Sunset. For more detail, see this packet flow diagram.
For each tunnel, think of a packet path that will allow you to test that tunnel. Refer to this discussion.
Ensure that IKE packets can travel freely between your IPSec gateways, as described here. If they cannot, the connection negotiation, described below, will be stuck in its first state.
Bring up one of your connections, using the ipsec auto commands at the command line:
ipsec auto --add west-east
then
ipsec auto --up west-east
At this stage of testing, do not bring up the connections by the alternate method, using ipsec.conf's auto= directive. You want to view the output as it happens.
If the resulting status report shows that you have established an ISAKMP and an IPSec Security Association (aka "SA", loosely translated as "tunnel" or "connection"), your tunnel is up. Repeat for each tunnel you are testing.
If negotiations for any one tunnel fail, troubleshoot as indicated in the next section. If you have successfully established all desired tunnels, proceed to test your connection(s) below. If you find that each tunnel in a multitunnel config may be created individually, but all may not be created at once, you may have encountered an old (1.6) bug. Update your Linux FreeS/WAN.
When you bring a tunnel up from the command line, you see a report on the negotiations involved in creating the connection, as these happen. There are also:
ipsec look, which provides a brief status report
ipsec auto --status, included in the barf
log files. More information about the log files is available here.
Often, the most relevant state information appears last in a log or status report.
Negotiations will proceed through various states. You will know these are done and a connection is established when you see both messages:
000 #21: "myconn" STATE_MAIN_I4 (ISAKMP SA established)...
000 #2: "myconn" STATE_QUICK_I2 (sent QI2, IPsec SA established)...
The key phrases are "ISAKMP SA established" and "IPSec SA established", which should appear with the relevant connection name. Often, this happens at STATE_MAIN_I4 and STATE_QUICK_I2, respectively.
A note on ipsec auto --status: this will tell you what states have been achieved, rather than the current state. Since determining the current state is rather more difficult to do, current state information is not available from Linux FreeS/WAN. If you are actively bringing a connection up, the status report's last states for that connection likely reflect its current state. Beware, though, of the case where a connection was correctly brought up but is now downed: Linux FreeS/WAN will not notice this until it attempts to rekey. Meanwhile, the last known state indicates that the connection has been established.
Linux FreeS/WAN proceeds though IKE (Phase 1, Main Mode, STATE_MAIN_*) negotiations first, then begins IPSec (Phase 2, Quick Mode, STATE_QUICK_*) negotiations. If you do not see success, note the place where negotiations stopped. This information is useful, since there are common errors specific to certain points in the process.
Look for verbose error text in the logs. While ipsec --auto dialog will tell you at which state Linux FreeS/WAN failed, it lacks detail. You can get more detail by modifying it with the --verbose flag on each invocation. For example:
ipsec auto --verbose --up west-east
Complete information can be gleaned from the log files.
The amount of description in the logs depends on ipsec.conf debug settings, klipsdebug= and plutodebug=. See the ipsec.conf(5) man page for details. You will normally want to set these to either "none" or "all". Note that you must have enabled the klipsdebug compile-time option for the klipsdebug configuration switch to work.
For negotiation problems, plutodebug is most relevant. klipsdebug applies mainly to attempts to use an already-established connection. See also this description of the division of duties within Linux FreeS/WAN.
After raising your debug levels, restart Linux FreeS/WAN to ensure that ipsec.conf is reread, then recreate the error to generate verbose logs.
This is a good time to produce a barf file, a collection of information useful for debugging Linux FreeS/WAN on your system. Use the command
ipsec barf > barf.west
See also the ipsec_barf(8) man page.
Look at the logs within the resulting file, and find the failure point. Are there a handful of lines which succinctly describe how things are going wrong or contrary to your expectation? Sometimes the failure point is not immediately obvious: Linux FreeS/WAN's errors are usually not marked "Error". Have a look in the FAQ for what some common failures look like. Tip: problems snowball. Focus your efforts on the first problem, which is likely to be the cause of later errors.
Repeat the process to find meaningful error text on the peer IPSec box. If the other end is not Linux FreeS/WAN, get it to produce detailed log output while you replicate the error, then capture that output to a file.
It is useful if both ends store information about the same event from two perspectives. Sometimes you will require information which only one side has. In this case, the peer can merely indicate the presence of an error, and its approximate point in the negotiations. If one side keeps retrying, it may be because there is a show stopper on the other side. Have a look at the other side and figure out what it doesn't like.
To interpret Linux FreeS/WAN log text, use the following resources:
the FAQ . Since this document is constantly updated, the snapshot's FAQ may have a new entry relevant to your problem.
our background document . Special considerations which, while not central to Linux FreeS/WAN, are often tripped over. Includes problems with packet fragmentation, and considerations for testing opportunism.
the list archives. Each of the searchable archives works differently, so it's worth checking each. Use a search term which is generic, but identifies your error, for example "No connection is known for".
Often, you will find that your question has been answered in the past. Finding an archived answer is quicker than asking the list. You may, however, find similar questions without answers. If you do, send their URLs to the list with your trouble report. The additional examples may help the list tech support person find your answer.
Look into the code where the error is being generated. The pluto code is nicely documented with comments and meaningful variable names.
If you have failed to solve your problem with the help of these resources, send a detailed problem report to the users list, following these guidelines.
Test a connection by sending packets through it. The simplest way to do this is with ping. Remember, in the planning stage, choosing a path which would test the tunnel? Now, ping along that path.
If your ping returns, test any other connections you've brought up. If they all check out, great. You may wish to test with large packets for MTU problems.
If your ping fails to return, generate an ipsec barf debugging report on each IPSec gateway. On a non-Linux FreeS/WAN implementation, gather equivalent information. Use this, and the tips in the next sections, to troubleshoot. Are you sure that both endpoints are capable of hearing and responding to ping?
IPSec may be dropping your ping packets since it does not "think" they belong in the tunnels you have constructed. This is an error about assumptions, and it takes two forms.
In the first, your ping is not returning because its path does not fall within your tunnel. This ping does not test the tunnel you intend to test. Referring to this discussion about appropriate tests, determine an alternate ping path which would test the tunnel.
In the second form of this error, you have not correctly configured the functionality you want. In the example above, you may have configured one of the possible tunnels between West and East (say west-east) but not the tunnel required to secure the important traffic you're now testing (say a sunset-east tunnel to secure traffic between office net Sunset and laptop East). See also this FAQ titled "I can't ping". NAT and masquerading may have an effect on which tunnels you need to configure; that's discussed in " Before there's trouble".
Both forms show identical symptoms. After all, the difference is one of intent. In both forms, Linux FreeS/WAN receives a packet destined for a peer IPSec gateway. Finding no active tunnel in which this packet belongs, it drops the packet on the floor. If your debug levels are appropriate, it logs this with a "klipsdebug... no eroute" message, which is discussed in this FAQ.
Note: When testing a tunnel that protects a multi-node subnet, you may wish to try several subnet nodes as ping targets, in case one node is routing incorrectly.
If you've confirmed your configuration assumptions, the problem is almost certainly with routing or firewalling. Isolate the problem using interface statistics, firewall statistics, or a packet sniffer.
Background:
Linux FreeS/WAN supplies all the special routing it needs; you need only route packets out through your IPSec gateway. Verify that on the machines you are using for your ping-test, your routing is as expected. I have seen a tunnel "fail" because the subnet machine was not routing packets back to the IPSec gateway; rather, it routed them through an alternate gateway.
Linux FreeS/WAN requires special firewalling considerations, described in our firewalling document. Check the firewall rules on your IPSec gateways and ensure that they allow IPSec traffic through. Be sure that no other machine - for example a router between the gateways - is blocking your IPSec packets.
Interface reports and firewall statistics can help you track down lost packets at a glance.
Check any firewall statistics you may be keeping on your IPSec gateways, for dropped packets.
Both cat /proc/net/dev and ifconfig display interface statistics, and both are included in an ipsec barf. Use either to check if any interface has dropped packets. If you find that one has, test whether this is related to your ping. While you ping continuously, print that interface's statistics several times. Does its drop count increase in proportion to the ping? If so, check why the packets are dropped there.
Check the firewall rules that apply to that interface. If the interface is an IPSec interface, more information may be available in the log. Grep for the word "drop" in a log which was created with klipsdebug=all as the error happened.
See also this detailed discussion by KLIPS programmer Richard Guy Briggs on interpreting ifconfig.
If you have checked configuration assumptions, routing, and firewall rules, and your interface statistics yield no clue, it remains for you to investigate the mystery of the lost packet by the most thorough method: with a packet sniffer. Sniff packets at each interface along the projected ping path until you find where your packets disappear. In this way, you can isolate the problem area, and narrow your troubleshooting focus.
Install an up-to-date sniffer (tcpdump, ethereal, ksnuffle) on your IPSec gateway machines. A sniffer on the ping endpoints is also useful. The sniffer should be somewhat modern (tcpdump 3.3+, ethereal-0.8.18) so that you may view packets on the ipsec virtual interface as well as the underlying physical one.
Working from your schematic, anticipate your ping's path. Which machines will your ping be visible on, in which order? Now, which interfaces will it be visible on, in which order? Within a machine running Linux FreeS/WAN, this packet flow diagram will help you anticipate the packet's path. Note that from the perspective of the tunneled packet, the entire tunnel is one hop. That's explained in this FAQ.
Ping, and as you do, sniff the packets. Examine each interface along the projected path, checking for your ping's arrival. If it doesn't arrive at the next stop, you have narrowed down where to look for it.
Note that an encapsulated IPSec packet will look different, when sniffed, from the plaintext packet which generated it. However, you can observe plaintext packets entering an IPSec interface and the resulting cyphertext packets as they emerge from the corresponding physical interface.
Once you isolate where the packet is lost, take a closer look at firewall rules, routing and configuration assumptions as they affect that specific area. If the packet is lost on an IPSec gateway, comb through klipsdebug output for anomalies.
If the packet goes through both gateways successfully and reaches the ping target, but does not return, suspect routing. Check that the ping target routes packets back to the IPSec gateway.
The guidelines are the same as for the Pluto logs, above.
For connection use problems, set klipsdebug=all. Note that you must have enabled the klipsdebug compile-time option to do this. Restart Linux FreeS/WAN so that it rereads the configuration file, then recreate the error condition. When searching through klipsdebug data, look especially for the keywords "drop" (as in dropped packets) and "error".
Often the problem with connection use is not software error, but rather that the software is behaving contrary to expectation.
To interpret the Linux FreeS/WAN log text you've found, use the same resources as indicated for troubleshooting connection negotiation: the FAQ , our background document , and the list archives. Looking in the KLIPS code is recommended only for the brave.
If you are still stuck, send a detailed problem report to the users' list.
If each of your connections passed the ping test, you may wish to test by pinging with large packets (2000 bytes or larger). If it does not return, suspect MTU issues, and see this discussion.
In most users' view, a simple ping test, and perhaps a large-packet ping test suffice to indicate a working IPSec connection.
Some people might like to do additional stress tests prior to production use. They may be interested in this testing protocol we use at interoperation conferences, aka "bakeoffs". We also have a testing directory that ships with the release.
Ask for troubleshooting help on the users' mailing list, users@lists.freeswan.org. While sometimes an initial query with a quick description of your intent and error will twig someone's memory of a similar problem, it's often necessary to send a second mail with a complete problem report.
The essay How to Report Bugs Effectively contains good guidelines.
When reporting problems to the mailing list(s), please include:
a brief description of the problem
if it's a compile problem, the actual output from make, showing the problem. Try to edit it down to only the relevant part, but when in doubt, be as complete as you can. If it's a kernel compile problem, any relevant out.* files
if it's a run-time problem, pointers to where we can find the complete output from "ipsec barf" from BOTH ENDS (not just one of them). Remember that it's common outside the US and Canada to pay for download volume, so if you can't post barfs on the web and send the URL to the mailing list, at least compress them with tar or gzip.
If you can, try to simplify the case that is causing the problem. In particular, if you clear your logs, start FreeS/WAN with no other connections running, cause the problem to happen, and then do ipsec barf on both ends immediately, that gives the smallest and least cluttered output.
any other error messages, complaints, etc. that you saw. Please send the complete text of the messages, not just a summary.
what your network setup is. Include subnets, gateway addresses, etc. The network schematic is a good format for this information.
exactly what you were trying to do with Linux FreeS/WAN, and exactly what went wrong
a fix, if you have one. But remember, you are sending mail to people all over the world; US residents and US citizens in particular, please read doc/exportlaws.html before sending code -- even small bug fixes -- to the list or to us.
When in doubt about whether to include some seemingly-trivial item of information, include it. It is rare for problem reports to have too much information, and common for them to have too little.
To report a problem, send mail about it to the users' list. If you are certain that you have found a bug, report it to the bugs list. If you encounter a problem while doing your own coding on the Linux FreeS/WAN codebase and think it is of interest to the design team, notify the design list. When in doubt, default to the users' list. More information about the mailing lists is found here.
For a number of reasons -- including export-control regulations affecting almost any private discussion of encryption software -- we prefer that problem reports and discussions go to the lists, not directly to the team. Beware that the list goes worldwide; US citizens, read this important information about your export laws. If you're using this software, you really should be on the lists. To get onto them, visit lists.freeswan.org.
If you do send private mail to our coders or want a private reply from them, please make sure that the return address on your mail (From or Reply-To header) is a valid one. They have more important things to do than to unravel addresses that have been mangled in an attempt to confuse spammers.
The following sections supplement the Guide: information available on your system; testing between security gateways; ifconfig reports for KLIPS debugging; using GDB on Pluto.
Linux FreeS/WAN logs to:
/var/log/secure (or, on Debian, /var/log/auth.log)
/var/log/messages
Check both places to get full information. If you find nothing, check your syslogd.conf(5) to see where your /etc/syslog.conf or equivalent is directing authpriv messages.
Other man pages are on this list and in
/usr/local/man/man3
/usr/local/man/man5
/usr/local/man/man8/ipsec_*
Sometimes you need to test a subnet-subnet tunnel. This is a tunnel between two security gateways, which protects traffic on behalf of the subnets behind these gateways. On this network:
Sunset==========West------------------East=========Sunrise
IPSec gateway IPSec gateway
local net untrusted net local net
you might name this tunnel sunset-sunrise. You can test this tunnel by having a machine behind one gateway ping a machine behind the other gateway, but this is not always convenient or even possible.
Simply pinging one gateway from the other is not useful. Such a ping does not normally go through the tunnel. The tunnel handles traffic between the two protected subnets, not between the gateways . Depending on the routing in place, a ping might
either succeed by finding an unencrypted route
or fail by finding no route. Packets without an IPSEC eroute are discarded.
Neither event tells you anything about the tunnel. You can explicitly create an eroute to force such packets through the tunnel, or you can create additional tunnels as described in our configuration document, but those may be unnecessary complications in your situation.
The trick is to explicitly test between both gateways' private-side IP addresses. Since the private-side interfaces are on the protected subnets, the resulting packets do go via the tunnel. Use either ping -I or traceroute -i, both of which allow you to specify a source interface. (Note: unsupported on older Linuxes). The same principles apply for a road warrior (or other) case where only one end of your tunnel is a subnet.
When diagnosing problems using ifconfig statistics, you may wonder what type of activity increments a particular counter for an ipsecN device. Here's an index, posted by KLIPS developer Richard Guy Briggs:
Here is a catalogue of the types of errors that can occur for which
statistics are kept when transmitting and receiving packets via klips.
I notice that they are not necessarily logged in the right counter.
. . .
Sources of ifconfig statistics for ipsec devices
rx-errors:
- packet handed to ipsec_rcv that is not an ipsec packet.
- ipsec packet with payload length not modulo 4.
- ipsec packet with bad authenticator length.
- incoming packet with no SA.
- replayed packet.
- incoming authentication failed.
- got esp packet with length not modulo 8.
tx_dropped:
- cannot process ip_options.
- packet ttl expired.
- packet with no eroute.
- eroute with no SA.
- cannot allocate sk_buff.
- cannot allocate kernel memory.
- sk_buff internal error.
The standard counters are:
struct enet_statistics
{
int rx_packets; /* total packets received */
int tx_packets; /* total packets transmitted */
int rx_errors; /* bad packets received */
int tx_errors; /* packet transmit problems */
int rx_dropped; /* no space in linux buffers */
int tx_dropped; /* no space available in linux */
int multicast; /* multicast packets received */
int collisions;
/* detailed rx_errors: */
int rx_length_errors;
int rx_over_errors; /* receiver ring buff overflow */
int rx_crc_errors; /* recved pkt with crc error */
int rx_frame_errors; /* recv'd frame alignment error */
int rx_fifo_errors; /* recv'r fifo overrun */
int rx_missed_errors; /* receiver missed packet */
/* detailed tx_errors */
int tx_aborted_errors;
int tx_carrier_errors;
int tx_fifo_errors;
int tx_heartbeat_errors;
int tx_window_errors;
};
of which I think only the first 6 are useful.
You may need to use the GNU debugger, gdb(1), on Pluto. This should be necessary only in unusual cases, for example if you encounter a problem which the Pluto developer cannot readily reproduce or if you are modifying Pluto.
Here are the Pluto developer's suggestions for doing this:
Can you get a core dump and use gdb to find out what Pluto was doing
when it died?
To get a core dump, you will have to set dumpdir to point to a
suitable directory (see ipsec.conf(5)).
To get gdb to tell you interesting stuff:
$ script
$ cd dump-directory-you-chose
$ gdb /usr/local/lib/ipsec/pluto core
(gdb) where
(gdb) quit
$ exit
The resulting output will have been captured by the script command in
a file called "typescript". Send it to the list.
Do not delete the core file. I may need to ask you to print out some
more relevant stuff.
Note that the dumpdir parameter takes effect only when the IPsec subsystem is restarted -- reboot or ipsec setup restart.