- Understanding the networking technologies and protocols that underlie the problem.
- Determining the different elements of the problem and their attributes.
- Determining which troubleshooting steps and tools should be applied to resolve the problem.
I displayed these items as a bullet list instead of a numbered list because network troubleshooting generally isn't as easy as 1-2-3. In other words, it's often more of an art (i.e. based on intuition) than a science (based on a methodology).
- They are used to store the information about other subnets on the network and how you can reach hosts on these networks.
- They are used to determine which host (called the next-hop IP address) each packet should be forwarded to in order to reach the host this packet is ultimately destined for.
- They are used to determine which network interface (called the next-hop interface) should be used to forward this packet so it gets to its ultimate destination.
Understanding routing tables is therefore essential if you want to be able to effectively troubleshoot routing issues on a TCP/IP network. Let's look at how routing tables work, what they look like in different scenarios, and what troubleshooting steps and tools might be indicated in different situations. We'll begin by examining the routing table on a single-homed server (a server with a single network interface) that has a single IP address assigned to it. I've chosen this example because it's the easiest one to understand, and in next month's follow up article we'll look at more complicated scenarios including servers with multiple IP addresses (such as web servers) and servers with multiple network interfaces (such as servers that are connected to both the LAN and to a separate network used for running backup jobs).
Routing table for single-homed server with a single IP address
The following routing table is for a server that has IP address 172.16.11.30 on the 172.16.11.0/24 network:
C:\>route print IPv4 Route Table =========================================================================== Interface List 0x1 ........................... MS TCP Loopback interface 0x10003 ...00 03 ff 25 88 8c ...... Intel 21140-Based PCI Fast Ethernet Adapter (Generic) =========================================================================== =========================================================================== Active Routes: Network Destination Netmask Gateway Interface Metric 0.0.0.0 0.0.0.0 172.16.11.1 172.16.11.30 20 127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1 172.16.11.0 255.255.255.0 172.16.11.30 172.16.11.30 20 172.16.11.30 255.255.255.255 127.0.0.1 127.0.0.1 20 172.16.255.255 255.255.255.255 172.16.11.30 172.16.11.30 20 220.127.116.11 240.0.0.0 172.16.11.30 172.16.11.30 20 255.255.255.255 255.255.255.255 172.16.11.30 172.16.11.30 1 Default Gateway: 172.16.11.1 =========================================================================== Persistent Routes: None
To display this routing table, you open a command prompt window and type route print at the command line. Let's take this table apart so we can understand how it works.
Each routing entry (or route) in the routing table is composed of five fields:
- Network destination: An IP address or subnet representing a possible target destination to which IP packets may be forwarded.
- Netmask: A bitmask used for matching the destination field in a packet's IP address to one of the possible network destinations above.
- Gateway: The next-hop IP address to which the packet must be forwarded to reach the particular destination network.
- Interface: The next-hop interface which must be used to forward the packet to reach the particular destination network.
- Metric: The cost of the route.
Example 1: Destination host on local subnet
For our first example, let's say this particular server (172.16.11.30) has to send a packet to another host with IP address 172.16.11.80, which is on the same subnet. This packet will thus have source address 172.16.11.30 and destination address 172.16.11.80. Here is how Windows uses its routing table to decide which route to use:
- Windows first takes each route from the table in turn and performs a bitwise AND between the destination address in the packet (172.16.11.80) and the bitmask (Netmask) of the selected route. Here are the results, where each route in the table is identified by its network destination:
Route Netmask 172.16.11.80 AND Netmask 0.0.0.0 0.0.0.0 0.0.0.0 127.0.0.0 255.0.0.0 18.104.22.168 172.16.11.0 255.255.255.0 172.16.11.0 172.16.11.30 255.255.255.255 172.16.11.80 172.16.255.255 255.255.255.255 172.16.11.80 22.214.171.124 126.96.36.199 188.8.131.52 255.255.255.255 255.255.255.255 172.16.11.80
- For each route, the result of this ANDing is then compared with the Network Destination field of the route, and a match means the route can be used to forward the packet to its destination address. If more than one match is found, Windows uses the route with the longest match (the route whose Netmask has the highest number of 1 bits). If this doesn't result in a unique match, Windows uses the match that has the lowest cost (Metric). Finally, if more than one match has the same lowest cost, Windows arbitrarily chooses one of them as the route to use. From the table above, you can see that this ANDing process results in two matches (routes 1 and 3) so Windows chooses the one that has the longest match, which is row 3. The result of all this is that Windows now knows which route to use to get this packet to its destination. Here's what this route looks like in the server's routing table:
Network Destination Netmask Gateway Interface Metric 172.16.11.0 255.255.255.0 172.16.11.30 172.16.11.30 20
- Windows now uses the following algorithm to decide what to do next:
- If the route's Gateway field matches the address of one of the network interfaces on the server (or if the Gateway is empty) then Windows sends the packet directly to its destination address using the interface specified in the route.
- If the route's Gateway field does not match the address of any of the network interfaces on the server, then Windows sends the packet to the address in the route's Gateway field.
Clearly, condition A is the case here since the route's Gateway field (172.16.11.30) is the address assigned to the server's single network card. Windows therefore determines that the destination address is on the local subnet and that means Windows can send the packet directly to that address without needing to forward it to any routers. So in this case, Windows simply sends the packet to 172.16.11.80 using the server's 172.16.11.30 network interface, and the receiving host gets it.
Example 2: Destination host on remote subnet
Now let's go through the same process, but this time let's say the server is trying to send the packet to a host on a different subnet, say a host with the address 172.16.10.200. In other words, the packet has a source address of 172.16.11.30 and a destination address of 172.16.10.200. Here's how Windows uses its routing table to decide which route to use this time:
- Windows takes each route from the table and performs a bitwise AND between the destination address in the packet (172.16.10.200) and the bitmask (Netmask) of the route. The results this time are like this:
Route Netmask 172.16.10.200 AND Netmask 0.0.0.0 0.0.0.0 0.0.0.0 127.0.0.0 255.0.0.0 184.108.40.206 172.16.11.0 255.255.255.0 172.16.10.0 172.16.11.30 255.255.255.255 172.16.10.200 172.16.255.255 255.255.255.255 172.16.10.200 220.127.116.11 18.104.22.168 22.214.171.124 255.255.255.255 255.255.255.255 172.16.10.200
- For each route, the result of ANDing is compared with the Network Destination field of the route, and a match means the route can be used to forward the packet to its destination address. From our second table above, you can see this time that there is only one match i.e. row one where the route's Network Destination field 0.0.0.0 matches the result of the AND action. So the route that Windows will use to forward the packet to its destination will be the following route:
Network Destination Netmask Gateway Interface Metric 0.0.0.0 0.0.0.0 172.16.11.1 172.16.11.30 20
- Windows then uses the previously-described algorithm to decide what to do next, and this time condition B is the case here since the route's Gateway field (172.16.11.1) does not match the address assigned to the server's single network card, which is 172.16.11.30. Windows therefore determines that the destination address is on a remote subnet and that means Windows cannot send the packet directly to its destination but must instead forward the packet a router, which will then take care of getting the packet to its destination by forwarding it along further. So in this case, Windows sends the packet to the address shown in the Gateway field of the selected route (172.16.11.1) using the server's 172.16.11.30 network interface.
Once the router at 172.16.11.1 receives the packet, it determines what actions need to be taken next to forward the packet on to its final destination of 172.16.10.200, and of course this depends on whether the 172.16.11.10/24 network is a neighboring subnet to 172.16.11.11/24 (i.e. connected by a single router to it) or a remote network (connected by a series of routers with intermediate networks in between them).
What can fail in the above process? First, it's possible that Windows might be unable to select a route whose Network Destination field matches the bitwise AND of the Netmask field of the route and the packet's destination address. If this happens, you've got a routing error, and this will probably be indicated to you by some network application running on your server. What happens usually is that Windows uses TCP to notify the upper layer of the network stack that the packet can't be sent, and an error message of some kind usually results.
In this situation, you probably have either a corrupt routing table or an invalid persistent route in your routing table. Persistent routes are routes you add manually to the table using the route -p add command and which persist across reboots since their values are stored in the registry. If you add routes that are invalid, they can produce strange results, though most often they simply result in traffic being dropped mysteriously.
On the other hand, if the destination host is on a remote subnet and Windows forwards the packet to a router (the default gateway address) and this router can't select a route, then what usually happens in this case is that the route returns an ICMP message of "Destination Unreachable -- Host Unreachable" to the host that sent the packet. In this case, TCP will notify upper layers and some sort of error message will be displayed.
In either situation, a useful way to proceed is to examine the routing tables on the sending host and any intermediate routers along the way to the destination host, and see if these routing tables are consistent or look corrupted. A corrupted routing table can be restored (at least on Windows machines) by resetting the TCP/IP stack using the netsh int ip reset command (see KB299357 for details). Note that this reset operation does not remove persistent routes you've added to your routing table.
Now that you know a bit about how routing tables work and how to troubleshoot them, in the next article we'll look at more complex examples such as servers with multiple addresses and several network cards.
About the author:
About the author:
Mitch Tulloch is a writer, trainer and consultant specializing in Windows server operating systems, IIS administration, network troubleshooting, and security. He is the author of 15 books including the Microsoft Encyclopedia of Networking (Microsoft Press), the Microsoft Encyclopedia of Security (Microsoft Press), Windows Server Hacks (O'Reilly), Windows Server 2003 in a Nutshell (O'Reilly), Windows 2000 Administration in a Nutshell (O'Reilly), and IIS 6 Administration (Osborne/McGraw-Hill). Mitch is based in Winnipeg, Canada, and you can find more information about his books at his Web site: www.mtit.com.
This was first published in April 2007