Learn how to use Network Monitor 3.0 for troubleshooting a remote scripting error in this tip originally published on WindowsNetworking.com.
In the previous article, Troubleshooting the mystery error -- Managing Windows networks using scripts, we began troubleshooting a mysterious error that occurred when we tried to remotely change the IP address on an XP machine using the ChangeIPAddress.vbs script we developed. The mysterious error that occurred was this:
SWbemObjectEx: The remote procedure call failed
In the previous article, I mentioned that I had contacted some scripting gurus concerning this error, and the best answer I received was that a hotfix had probably broken WMI functionality and the result was that this script worked remotely but generated an error.
But an astute reader contacted me afterward with the following comment:
Imagine you just telnet into a server as administrator and change the IP address of the server. Will you lose the connection? It will hang a while and this is the same. But changing the default gateway on the server will not interrupt the existing (telnet) connection (assuming you do this from the same subnet). If you try to change the default gateway setting from remote site, you should experience the same delay."
Good point! How can we test this explanation?
Using Network Monitor 3.0
Microsoft recently released a new version of Network Monitor, a packet-sniffing tool that is included as part of Microsoft Systems Management Server. Network Monitor 3.0 has several enhancements over the previous version of this tool, namely, the following:
For more information about Network Monitor 3.0, see Paul Long's TechNet blog.
Here's my plan then. I'm going to use NM3 to capture a network trace from the machine on which I'm running the ChangeIPAddress.vbs script. My test setup for this is as follows:
Administrator workstation
Name: test124.test
To continue reading for free, register below or login
To read more you must become a member of SearchNetworking.com
.com
IP address: 172.16.11.124 (static)
Target machine
Name: test125.test.com
IP address: 172.16.11.125 (static)
Domain controller
Name: dc181.test.com
IP address: 172.16.11.181
But before I try to run ChangeIPAddress.vbs on test124 in order to change the IP address of test125, let's take a quick look at NM3.
When you start NM3, it looks like this (Figure 1):
[IMAGE]
Figure 1: Network Monitor 3.0 opening screen (Click the image for a larger view)
Before we go any further, let's select the Enable Conversations checkbox so we can view each type of protocol session that occurs during our trace.
Now click Create A New Capture Tab. This opens a new tab named Capture1 that we can use to create our network trace (Figure 2):
[IMAGE]
Figure 2: Opening a new capture tab (Click the image for a larger view)
Now let's test NM3 with something simple. We'll click the Play button to start a capture, and then from machine test124 we'll open a command prompt and type ping 172.16.11.125 -- we're pinging test125 from test124. The result is this (Figure 3):
[IMAGE]
Figure 3: Trace of pinging 172.16.11.125 (Click the image for a larger view)
This is just what we expect: two ARP packets (an ARP request followed by an ARP response) and then a series of ICMP packets (Echo Request messages followed by Echo Reply messages). If you know basic TCP/IP networking, this should be easy to understand.
Let's look at the "conversations" that occurred. Expand the My Traffic node to display these, shown in Figure 4:
[IMAGE]
Figure 4: Showing conversations (Click the image for a larger view)
Note that two conversations occurred: ARP and IPv4 (ICMP). Again, this should be pretty obvious if you know basic TCP/IP networking.
Let's now select the ARP Request packet and look inside it (Figure 5):
[IMAGE]
Figure 5: Examining a packet (Click the image for a larger view)
Now that we've had a quick introduction to NM3 (there's lots more!) let's try using it to troubleshoot our mystery error.
Capturing Traces
I'll start by rebooting both workstations to clear any caches (ARP, DNS, etc.) and then I'll open a command prompt on test124 and type ChangeIPAddress.vbs 172.16.11.144 in order to change the IP address of test125 from 172.16.11.125 to 172.16.11.144. (I've hard-coded the target computer as "test125" within this script.) Here's the result (Figure 6):
[IMAGE]
Figure 6: Result of running ChangeIPAddress.vbs 172.16.11.144 (Click the image for a larger view)
Here's an overview of what happened: The capture lasted almost 90 seconds in total, and there were 274 frames captured. The error message appeared around frame 241, and the command prompt returned at frame 274. (I know this because I watched the command output while the trace was being captured.) That's a lot of traffic to analyze! Looking at Figure 6 above, we can at least make a start at analyzing it:
…and so on.
Obviously we can't display all 274 frames in the figure, so I copied the Frame Summary information to a text file. (I also saved the capture as a .cap file). You can see the Frame Summary that resulted when we ran ChangeIPAddress.vbs here.
That's pretty overwhelming, isn't it? How can one begin to understand what this capture is telling you?
Well, when you're troubleshooting, a good place to begin is with what you know, not what you don't know. And we know that the other script (ChangeGateway.vbs) that we developed in our previous article worked without generating any error messages. So before we look further at ChangeIPAddress.txt, let's reboot our workstations and do another capture, this time showing the result of running the command ChangeGateway.vbs 172.16.11.2 1 on test124 in order to change the default gateway of test125 from 172.16.11.1 to 172.16.11.2 (and specifying the metric as 1). Here's what this second capture looks like (Figure 7):
[IMAGE]
Figure 7: Result of running ChangeGateway.vbs 172.16.11.2 1 (Click the image for a larger view)
This time there are only 217 frames to analyze (!) as you can see in the Frame Summary.
Analysis of Capture for ChangeGateway.vbs
Let's try and analyze this second capture (the one that worked without generating an error) by breaking the Frame Summary down piece by piece. Here it goes:
[TABLE]This is just NM3 header stuff -- ignore it.
[TABLE]Here is name and address resolution stuff (DNS and ARP):
[TABLE]Test124 just established a TCP connection with test125.
[TABLE]Test124 establishes an RCP binding with test125 and invokes DCOM.
TIP: If you're having trouble following the RPC portion of this trace, see KB 159258 for help.
[TABLE]Here's another TCP three-way handshake between the machines.
[TABLE]Kerberos authentication (the machines are both domain-joined).
[TABLE]There's more RPC and DCOM. I think "Alter Cont" indicates alternate context being used, but I'm actually not sure. Still, everything must be OK since the script worked without generating any errors.
[TABLE]Here we see another TCP handshake.
[TABLE]… And more Kerberos stuff.
[TABLE]There's lots of RPC/DCOM stuff in there. Looks cryptic, doesn't it? But if you look carefully you'll see some WMI stuff happening i.e. WMI-IWbemLoginClientID, WMI-IWbemLevel1Login, WMI-IWbemServices, WMI-IWbemFetchSmartEnum, and so on. Searching MSDN tells us more about what's happening here. For example, the Microsoft Developer Network library tells us that "The IWbemServices interface is used by clients and providers to access WMI services" so it looks like all these I-thingies are WMI interfaces (APIs) that are being called on the remote machine (using DCOM) by the workstation we're running our script from. And some of these interfaces actually seem to be undocumented, so we won't worry too much about trying to understand them.
From here on things get kind of dense. First there's a bunch more TCP stuff with RPC "Continued Response" packets that seem to indicate connections made earlier are being used for some purpose. I'm going to skip a few frames from this next portion of the trace:
[TABLE]Only two seconds have elapsed so far. Now there's a bunch of DCOM stuff followed by TCP connections terminating using FIN/ACKs, so I guess the script has probably done its job and is cleaning up now:
[TABLE]Now there's some DNS and LDAP stuff going on between test124 and the domain controller. I'm not sure why this is happening, but I'll skip some of these frames as there's a lot of them:
[TABLE]At this point the script has already ended so I terminated the capture.
Analysis of Capture for ChangeIPAddress.vbs
We now have a bit of an idea of what a capture of a successful remote script looks like:
And the whole thing took just over 2 seconds.
Now let's look at our capture for ChangeIPAddress.vbs (the script that generates an RPC error when we run it remotely) and see how it differs from the above.
[TABLE]This is just some Netmon stuff.
[TABLE]This is an ARP, a DNS, then a TCP handshake -- the same as before.
[TABLE]RPC, then DCOM, then another TCP handshake, then some Kerberos stuff. It looks the same as before.
[TABLE]We see the same pattern.
[TABLE]Then a whole bunch of RPC/DCOM stuff, just like the other trace.
[TABLE]Here's RPC together with TCP. You can see the WMI interface calls happening.
[TABLE]Uh-oh, what's this? The script has successfully changed the IP address of the target machine (test125) from 172.16.11.125 to 172.16.11.144, so why is the target machine using ARP to try and resolve its own IP address into a MAC address? This is an example of gratuitous ARP, which happens when a node makes an ARP request for its own IP address. Why does the target machine do this? To make sure it's new IP address 172.16.11.144 isn't being used by any other node on the network. If it issues several ARP requests and no ARP response is received, it decides that its new address is unique to the network and the address can be kept. But if another node issues an ARP response to this request, the first node assumes there is an address conflict on the network and it disables its IP address (assigns it to 0.0.0.0).
TIP: See Address Resolution Protocol (ARP) Chapter 3 of Microsoft® Windows® 2000 TCP/IP Protocols and Services Technical Reference by Thomas Lee and Joseph Davies if you want to learn more about gratuitous ARP.
At this point things seem to fall apart -- you can tell this by the fact that the time interval between packets is increasing significantly. What seems to be happening next is that the source node (test124) keeps trying to acknowledge TCP with the target but isn't getting anywhere:
[TABLE]Let's look at packet 159 above more closely using NM3 (Figure 8):
[IMAGE]
Figure 8: TCP connection problems (Click the image for a larger view)
Note from this figure that the source machine (test124) still thinks the destination machine has IP address 172.16.11.125, and it keeps trying to ACK with test125 to maintain the TCP connection established earlier.
Now let's look at frame 161 (Figure 9):
[IMAGE]
Figure 9: RPC/DCOM problems (Click the image for a larger view)
Note that the RPC binding established earlier by the source machine (test124) with the target machine (test125) is trying to invoke DCOM to call the EnableStatic Method of the Win32_NetworkAdapterConfiguration class. (To see this, look at the right side of the Hex Details pane where you can see the hex payload of the RPC packet displayed in UNICODE text.) But in trying to invoke DCOM, the source machine still thinks the destination address of the target machine is 172.16.11.125 (see the Frame Details pane in the figure).
So it looks like the reader was right!
The rest of the ChangeIPAddress.vbs capture is interesting to try and analyze, but it looks like we've identified the reason our remote script doesn't run properly. Well, it does work of course if we use the On Error Resume Next workaround that we mentioned in the previous article.
About the author:
Mitch Tulloch is a writer, trainer and consultant specializing in Windows server operating systems, IIS administration, network troubleshooting, and security. He is the author of 15 books including the Microsoft Encyclopedia of Networking (Microsoft Press), the Microsoft Encyclopedia of Security (Microsoft Press), Windows Server Hacks (O'Reilly), Windows Server 2003 in a Nutshell (O'Reilly), Windows 2000 Administration in a Nutshell (O'Reilly), and IIS 6 Administration (Osborne/McGraw-Hill). Mitch is based in Winnipeg, Canada, and you can find more information about his books at his website: www.mtit.com.
[TABLE]
WindowsNetworking.com contains a wealth of networking information for administrators: Featuring information on how to setup and troubleshoot various networks of any size. Also includes a comprehensive archive of hundreds of reviewed networking software and hardware solutions. Frequently updated with articles and tips by a team of leading authors, it remains a favorite within the networking community.
