Excessive ARP requests, is it malware or legitimate activity?
Hi there,
In this blog post, I would like to talk about a network trace analysis scenario where we were requested to find out if excessive ARP requests were legitimate or not.
Here are more details about the problem:
Problem description:
==================
Some Windows Vista clients experience network connectivity problems. After the client gets connected to the network, it is disconnected after a short while like 1 minute or so. After some initial troubleshooting, it was found out that the connectivity problem occurred because the switch port was disabled. And our customer’s network team stated that the switch port was disabled because excessive ARP requests were received from the client in a very short amount of time (like getting 10+ ARP requests within 10 milliseconds from the same source MAC etc).
How we did the troubleshooting:
=============================
We collected a network trace to better understand the root cause of the problem from one of those client machines. But the problem was that we were not able to see what the client would do after resolving those MAC addresses by ARP protocol because the switch was shutting down the port due to the mentioned security setting.
It’s also very difficult to associate an ARP packet to a process on a Windows machine by using the methods mentioned in my earlier post. So I decided to use the method 5 mentioned in the same post). I requested the customer to disable that security setting on a switch port where the problem was seen. So that we would be able to see what the client would do after resolving the MAC addresses via ARP protocol. We collected a new network trace after making such a configuration change and got to the bottom of the problem afterwards:
a) In the new network trace, it was clearly seen again that the client was sending excessive ARP requests (20 ARP requests sent for different IP addresses within the same millisecond!)
Wireshark filter:
arp.src.proto_ipv4 == 10.0.80.1 and arp.opcode==0x1
No. Time Source Destination Protocol Info
326437 30.192068 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.202? Tell 10.0.80.1
326438 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.165? Tell 10.0.80.1
326440 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.83? Tell 10.0.80.1
326441 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.59? Tell 10.0.80.1
326442 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.220? Tell 10.0.80.1
326443 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.215? Tell 10.0.80.1
326451 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.28? Tell 10.0.80.1
326453 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.60? Tell 10.0.80.1
326454 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.127? Tell 10.0.80.1
326455 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.81? Tell 10.0.80.1
326458 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.29? Tell 10.0.80.1
326462 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.98? Tell 10.0.80.1
326465 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.65? Tell 10.0.80.1
326467 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.129? Tell 10.0.80.1
326468 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.37? Tell 10.0.80.1
326470 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.239? Tell 10.0.80.1
326471 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.15? Tell 10.0.80.1
326472 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.26? Tell 10.0.80.1
326473 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.235? Tell 10.0.80.1
326475 0.000000 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.77? Tell 10.0.80.1
- There're 20 ARP requests sent in a row in the same millisecond
b) Then I decided to check a few actions that the client would take after resolving the MAC address. I started with the first one (10.0.80.202)
Wireshark filter:
((arp.src.proto_ipv4==10.0.80.202 and arp.dst.proto_ipv4==10.0.80.1) or (arp.src.proto_ipv4==10.0.80.1 and arp.dst.proto_ipv4==10.0.80.202)) or (ip.addr==10.0.80.202 and ip.addr==10.0.80.1)
=> The client first resolves the MAC address of 10.0.80.202 about 2 minutes before the problem:
No. Time Source Destination Protocol Info
62994 2.518411 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.202? Tell 10.0.80.1
62996 0.000000 HewlettP_11:21:6b HewlettP_10:11:91 ARP 10.0.80.202 is at 00:23:7d:13:2f:6b
=> Then the client establishes a TCP session to TCP port 5357 and exchange some data with that system:
62997 0.000000 10.0.80.1 10.0.80.202 TCP 59327 > wsdapi [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=2
63000 0.000000 10.0.80.202 10.0.80.1 TCP wsdapi > 59327 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=8
63001 0.000000 10.0.80.1 10.0.80.202 TCP 59327 > wsdapi [ACK] Seq=1 Ack=1 Win=65700 Len=0
63008 0.000000 10.0.80.1 10.0.80.202 TCP 59327 > wsdapi [PSH, ACK] Seq=1 Ack=1 Win=65700 Len=223
63009 0.000000 10.0.80.1 10.0.80.202 TCP 59327 > wsdapi [PSH, ACK] Seq=224 Ack=1 Win=65700 Len=658
63013 0.000000 10.0.80.202 10.0.80.1 TCP wsdapi > 59327 [ACK] Seq=1 Ack=882 Win=65536 Len=0
63133 0.046800 10.0.80.202 10.0.80.1 TCP wsdapi > 59327 [ACK] Seq=1 Ack=882 Win=65536 Len=1460
63134 0.000000 10.0.80.202 10.0.80.1 TCP wsdapi > 59327 [PSH, ACK] Seq=1461 Ack=882 Win=65536 Len=908
63136 0.000000 10.0.80.1 10.0.80.202 TCP 59327 > wsdapi [ACK] Seq=882 Ack=2369 Win=65700 Len=0
=> Then the client goes into idle state in that TCP session for about 115 seconds. And then it resolves the MAC address of 10.0.80.202 once more (most likley the the ARP cache entry was deleted so source TCPIP stack needed to re-resolve the MAC address:
326437 115.073211 HewlettP_10:11:91 Broadcast ARP Who has 10.0.80.202? Tell 10.0.80.1
326483 0.000000 HewlettP_11:21:6b HewlettP_10:11:91 ARP 10.0.80.202 is at 00:23:7d:13:2f:6b
=> Right after resolving the MAC address of the target system once more, the client immediately forcefully closes the TCP session with a TCP RST segment:
326484 0.000000 10.0.80.1 10.0.80.202 TCP 59327 > wsdapi [RST, ACK] Seq=882 Ack=2369 Win=0 Len=0
c) After doing the same check for the other IP addresses that the client was trying to resolve the MAC addresses for, I saw that the behavior was the same (the client was connecting to another target at TCP port 5753 and was exchanging some data with that system and after staying idle for about 2 minutes, was re-resolving the MAC address and was forcefully closing the TCP session with a TCP RST.
SUMMARY:
=========
1) After analyzing the network trace as described above, it was more clear why the client was sending soo many ARP requests within the same milliseconds to various other targets:
To forcefully close its existing TCP sessions established to TCP port 5753 at those target machines.
2) The next question was that if that activity was legitimate or not. After doing some more research on the related protocol, I came up with the following results:
The activity on these TCP sessions (which were forcefully closed by the client) was generated by WSDAPI protocol. Also the data within the TCP segments confirmed that the activity was legitimate (seeing meaningful XML requests and responses in the traces as documented below)
https://msdn.microsoft.com/en-us/library/ee886312(VS.85).aspx Get (Metadata Exchange) HTTP Request and Message
https://msdn.microsoft.com/en-us/library/ee886311(VS.85).aspx GetResponse (Metadata Exchange) Message
This protocol seems to be used for discovering printers, Web cameras, video systems and manipulating them remotely. More details could be seen at the following articles:
https://msdn.microsoft.com/en-us/library/aa385800(VS.85).aspx About Web Services on Devices
3) Even though it wasn’t very clear why the client was closing many WSDAPI sessions at the same time, it was now clear that it was a legitimate activity and our customer wanted to take it further from that point on.
Hope this helps
Thanks,
Murat
Comments
- Anonymous
July 03, 2015
Mate thanks very much for sharing this. Similar issue occured within our environment and your post helped to trace back the root cause! - Anonymous
July 23, 2015
HP, what was your root cause? - Anonymous
August 21, 2015
I'm glad that it has helped. - Anonymous
December 09, 2015
thanks,