Server cluster generates UnsolicitedARPReplies in log (Gratuitous/GARP)

This article applies to:Clavister Security Gateway 8.x, 9.x and 10.x

Note changes as of 8.30.01 for Linux Heartbeat clusters and Microsoft Cluster Server

Note changes as of 8.50.02 for Microsoft Network Load Balancing

Background

Some high-availability server clusters work by sharing an IP address, where only one server “owns” this address at a time. If the server currently owning the IP address goes down, the backup server will start serving requests on that IP address.

A basic problem here is getting surrounding network equipment (routers, Security Gateways) to understand that the IP address should be routed a new hardware address. This is often done through gratitious ARP responses, whereby the new server sends responses to all hosts that need to know about the change, even though these units have not sent ARP queries.

The problem

Clavister Secturity Gateway will not listen to ARP responses that it has not sent out queries for. This is to make “ARP spoofing” (IP address spoofing on the local network through bogus ARPs) harder.

This means that the ARP cache of the Security Gateway will not be updated by the gratuitous ARPs sent by the new server. Rather, it will take up to 15 minutes (using default settings) for the Security Gateway to start routing traffic to the new server.

You will likely also see entries like these in your logs (syslog example):

Jun 6 11:38:13 mygw EFW: ARP: rule=UnsolicitedARPReplies 
action=drop reason=already_exists recvif=int 
hwsender=0090:0b02:5c70 hwdest=0001:020d:fb16 arp=reply 
srcenet=0090:0b02:5c70 srcip=192.168.16.136 
destenet=0001:020d:fb16 destip=192.168.16.1

The solution

Clavister Secturity Gateway can be made to accept gratuitous ARPs by turning off the extra checks and making it fully compliant with the ARP specification: RFC 826.

Make the following modifications to Advanced Settings -> ARP:
Note: In newer cOS Core versions (10+) the ARP settings can be found under Network->ARP->Advanced Settings.

• UnsolicitedARPReplies: Accept or AcceptLog (default: DropLog)
  This is necessary if the server cluster transmits gratitious requests.
• ARPRequests: Accept (default: Drop)
  This may be necessary if the server cluster transmits queries rather than requests.
• ARPChanges: Accept or AcceptLog (default: AcceptLog)
  This is necessary, since the hardware address will change when failover occurs.
• ARPMatchEnetSender: Ignore or Log (default: DropLog)
  Some server cluster implementations build somewhat ugly ARP packets, where the sender address in the MAC header does not match the address supplied in the ARP data. If that is the case, this setting cannot be Drop or DropLog.
• ARPMulticast: Accept, Log, Drop or DropLog (default: DropLog)
  Some cluster implementations like Windows 2003 servers with IIS requires that this option be set to Accept or Log.

Note that Clavister Security Gateway before 8.30.01 only listens to ARP packets destined to the Security Gateway itself. This presents a problem for the Linux Heartbeat cluster system and MS Cluster Server, which does not send targeted ARP queries when it attempts to update the ARP caches of nearby units.

As of v8.30.01, the Security Gateway will listen to any ARP queries if ARPRequests is set to Accept, and any ARP responses if UnsolicitedARPReplies is set to Accept or AcceptLog.

Note that Clavister Security Gateway before 8.50.02 always sent ARP responses to the MAC address found in the ethernet header of the query. Microsoft NLB apparently does rely on the response to be sent to the source MAC address in the ARP data.

As of v8.50.02 the core sends ARP responses to the source MAC address in the ARP data.

Related articles