Symptoms

The main symptoms of the problem is difficulty in reaching certain machines or even networks or for those machines to gain access to other network segments and Internet access. The logs in the Firewall contains log entries about “Default_access_rule” when traffic is initiated from a machine that is experiencing the problem.

More information about the “Default_Access_Rule” log message can be found in this article : https://kb.clavister.com/324735778/the-meaning-of-the-default-access-rule-log-entry

Identifying the problem

The problem is mainly identified using the CLI and the CLI command “routes -all”. It is normal to have a 5, 10 or even 20+ Core related routes as there are scenarios where the Firewall is specifically configured with many Core routes, but the assumption here is that such a scenario is not in use for this example. This is how the output can look on an HA cluster that does not have the problem:

VSG-1:/> routes -all
Flags Network Iface Gateway Local IP Metric
------ ------------------ -------------- --------------- --------------- ------
192.168.10.1   core (Shared IP)  0
192.168.1.1    core (Shared IP)  0
192.168.201.11 core (Iface IP)   0
192.168.201.10 core (Shared IP)  0
192.168.200.11 core (Iface IP)   0
192.168.200.10 core (Shared IP)  0
192.168.98.11  core (Iface IP)   0
192.168.98.10  core (Shared IP)  0
127.0.0.1      core (Shared IP)  0

And this is how it can look on a cluster that has. Please note that this problem can also happen on standalone units, it is not unique to clusters.

VSG-1:/> routes -all
Flags Network Iface Gateway Local IP Metric
------ ------------------ -------------- --------------- --------------- ------
192.168.10.1   core (Shared IP) 0
192.168.1.1    core (Shared IP) 0
192.168.201.11 core (Iface IP) 0
192.168.201.10 core (Shared IP) 0
192.168.200.11 core (Iface IP) 0
192.168.200.10 core (Shared IP) 0
192.168.98.11  core (Iface IP) 0
192.168.98.10  core (Shared IP) 0
127.0.0.1      core (Shared IP) 0
192.168.98.1   core (Shared IP) 0
192.168.99.1   core (Shared IP) 0
192.168.100.1  core (Shared IP) 0
192.168.102.15 core (Shared IP) 0
192.168.103.16 core (Shared IP) 0
192.168.104.1  core (Shared IP) 0
192.168.115.10 core (Shared IP) 0

So we are seeing many IP addresses that are routed to the Core interface. Many IP addresses are also the lowest address in a network and that has a high chance of conflicting with, for example, a router or existing server. A limitation in the output from this command is that it is not possible to see the originator of the Core route, which sub-system that created it.

There is also a rather odd behavior where we can have these Core routes in all routing tables except the <main> routing table. More on that topic below.

Core route sources

The problem can be difficult to identify as it is not always obvious that this is the problem in a network. It can be a problem that has been on-going for a long time as well and various workarounds has been implemented to try bypass the problem in some way.

Core routes can be added from multiple sources, to list a few:

• Interface IP addresses (Ethernet, VLAN etc.)
• Manually added route to the Core interface
• IPsec tunnels
• NAT Pools

There are also other systems that can setup a dynamic route but it is at this time unconfirmed if these could generate Core routes as well. But we list them anyway as it can be good to know.

• L2TP/PPTP/SSL-VPN/OneConnect server
• DHCP Relay
• OSPF (pretty much the main purpose of OSPF)

Of the ones listed above the most common source of the Core routes is IPsec interface(s). The reason for this is due to how IPsec interfaces has a requirement that they MUST own an IP address. The IPsec interfaces needs to own an IP address in order for it to know which source IP it should use if we for instance use NAT or if something is initiated from the Core itself into the tunnel. An example of this would be if we want to send logs generated by the Firewall itself through an IPsec interface, then the Firewall needs to know which source IP it should use.

Normally the Firewall uses the source IP of an already existing Core IP that matches the network set as “Local network” on the IPsec tunnel (an example of such a Core route would be if a Core routed is added by an Ethernet or VLAN interface based on the IP addresses assigned to it, IP address is a required field).

But there are mainly two scenarios where this causes problems.

1. If we define "all-nets" as the local network, the Firewall has no idea which source to use and will use 127.0.0.01 as the sender.
2. If we use a local network that does not match any of the existing Core routes in the Firewall the Firewall will Core route the first IP address that belongs to the local network.

So if we imagine that we have a scenario where an administrator has configured an IPsec tunnel with .e.g a group of 20 different networks, ranges and IP’s and use that as the local network on an IPsec tunnel, that would generate a vast number of Core routes that has a high chance of causing conflicts and problems.

IPsec_Tunnel_A
    \-Local Network
     \-192.168.99.0/24
     \-192.168.100.0/24
     \-192.168.100.15-192.168.200
     \-192.168.104.0/24
     \-192.168.115.10

And if the local network of the Firewall is not part of any of the above local network examples, we would have a lot of automatically generated Core routes.

To make things even trickier, there is an old bug/limitation that causes a rather unusual behavior (COP-15742). The description of the problem/limitation is as follows:

When defining a local network on an IPsec tunnel that is not part of a local interface (so that a Core route already exists), the IPsec interface will generate a Core route in all routing tables except the <main> routing table.

Solution

Luckily, the solution to this problem if it originates from the IPsec tunnels is rather easy. On each IPsec tunnel under the Advanced tab there is an option to configure the IP address assigned to the IPsec tunnel manually as show below.

This will create an override and stops the IPsec tunnels from creating the Core routes automatically. A few things to keep in mind when configuring this manually.

1. Only one IP address can be used. Meaning it is not possible to configure multiple IP address to match multiple local networks on the tunnel.
   1. This however can be solved in many scenarios by manually specifying the IP sender on the IP policy NAT rule that allows traffic into the IPsec tunnel interface.
2. In case of a cluster, the HA IP address is optional and localhost can in many scenarios be used. The need to have valid IP addresses for the HA IP address is if there is a need to send data into the tunnel from the Firewall itself (such as logs or if you simply send a ping from the CLI into the tunnel).
3. The IP addresses specified under this option will be Core routed if they are not so already.

By manually telling the IPsec tunnel interface(s) which IP to use, there will be no need for the Firewall to create multiple Core routes that could cause conflicts. We just need to make sure to avoid using an IP address that conflicts with some existing equipment or server/client.

At this time (2021-03-17), the main source of this particular problem scenario has only originated from IPsec interfaces. But being aware of other systems that may add routes automatically will hopefully help in narrowing down the route origin.

Related articles