Problem with auto-created Core routes

Last modified on 22 Mar, 2021. Revision 11
When checking the status of the various routing tables there are many routes added towards the Core interface that is causing problems
Up to date for
cOS Core 13.00.09
Supported since
cOS Core 9.xx and up
Not valid for
cOS Core 8.xx
Status OK
Author
Peter Nilsson


Symptoms

The main symptoms of the problem is difficulty in reaching certain machines or even networks or for those machines to gain access to other network segments and Internet access. The logs in the Firewall contains log entries about “Default_access_rule” when traffic is initiated from a machine that is experiencing the problem.

More information about the “Default_Access_Rule” log message can be found in this article : https://kb.clavister.com/324735778/the-meaning-of-the-default-access-rule-log-entry

Identifying the problem

The problem is mainly identified using the CLI and the CLI command “routes -all”. It is normal to have a 5, 10 or even 20+ Core related routes as there are scenarios where the Firewall is specifically configured with many Core routes, but the assumption here is that such a scenario is not in use for this example. This is how the output can look on an HA cluster that does not have the problem:

VSG-1:/> routes -all
Flags Network Iface Gateway Local IP Metric
------ ------------------ -------------- --------------- --------------- ------
192.168.10.1 core (Shared IP) 0
192.168.1.1 core (Shared IP) 0
192.168.201.11 core (Iface IP) 0
192.168.201.10 core (Shared IP) 0
192.168.200.11 core (Iface IP) 0
192.168.200.10 core (Shared IP) 0
192.168.98.11 core (Iface IP) 0
192.168.98.10 core (Shared IP) 0
127.0.0.1 core (Shared IP) 0

And this is how it can look on a cluster that has. Please note that this problem can also happen on standalone units, it is not unique to clusters.

VSG-1:/> routes -all
Flags Network Iface Gateway Local IP Metric
------ ------------------ -------------- --------------- --------------- ------
192.168.10.1 core (Shared IP) 0
192.168.1.1 core (Shared IP) 0
192.168.201.11 core (Iface IP) 0
192.168.201.10 core (Shared IP) 0
192.168.200.11 core (Iface IP) 0
192.168.200.10 core (Shared IP) 0
192.168.98.11 core (Iface IP) 0
192.168.98.10 core (Shared IP) 0
127.0.0.1 core (Shared IP) 0
192.168.98.1 core (Shared IP) 0
192.168.99.1 core (Shared IP) 0
192.168.100.1 core (Shared IP) 0
192.168.102.15 core (Shared IP) 0
192.168.103.16 core (Shared IP) 0
192.168.104.1 core (Shared IP) 0
192.168.115.10 core (Shared IP) 0

So we are seeing many IP addresses that are routed to the Core interface. Many IP addresses are also the lowest address in a network and that has a high chance of conflicting with, for example, a router or existing server. A limitation in the output from this command is that it is not possible to see the originator of the Core route, which sub-system that created it.

There is also a rather odd behavior where we can have these Core routes in all routing tables except the <main> routing table. More on that topic below.

Core route sources

The problem can be difficult to identify as it is not always obvious that this is the problem in a network. It can be a problem that has been on-going for a long time as well and various workarounds has been implemented to try bypass the problem in some way.

Core routes can be added from multiple sources, to list a few:

  • Interface IP addresses (Ethernet, VLAN etc.)
  • Manually added route to the Core interface
  • IPsec tunnels
  • NAT Pools

There are also other systems that can setup a dynamic route but it is at this time unconfirmed if these could generate Core routes as well. But we list them anyway as it can be good to know.

  • L2TP/PPTP/SSL-VPN/OneConnect server
  • DHCP Relay
  • OSPF (pretty much the main purpose of OSPF)

Of the ones listed above the most common source of the Core routes is IPsec interface(s). The reason for this is due to how IPsec interfaces has a requirement that they MUST own an IP address. The IPsec interfaces needs to own an IP address in order for it to know which source IP it should use if we for instance use NAT or if something is initiated from the Core itself into the tunnel. An example of this would be if we want to send logs generated by the Firewall itself through an IPsec interface, then the Firewall needs to know which source IP it should use.

Normally the Firewall uses the source IP of an already existing Core IP that matches the network set as “Local network” on the IPsec tunnel (an example of such a Core route would be if a Core routed is added by an Ethernet or VLAN interface based on the IP addresses assigned to it, IP address is a required field).

But there are mainly two scenarios where this causes problems.

  1. If we define "all-nets" as the local network, the Firewall has no idea which source to use and will use 127.0.0.01 as the sender.
  2. If we use a local network that does not match any of the existing Core routes in the Firewall the Firewall will Core route the first IP address that belongs to the local network.

So if we imagine that we have a scenario where an administrator has configured an IPsec tunnel with .e.g a group of 20 different networks, ranges and IP’s and use that as the local network on an IPsec tunnel, that would generate a vast number of Core routes that has a high chance of causing conflicts and problems.

IPsec_Tunnel_A
\-Local Network
\-192.168.99.0/24
\-192.168.100.0/24
\-192.168.100.15-192.168.200
\-192.168.104.0/24
\-192.168.115.10

And if the local network of the Firewall is not part of any of the above local network examples, we would have a lot of automatically generated Core routes.

To make things even trickier, there is an old bug/limitation that causes a rather unusual behavior (COP-15742). The description of the problem/limitation is as follows:

When defining a local network on an IPsec tunnel that is not part of a local interface (so that a Core route already exists), the IPsec interface will generate a Core route in all routing tables except the <main> routing table.

Solution

Luckily, the solution to this problem if it originates from the IPsec tunnels is rather easy. On each IPsec tunnel under the Advanced tab there is an option to configure the IP address assigned to the IPsec tunnel manually as show below.

This will create an override and stops the IPsec tunnels from creating the Core routes automatically. A few things to keep in mind when configuring this manually.

  1. Only one IP address can be used. Meaning it is not possible to configure multiple IP address to match multiple local networks on the tunnel.
    1. This however can be solved in many scenarios by manually specifying the IP sender on the IP policy NAT rule that allows traffic into the IPsec tunnel interface.
  2. In case of a cluster, the HA IP address is optional and localhost can in many scenarios be used. The need to have valid IP addresses for the HA IP address is if there is a need to send data into the tunnel from the Firewall itself (such as logs or if you simply send a ping from the CLI into the tunnel).
  3. The IP addresses specified under this option will be Core routed if they are not so already.

By manually telling the IPsec tunnel interface(s) which IP to use, there will be no need for the Firewall to create multiple Core routes that could cause conflicts. We just need to make sure to avoid using an IP address that conflicts with some existing equipment or server/client.

At this time (2021-03-17), the main source of this particular problem scenario has only originated from IPsec interfaces. But being aware of other systems that may add routes automatically will hopefully help in narrowing down the route origin.

Related articles

Configuring L2TP/IPsec Server using PSK
11 Jan, 2023 ipsec core vpn
Setup of a Layer-3 bridge over IPsec in cOS Core
12 Apr, 2023 core proxyarp arp ipsec routing
cOS Core IKEv2 split tunneling with Windows and local user database.
28 Mar, 2023 ikev2 windows vpn routing splittunneling
Configuring public certificates in NetWall firewalls
18 Mar, 2024 core certificate oneconnect ipsec vpn
cOS Core L2TP server setup with Windows Server CA certificates
21 Feb, 2023 ipsec certificate windows ca core
Certificate update in InControl global domain on certificate that is used on firewall(s)
18 Mar, 2024 core incontrol certificate oneconnect ipsec vpn
Setting up OSPF with IPsec in cOS Core
16 Apr, 2024 core routing ospf ipsec
Using /31 network masks in cOS Core (RFC-3021)
1 Jun, 2022 core routing management
cOS Core IPsec IKEv1 "No_Proposal_Chosen" error in 14.00.10
4 Aug, 2023 core ipsec troubleshoot ike
IPsec license usage calculation
14 Apr, 2021 core license ipsec
Does IPsecBeforeRules trigger before Access rules?
8 Sep, 2020 core ipsec rules access
Split tunneling in cOS Core with Windows L2TP/IPsec clients
29 Mar, 2023 ipsec core windows vpn l2tp
Troubleshooting IPsec tunnels (IKEv1)
7 Dec, 2022 ipsec ike troubleshoot core
cOS Core IKEv2 tunnel setup with certificates for iOS clients
5 Apr, 2023 core nps ipsec radius legacy
The meaning of the Default_Access_Rule log entry
7 Nov, 2022 core arp log routing
Troubleshooting cOS Core rules/routes with ping simulation
17 Mar, 2023 core routing rules ping icmp cli
Freeing up more memory in the Firewall
23 Aug, 2022 core connections ipsec memory
Is Statless (FwdFast) faster than a normal IP policy?
27 Jan, 2021 core stateless routing brokenlink
Route failover with IPsec tunnels in cOS Core
13 Feb, 2023 ipsec core routing failover
Public network transparency using cOS Core Proxy ARP instead of subnetting
18 Apr, 2023 core routing transparentmode proxyarp