Introduction

cOS Core Intrusion Detection and Prevention (IDP) is a feature that can detect and eliminate cyber attacks against network clients and hosts situated behind a NetWall firewall, in particular against protected servers. It is a critical component of any network security strategy, as it helps to identify and mitigate many potentially damaging security threats.

IDP can be used in a variety of ways, depending on the specific needs of an organization. For example, it can be used to monitor network traffic for suspicious activity, such as attempts to exploit vulnerabilities in software or hardware, or to detect attempts to gain unauthorized access to a network. It can also be used to block malicious traffic, such as worms, trojans or backdoor exploits.

IDP functions by monitoring network traffic as it passes through the Clavister firewall, searching for patterns that indicate an intrusion is being attempted. Once detected, IDP can take steps to neutralize both the intrusion attempt as well as its source.

Best Practice

Configuring IDP can be quite difficult as each installation is unique. What works well for Site-A may not work well at all for Site-B due to different traffic patterns, network equipment, servers, clients, and so on. Until we get to specific use cases, some generic recommendations will be discussed which reiterate much of the information found in the cOS Core Administration Guide.

IDP Deployment Questions

In order to have an effective and reliable IDP system, the following questions should be considered by the administrator:

• Which traffic should be analyzed using IDP? This will decide the source/destination network/interface and service combination that will need to be specified in an IDP Rule.
• What should we search for in that traffic?
  • This will determine which signatures, sets of signatures or signature filters will need to be specified in the IDP Rule Action objects added to an IDP Rule. Are all signatures in a signature group or subgroup required? Should some specific IDP signatures be excluded to improve throughput or hit-rate? Perhaps only a small number of individual IDP signatures are needed to defend against a small number of known current threats.
  • Selecting as few IDP signatures as possible is important from a system performance viewpoint. Using an excessive number of IDP signatures will most likely impact firewall throughput.
• What kind of devices are we trying to protect? Using, for example, signatures related to exploits in IIS is not very useful if the web server is using Apache and there would be an increased risk of false positives.
• What action should be taken when IDP triggers?
  • Each IDP Rule Action has an Action property that specifies what to do if a signature triggers. Should the connection be dropped? Should it be allowed but logged?
  • In addition, should the source IP of the triggering traffic be blacklisted so future traffic from that IP is dropped? How long should the blacklisting last? Should only the triggering protocol be blacklisted? Should existing connections from a blacklisted IP be dropped?
• Should a scan limit be set so that scanning does not slow down overall system performance? Depending on the traffic mix, it may be sufficient to only scan e.g. the first 800 bytes of each new connection's traffic flow.

IDP rule processing

The following is a list of IDP setup steps and subsequent IDP processing:

• An IDP Rule is configured by the administrator which specifies the targeted traffic using a combination of source/destination network/interface and service. This can optionally include a scan limit to improve performance so that only the initial portion of a connection data flow is examined.
• Each rule has one or more IDP Rule Action objects added to it as children and these specify the IDP signatures to apply to the targeted traffic and what action to take if a signature triggers.
• Matching of the signatures to the targeted traffic is then applied by cOS Core as it streams through the firewall.
• If any of the configured IDP signatures trigger, the action specified for those signatures is applied. If a signature is repeated across multiple IDP Rule Action objects, only the action of the first triggering IDP Rule Action will be applied for that signature.

For more information about specific settings and more, please see the IDP section in the cOS Core Administration Guide : https://docs.clavister.com/repo/cos-core-administration-guide/

Some Use Cases

Is is important to note that signature groups selected for IDP may not remain valid over time. As the IDP signature database updates and changes, new threats are identified and new signatures are created. This means that the signature groups that were once effective may become obsolete and new signature groups or signature filters may need to be selected or updated.

For example, a signature group (or signature filter) that was effective in detecting and preventing a certain type of attack on a web server may not be effective against a new variant of the same attack. Similarly, a signature group that was effective in detecting and preventing attacks on a file server may become outdated as new threats are identified.

Therefore, it is crucial to keep the IDP signature database up to date and periodically review and update our signature groups and/or signature filters to ensure that our firewall is providing the best possible protection for our network. Regularly reviewing and updating the signature groups and filters can help stay ahead of new and emerging threats, and ensure that networks are secure against the latest attack techniques.

In the various use cases, we have some options on how we can/should configure our IDP rules. The configuration consists of mainly two parts:

1. Setup IDP rules for which traffic to scan and in what direction (traffic flow direction).
2. Determine which signature groups or signature filters that should be used. Note: We will not go into the level of individual signatures as the IDP signature databases changes frequently with signatures added/removed sometimes several times a week.

We will not go into details on how the firewalls routing or IP policies are configured. Instead, we will focus on the IDP rules only, with more details provided only if needed in each use case.

Use case 1 - Web server behind the firewall

In this use case, we will explore the use of IDP in a network environment were we have an Apache web server located behind the firewall’s DMZ interface (this interface and others may have different actual Ethernet interface names, depending on the platform running cOS Core).

Looking at the above schematic, the main traffic would be initiated from the Internet towards our web server. There would be some outgoing connections. For example, if the web server needs to download automatic updates. For this scenario we will only focus on the incoming traffic.

For scanning traffic initiated from the Internet to the web server located behind the DMZ, the IDP rule would then look like the one below:

Since the Web Server is located behind the DMZ interface on a private IP address, the firewall performs SAT address translation for incoming connections arriving at the Wan interface IP (192.0.2.2) and traffic will then be forwarded to the Web Server behind the DMZ (172.16.0.10). Therefore the IDP rule needs to be configured to trigger on how the firewall receives the traffic before any address translation is applied by cOS Core. Which services/port to use can of course vary depending on what is installed on the server and what it should respond to.

Signature group selection

Selecting which signature groups to use can vary between installations, but below is an example of which signature filters and groups we can use to protect a web server.

IDS_*, POLICY_*

*cve-202*

*apache*

*wordpress*

The above example would (at the time of writing this) result in an estimated 1000 IPS signatures. Here are some details about what the above setup does:

• Rule 1 will ignore any signatures in the IDS and POLICY signature groups. IDS signatures are used mainly for logging. POLICY signatures are a way to detect applications. However, since the introduction of Application Control into cOS Core (which has its own engine and much larger database), POLICY signatures have become redundant and should therefore not be used. It is important to place this rule at the top, as the following rules are filters that query specific words to find matching signatures.
• Rule 2 is including any signature released in the year 2020 and above that has a CVE ID ( Common Vulnerabilities and Exposures ). It is a useful way to exclude older types of signatures and only have signatures with a CVE ID in its name starting year 2020.
• Rule 3 and 4 are wildcard search filters that query the IDP database and include any IPS signature or group which contains either the word Wordpress or Apache. This is of course an example, depending on the applications, services etc. used on the destination server, other queries or additional queries may be appropriate.

Use case 2 - Users behind the firewall

In this use case, we will explore the use of IDP in a network environment were we have two interfaces assigned to two different network ranges and users.

Looking at the above schematic we have a fairly straight forward scenario where network connections are only initiated from the Lan and WIFI interface, there would be no incoming connections (connections initiated from something on the internet) towards either of the Lan or WIFI interfaces and networks. For this scenario two IDP rules would be sufficient.

1. Scan traffic initiated from the Lan interface and local users in the Lannet to the Internet.
2. Scan traffic initiated from the WIFI interface and users connected to the AP and the WIFI-net.

The IDP rules would then look like this based on the above requirement.

The services used will be up to the administrator to determine. In the above example we allowed unrestricted access to outgoing ports and protocols on our IP policy while the WIFI network is more restricted where we only allow HTTP, HTTPS and DNS.

Note: “all_services” should probably be a narrower set of services but it is useful for this simplified example.

Signature group selection

Selecting which signatures to use can vary between installations, but if we assume that in this scenario we have a mix of PCs, phones, tablets, Windows, MAC etc. the signatures we want to use can be the same on both IDP rules. However,we choose to be more restricted for the WIFI interface as there would be a lot less services allowed and there would be no reason to apply signature filters and groups we know would never trigger since this will yield better performance and less chance for false positives.

IDS_*, POLICY_*

*cve-202*

IPS_APP_GENERIC,ips_trojan*,ips_file*,ips_db_*,ips_game*,ips_generic*,ips_http*,ips_OS-specific*, ips_malware*

*dns*,*java*,*bot*,*browser*,*infection*,*ransom*,*ftp*,*pop3*,*imap*,*smtp*,*ssl*,*telnet*,*ssh*,*rdp*,*vnc*,*icmp*,*ping*

IDS_*, POLICY_*

*cve-202*

IPS_APP_GENERIC,ips_trojan*,ips_file*,ips_db_*,ips_game*,ips_generic*,ips_http*,ips_OS-specific*, ips_malware*

*dns*,*java*,*bot*,*browser*,*infection*,*ransom*,*ssl*

Estimated signature count for the Outgoing_Lan rule is about 7000 and for the Outgoing_WIFI is 6000. This would not result in a total of 13000 signatures. See Q&A section further down for more information about this point.

Details about the Outgoing_Lan rule:

• The first Ignore rule will ignore any signatures in the IDS and POLICY signature groups. This was already discussed in the first use case.
• Rule 2 is including any signature released in the year 2020 and above that has a CVE ID ( Common Vulnerabilities and Exposures ). This was already discussed in the first use case.
• Rule 3 is a selection of group(s) that would be appropriate based on what we either expect or even suspects could generate traffic in this network segment.
• Rule 4 Contains various wildcard search filter that queries the IDP database and includes any IPS signature or group which contains the selected word such as "*dns*". Which words to use depends on what we expect/know would be generating traffic in the Lan network. This should be considered an example on how it can be used and should always be customized based on the customer environment. Note that Rule #3 and #4 can be combined but we choose to configure them here in two separate rules to make it easier to read. Performance/configuration wise there would be no difference.

Details about the Outgoing_WIFI rule:

• The first Ignore rule will ignore any signatures in the IDS and Policy signature groups. This was already discussed in the first use case.
• Rule 2 is including any signature released in the year 2020 and above that has a CVE ID ( Common Vulnerabilities and Exposures ). This was already discussed in the first use case.
• Rule 3 is a selection of group(s) that would be appropriate based on what we either expect or even suspects could generate traffic in this network segment.
• Rule 4 Contains various wildcard search filter that queries the IDP database and includes any IPS signature or group which contains the selected word such as "*dns*". Which words to use depends on what we expect/know would be generating traffic in the WIFI network. This should be considered an example on how it can be used and should always be customized based on the customer environment. In the example we have selected fewer words as we do not expect e.g. FTP traffic from the WIFI network as we only allow HTTP, HTTPS and DNS ports/protocols. Which would result in fewer signatures and is a way to increase the performance.
  
  • Rule #3 and #4 can be combined, we choose to configure them in two separate rules to make it easier to read. Performance/configuration wise there would be no difference.

Boosting performance

IDP can be quite resource demanding depending on how it is configured, the amount of traffic it is enabled on, the amount of signatures used, traffic pattern and more. In case the CPU load of the firewall becomes high after the activation there are some options that may be considered in order to boost the performance of IDP.

1. Further refine the amount of signatures, signature groups and signature filters used. The more signature used, the more the IDP engine needs to go through and scan traffic. If the amount of signatures that the administrator wants to use is not possible to achieve on the hardware platform, the groups/filters used may need to be restricted more in order to avoid problems. There is no direct recommendation here on what can be removed/changed. An alternative would be to look at upgrading hardware or split the network traffic between multiple firewalls.
2. Turn off "Protect against insertion/evasion attacks" option on the IDP rule(s).
   
   1. A description of what this option does: "An insertion/evasion attack is a form of attack which is specifically aimed at evading IDP mechanisms. It exploits the fact that in a TCP/IP data transfer, the data stream must often be reassembled from smaller pieces of data because the individual pieces either arrive in the wrong order or are fragmented in some way. Insertions or evasions are designed to exploit this reassembly process. For more information, see the cOS  Core Administration Guide: https://docs.clavister.com/repo/cos-core-administration-guide/
   2. There are two motivations for disabling this option:
      1. Increasing throughput - Where the highest throughout possible is desirable, turning the option off can provide a slight increase in processing speed.
      2. Excessive false positives - If there is evidence of an unusually high level of insertion/evasion false positives then disabling the option may be prudent while the false positive causes are investigated.
3. Setting a scan limit. This determines how many initial bytes of the connection traffic will be scanned to look for a signature match before scanning stops. This can have the advantage of significantly improving IDP performance but has the disadvantage of potentially missing threats which occur much later in a connection's data flow. The default scan limit of 800 bytes has been determined to be a good mix between signature hit-rate and performance.
4. Use dynamic blacklisting. By default this option is not enabled, but it is recommended to always enable this option as once an attack/signature has triggered and blocked the traffic, it is reasonable to continue to block the source IP address of the "bad" host. Otherwise the IDP scanning has to be done multiple times for each connection/data stream, making it inefficient and has a risk of letting through traffic that should be blocked, not to mention it would require more CPU resources to repeat the scan again instead of just flat-out drop the incoming request if the host is already blacklisted.

Questions and Answers (Q&A)

Question: When should IDS type signature group(s) be used?

Answer: In this guide we have specifically excluded the IDS signature group, the main reason for that is IDS is mainly meant to be used for audit/scanning purposes. IDS signatures has a higher chance of triggering a false positive but it can still be useful for statistical and logging purposes if used in combination with e.g. Clavister InCenter.

A second reason is that IDS signatures are considered a legacy type signature group which is no longer actively being created signatures for by the signature vendor. In the past the signature vendor created both IDS and IPS signatures for many exploits where the signature that had a higher chance of triggering a false positive was placed in the IDS main group.

Question: What is a false positive?

Answer: A false positive refers to a situation where the system incorrectly identifies a benign activity as a malicious one. An IDP system is designed to monitor network traffic, analyze it, and detect any suspicious behavior that could indicate an attempted intrusion or attack. False positives can occur when the system mistakenly flags legitimate traffic as malicious. This can happen due to a variety of reasons, such as an incorrectly configured rule, a bug in the system, or unusual but legitimate network behavior.

False positives can have serious consequences, as they can lead to unnecessary alerts, and may even cause the system to block legitimate traffic, which can disrupt business operations. Therefore, it is important for IDP system administrators to regularly review and fine-tune the system’s rules and parameters to minimize the occurrence of false positives and ensure the system is accurately detecting and preventing actual threats.

Question: Is there an easy way to see how many signatures we are using in our current configuration?

Answer: The CLI can be used to get a summary of how many signatures in total that are used. An example of the command + how the output can look:

VSG-14:/> rules -type=IDP -verbose
Contents of intrusion detection ruleset; default action is IGNORE

        
# Name Action Log Usage
Details
----- ----------------------------------------------------- ------- --- --------
1 Outgoing_WIFI Filter Yes 0
SRC: WIFI:192.168.2.0/24
DST: Wan:0.0.0.0/0
Service: all_services
Severity: ALL; Signature(s): IDS_*, POLICY_* Ignore Yes
Severity: ALL; Signature(s): *cve-202* Protect Yes
Severity: ALL; Signature(s): IPS_APP_GENERIC, Protect Yes
ips_trojan*,ips_file*,ips_db_*,ips_game*,
ips_generic*,ips_http*,ips_OS-specific*,
ips_malware*
Severity: ALL; Signature(s): *dns*,*java*,*bot*, Protect Yes
*browser*,*infection*,*ssl*
TCP Signatures: 5663
UDP Signatures: 773
ICMP Signatures: 0
Other Signatures: 0

Question: When is IDP applied to the traffic? Before or after an IP rule/policy lookup?

Answer: IDP is applied to traffic that is first allowed to flow by the firewall’s IP rule set, so it would be after as there would be no reason to scan traffic that would be dropped by the IP rule set to begin with.

Question: Is IDP scanning whole files or does it scan network traffic as the firewall receives it?

Answer: If the traffic is allowed by the IP rule set, IDP would be performed on any data stream being generated. The exception would be when a scan limit has been set as then it would scan e.g. the 800 first bytes and if no match has been found it stops scanning this particular connection/data stream.

Question: Should a scan limit be set?

Answer: Yes, it is recommended to set a scan limit as it can boost the performance of IDP processing significantly. It will however cause lower hit-rates, it will be up to the administrator to determine what would be appropriate.

Question: What is the difference between the 3 main signature groups, IDP, IPS and Policy?

Answer: A short description of each of the major signature groups:

• IDS - Intrusion Detection Signatures (IDS) can detect events that may be intrusions. They have lower accuracy than IPS and may give some false positives so it is recommended that the Audit action is always used. Using them with Protect may interrupt normal traffic.
• IPS - Intrusion Protection Signatures (IPS) are highly accurate and a match is almost certainly an indicator of a threat. Using the Protect action is recommended. These signatures can detect administrative actions and security scanners.
• Policy - Policy Signatures detect different types of application traffic. They can be used to block certain applications such as file sharing applications and instant messaging. Note: It is recommended to use Application Control (AC) instead of IDP policies as the introduction of AC made the policy signatures redundant since AC has superior capabilities to detect applications.

Question: Can/should the IDS or Policy signature group(s) be used in protect mode?

Answer: No, see the answer on the previous question for details and reason why it should not be used.

Question: If I use 5000 signatures in IDP rule A and the same signatures in IDP Rule B, would that then mean I’m using a total of 10000 signatures from the firewall’s perspective?

Answer: No, in the this example the total signature used would only be 5000. It is the number of unique signatures that matters.

Question: Can IDP be combined with IP reputation?

Answer: Yes and it is strongly recommended to do so as there are several signatures in the IDP database that are related to scanners and botnets. These can be complemented or in some cases even replaced by IP reputation. To further optimize the performance of IDP, using an ignore rule on “bot” and “scanner**” would be a possibility.

Question: What about other combinations such as Anti-Virus, Web Content Filtering and more. Is there any combination that would be “bad” or not recommended?

Answer: Generally it should be no problem to use IDP together with other features such as Anti-Virus. It would be up to the administrator to determine what would be appropriate to use. Please note, however, that using older IP rules and ALG’s may not always work well together with IDP, this recommendation is when only IP policies are used.

Question: Some of the signatures in the database seems to be very old (from e.g. 2005), should these really be used? Can we exclude them somehow?

Answer: In use case 1 and use case 2 we use a filter for CVE ID. Older signatures does not have this CVE ID and 90% if not more of all new signatures made today have a CVE ID, this would be a good way to only include only newer types of signatures. The older signatures can still be useful if we want to detect malicious intent if someone is e.g. doing a scan and attempts to flood an environment with random known exploits, if IDP then detects this and blacklists the host(s) the administrator would then have logs available that can provide him with alerts if this is the case (assuming alerts or other indicator is configured such as InCenter or InControl Log Analyzer data cube.

Question: Should IDP really be used on clients initiating traffic behind the firewall (from e.g. the Lan interface), do we want to protect the internet from our users?

Answer: There are mainly two situations where this could be useful.

1. If a client initiates a connection to a server on the internet and downloads/receives malicious content.
2. If a client is already infected and attempts to connect to a device that cause a signature to trigger. If this situation happens and blacklisting is used, the client user would hopefully contact the system administrator if their Internet stops working. It can then be determined, based on the logs from the firewall, that IDP triggered because the user's computer was infected with something.

Question: How many signatures can be used in total? Is there a limit?

Answer: Below are some approximate guidelines for how the total available memory relates to the maximum signatures selected:

• Up to 500 MBytes system memory: Approximately 15,000 signatures
• 500 MBytes to 1 GByte system memory: Approximately 22,000 signatures
• Over 1 GByte system memory: Approximately 30,000 signatures

Please note however that if using more 15-20 000+ signatures, there is a fairly good chance that many of the selected signature groups are not needed. A review of the used groups and filters is recommended to avoid false positives and also high CPU load situations.

Question: Is it possible to exclude a specific signature from scanning? A new traffic pattern caused signature XXX to block legit traffic but we do not want to remove the entire group the signature belongs to.

Answer: Similar to the “Ignore” used in the use case examples, we can also use this to ignore a specific signature ID. If we take the signature selection example from use case 1 and then exclude signature with ID 12345 it would look like this:

IDS_*, POLICY_*

12345

*cve-202*

*apache*

*wordpress*

Simply input the exact ID of the signature with the action set to Ignore and place it above the Protect action rules. The order is important here as we want to make sure the specific signature is ignored before it is included in any protect actions.

Question: Where can we find more information about individual IDP signatures or groups?

Answer: More details about each signature can be queried on the IDP advisory page located here : https://www.clavister.com/advisories/idp/

Question: What about Ransomware? Anything special to consider in this area?

Answer: Ransomware is in pretty much all cases a type of Trojan and many signatures for Trojans are included in the “IPS_Trojan” and “ransom” filters used in Use case 2. This however is no guarantee that it can offer full protection against such attacks. The following is a list of security functions/features that would help stop ransomware attacks.

1. IDP and the IDP filters mentioned above.
2. IP reputation (Botnet)
3. WCF (Web Content Filtering) and add at least Botnets, Hacking, Keyloggers, Malicious, Spyware and Suspicious to the restricted list.
4. Activate and use anti-virus when applicable.
5. Make sure that a local anti-virus and firewall is installed and is run on all devices and servers as cOS Core should never be a replacement for locally installed protection systems as cOS Core only looks at network datastreams, not local files, memory etc.

Related articles