Overview

InCenter v3 On-prem service is a log analytics tool that provides a set of network security and user authentication dashboards for Clavister NetWall and NetShield Firewalls, and IdAuth services.

It runs as a Kubernetes deployment under target customer’s environment.

Components

InCenter v3 On-prem consists of following micro-services:

InCenter micro-services

External services dependencies 

SMTP server

In order for the system to be able to send information by email, for instance for alerts or reports, an SMTP server can be specified. See the SMTP section in the Configuration manual for setting this up.

Install

Prerequisites

• Helm
• kubectl
• Ingress controller installed in Kubernetes cluster.

Steps

1. Create a values.yaml file with appropriate values for your environment. See the dedicated Helm values section for guidance.
2. Create a namespace:
   

   kubectl create namespace incenter

3. In current setup, the registry is protected with credentials
   

   helm registry login public-registry.clavister.cloud

4. Install InCenter:
   

   helm install incenter --namespace incenter oci://public-registry.clavister.cloud/incenter/incenter-onprem --version 3.9.3 -f values.yaml

Post-install

After the installation is complete, a message is displayed with information on how to get started. The message can also be viewed again with helm status incenter.

If you encounter any problems, please see the dedicated troubleshooting section in this document.

Configure

All configuration for the On-prem release is done by setting helm values. Some values are required and must be set prior to the installation process. All values may be changed at any time using the “helm upgrade” command. Please consult the official helm documentation for up-to-date information on how this is done.

A table of supported helm values is provided below.

Helm values

† This value should only be used when using an external OpenSearch cluster. See the section below on that topic.

Backup and Restore

MongoDB

One easy way to back-up and restore the data in the configuration database (MongoDB) is to use the mongodump and  mongorestore commands, and copying the resulting file.

Backup

Assuming the kubectl context is configured to use the Kubernetes cluster and the namespace that InCenter is installed in, these commands can be used, assuming and sh-like shell:

mongoPod=$(kubectl get pod -l app=mongodb -o name)
kubectl exec "${mongoPod}" -- mongodump --oplog --archive=/mongodb.backup.archive --gzip
kubectl cp "${mongoPod#pod/}":/mongodb.backup.archive mongodb.backup.archive

Restore

Please note that the following will completely remove all current data in the MongoDB service, and replace it with what has been stored in the archive.
Assuming that the back-up archive is in the current directory and is named mongodb.backup.archive and again, assuming the kubectl context is configured to use the Kubernetes cluster and the namespace that InCenter is installed in, these commands can be used, assuming and sh-like shell:

mongoPod=$(kubectl get pod -l app=mongodb -o name)
kubectl cp mongodb.backup.archive "${mongoPod#pod/}":/mongodb.backup.archive 
kubectl exec "${mongoPod}" -- mongorestore --oplogReplay --archive=/mongodb.backup.archive --gzip --drop

Stopping and starting back-end processes

For the safest results, the running processes using the database can be stopped before the back-up or restore is performed. To do this, the following command can be used before the ones mentioned above:

kubectl scale rs -l app_group=incenter --replicas=0

Afterwards, the processes can be started up again with:

kubectl scale deploy/core deploy/analytics deploy/authentication deploy/webui --replicas=1

OpenSearch

It is possible to back-up the data in OpenSearch by creating a snapshot and subsequently copying the files from the pod. The restore operation is similar, but in reverse order.

For both the back-up and restore operations access is needed to the pod for executing commands and copying files.

Backup

Register the snapshot repository:

kubectl exec -it deploy/logdb -c logdb -- \
  curl -X PUT \
  --insecure --silent --show-error \
  --cert config/certs/admin.crt --key config/certs/admin.key \
  --header 'content-type: application/json' \
  --url 'https://127.0.0.1:9200/_snapshot/repo' \
  --data '{
    "type": "fs",
    "settings": {
      "location": "/usr/share/opensearch/snapshots"
    }
  }'

Create the snapshot:

kubectl exec -it deploy/logdb -c logdb -- \
  curl -X PUT \
  --insecure --silent --show-error \
  --cert config/certs/admin.crt --key config/certs/admin.key \
  --header 'content-type: application/json' \
  --url 'https://127.0.0.1:9200/_snapshot/repo/1?wait_for_completion=true' \
  --data '{
    "indices": "*,-.*"
  }'

Copy all snapshots to a local directory:

podName=$(kubectl get pod -l app=logdb -o jsonpath='{.items..metadata.name}')
kubectl cp ${podName}:/usr/share/opensearch/snapshots ./snapshots

Restore

Copy the local snapshots to the pod:

podName=$(kubectl get pod -l app=logdb -o jsonpath='{.items..metadata.name}')
kubectl cp ./snapshots ${podName}:/usr/share/opensearch/snapshots

Register the snapshot repository:

kubectl exec -it deploy/logdb -c logdb -- \
  curl -X PUT \
  --insecure --silent --show-error \
  --cert config/certs/admin.crt --key config/certs/admin.key \
  --header 'content-type: application/json' \
  --url 'https://127.0.0.1:9200/_snapshot/repo' \
  --data '{
    "type": "fs",
    "settings": {
      "location": "/usr/share/opensearch/snapshots"
    }
  }'

Restore the latest snapshot:

latestSnapshot=$(kubectl exec -itq deploy/logdb -c logdb -- \
  curl -X GET \
  --insecure --silent --show-error \
  --cert config/certs/admin.crt --key config/certs/admin.key \
  --header 'content-type: application/json' \
  --url 'https://127.0.0.1:9200/_cat/snapshots/repo?s=start_time%3Adesc' | head -n 1 | cut -d' ' -f 1)

kubectl exec -it deploy/logdb -c logdb -- \
  curl -X POST \
  --insecure --silent --show-error \
  --cert config/certs/admin.crt --key config/certs/admin.key \
  --header 'content-type: application/json' \
  --url "https://127.0.0.1:9200/_snapshot/repo/${latestSnapshot}/_restore" \
  --data '{
    "rename_pattern": "(.*)",
    "rename_replacement": "$0-restored",
    "include_aliases": false
  }'

This will create indexes, which are named differently than the original indexes. Please check that the data has now been restored.

External OpenSearch cluster

It is possible to use an external OpenSearch cluster not running in the local namespace for installations that have higher expectations on logging infrastructure. Some configuring needs to be performed on the cluster for this to work. The following sections indicate some steps that should be performed.

Versions

The following versions of OpenSearch have been confirmed working with InCenter:

• 2.17.1
• 2.19.6
• 3.8.0

OpenSearch objects

A number of objects need to be installed to OpenSearch for smooth running of the log ingestion. The following will be installed:

• Roles for the InCenter user and the InCenter log ingestion user, with minimal permissions.
• (Optional) Users for InCenter and the InCenter log ingestion. These can be created manually as well, depending on your authentication policies, but the users must have the applicable roles mentioned above.
• Ingest pipelines for logs. These will parse, transform and possibly enrich the data that will be indexed, for better saved data format and query performance.
• ISM policies for index rollover and index retention settings.
• Index templates to ensure the right types for some of the log fields.
• Initial indexes to give them the right rollover index and name format.

The initialization script can be run by running the command below. Ensure to use the right version, URL and authentication details.

kubectl run init-opensearch --attach --image public-registry.clavister.cloud/incenter/logdbinit-onprem:0.0.0 \
  --restart=Never \
  --env OPENSEARCH_URL=https://opensearch-cluster-master.opensearch.svc.cluster.local:9200 \
  --env ADMIN_USERNAME=admin \
  --env ADMIN_PASSWORD=VeryStrong!P4ssword \
  --env INCENTER_USERNAME=incenter \
  --env INCENTER_PASSWORD=VeryStrong!P4ssword \
  --env LOGRECEIVER_USERNAME=logreceiver \
  --env LOGRECEIVER_PASSWORD=VeryStrong!P4ssword

Shipping logs

When using an external OpenSearch cluster, no components are automatically installed for getting logs into the database. The log shipper must put the syslog message received from the various nodes into an object with a “message” field, and nothing else. It should be indexed into the “node-logs” index, which has a default pipeline, which will parse the message and put it in the correct index.

For reference, this example configuration can be used for the old logstash image: “opensearchproject/logstash-oss-with-opensearch-output-plugin:8.9.0”:

input {
  udp {
    port => 1514
    type => "syslog"
  }
}

output {
  opensearch {
    hosts => ["https://opensearch-cluster-master.opensearch.svc.cluster.local:9200"]
    user => "logreceiver"
    password => "VeryStrong!P4ssword"
    index => "node-logs"
    ssl => true
    ssl_certificate_verification => false
    manage_template => false
  }
}

Please note that for logstash, don’t use the “syslog” input, but the “udp” input. This prevents creation of extra fields, which will be parsed later in the pipeline anyway. Also, it will create field type inconsistencies and thus errors, so no events will be created.

Also create a service as needed for the running container. Check the logs of the container if no logs are being ingested. It could indicate connection, authentication, permission or other issues with the configuration.

Uninstall

Steps

1. Uninstall the helm release:
   helm uninstall -n incenter incenter
2. Delete the namespace:
   kubectl delete ns incenter

Troubleshoot

General

If something is not working as expected, it is generally good to start by verifying that all micro-services are running and healthy. Here are some useful commands for that purpose:

kubectl get pods
kubectl describe deployment/<app>
kubectl logs deployment/<app>

If a solution to a problem cannot be solved using this troubleshooting guide, please do not hesitate to contact our support team.

Forgot my password

Immediately after installation a default admin account with a random password is created. See the install section for details on how to retrieve it.

If you have changed the default password and forgotten the new password, as of now there is no simple way to retrieve or reset it. If there are other admin accounts configured, one of them can be used to reset your password on the users page. If this is not possible, please contact our support team.

Reports not being sent

Responsible micro-services: analytics

Make sure that an SMTP server has been correctly configured using the smtp-prefixed helm values.

To help with troubleshooting, it is possible to send a report manually instead of waiting for its scheduled occurrence:

1. Go to the reports page in the web interface.
2. Select any report so that the side drawer is opened.
3. In the drawer, select the "more options" symbol next to the title, then select "Send now" and follow the instructions.

Alerts not being sent

Responsible micro-services: analytics

Make sure that an SMTP server has been correctly configured using the smtp-prefixed helm values.

Related articles