VyOS - Logging via containers to Logscale

Introduction

Generally network devices have their logs ingested into a central logging system through the industry standard syslog.

However syslog has several limitations, when utilising UDP message are limited to the max size of a single UDP packet (eg. 1024 bytes)

As VyOS is based on Debian Linux utilising journald logging, and its support for containers, it is possible to simply hook into journald and obtain enriched logs via a container skipping many limitations of syslog.

For example ingesting from journald will expose the source system unit (eg. cron / ssh / kernel) and simplify timestamp extraction.

This proof-of-concept will go over the process to setup the Logscale Collector as a container, additionally using Logscale Fleet Management to enable centralised monitoring and management.

Note: Whilst this article focuses on Logscale Collector, other collectors could also work (eg. Vector.dev) and I may publish follow up articles for Vector.dev

1. Import Parser

To ensure the logs are easily searchible, a parser should be configured, I have published a working parser on Github which can be imported into Logscale.

Navigate to the “Parsers” section under “Data connectors” and click “Add new parser”

Specify the Parser name and select “Import” and upload the file from Github, then click create.

2. Setup connector

Navigate to the “Data connections”, click “+ Add connection”, find the “Falcon Logscale Collector” item and then click “Configure”

Give connector and data source a name, then set the newly imported parser.

After creating the connector, generate the API keys and note for use later.

Reference: Documentation: Configure Falcon Logscale Collector

3. Define Fleet Configuration

Navigate to the “Fleet management” tab, switch to “Config overview”, and select “+ New config”.

Give the new config a name, and start with an empty config.

In the draft editor, copy the below config and adjust the sink siem_vyos-journal to have the connector token and URL obtained in the previous step, then click “Publish” to save the changes

sources:
  journal:
    type: journald
    sink: siem_vyos-journal
    # Optional. If not specified collect from the local journal
    directory: /var/log/journal
    # If specified only collect from these units
    #includeUnits:
    #  - systemd-modules-load.service
    # If specified collect from all units except these
    excludeUnits:
    ##  - systemd-modules-load.service
    # Default: false. If true only collect logs from the current boot
    currentBootOnly: true
sinks:
  siem_vyos-journal:
    type: humio
    token: <TOKEN>>
    url: https://<URL>.ingest.us-2.crowdstrike.com

Reference: Documentation: Configure Enrollment Tokens

4. Setup Fleet Management Enrollment Token

Navigate to the “Enrollment tokens”, and click “+ New token”.

Give the token a suitable name, and then set this token to use the config previously created.

Note: For advanced use you can leave the config undefined, and configure a group instead to match for the VyOS* OS and assign the config dynamically.

After creating the token, retrieve the “Enrollment token”, this is the long string of random characters at the end of the “Enrollment command”, this will be used when deploying the container.

References: Documentation: Fleet Config

5. Deploy Container to VyOS

The final step is to deploy the container to the VyOS devices, if there are multiple devices you can use the same enrollment token as many times as needed.

First step will be to create some persistent storage locations which will allow the collector to remember its enrollment and what it has processed between restarts.

I have created a custom logscale-collector image that includes additional environmental variables to enable enrollment, this will need to be added to the device before the container can be configured.

# Create persistent storage directories
mkdir -p /config/containers/humio-log-collector_var
mkdir -p /config/containers/humio-log-collector_etc

# Add container image (must be done outside of configure mode)
add container image semaja2/logscale-collector:latest

Now that everything is in place, we will need to switch to configure mode and configure the container.

To make this process simple the below commands have the ENROLL_TOKEN at the top, set the enrollment token obtained previously (eg. ENROLL_TOKEN=eyJoZ...)

Once the ENROLL_TOKEN variable is configured, the rest of the commands can be bulk entered and finally committed.

Caution: VyOS maintains roughly 1GB of journal logs, when the container is deployed it may cause high CPU/Memory usage whilst it processes the backlog, as such ensure ample resources are available.

# Set this shell variable to the enrollment token retrieved earlier
ENROLL_TOKEN=

# Configure container
edit container name logscale-collector
set image 'docker.io/semaja2/logscale-collector:latest'
set restart 'always'
set memory 512
set cpu-quota 1

## Provide host network to avoid extra configuration
set allow-host-networks

## Set host-name to match router, will be used in fleet management
set host-name $HOSTNAME

## Configure enrolment token from fleet management
## Can be removed after initial enrolment if persistent storage enabled
set environment HUMIO_LOG_COLLECTOR_ENROLL_TOKEN value $ENROLL_TOKEN


## Add host /var/log to container to extract logs, mount as read only to avoid tampering
set volume logs destination '/var/log'
set volume logs mode 'ro'
set volume logs source '/var/log'

# Add host /etc/os-release to enable fleet management to report correct OS version
set volume os-version mode 'ro'
set volume os-version source '/etc/os-release'
set volume os-version destination '/etc/os-release'

## Add persistent storage locations
set volume conf destination '/etc/humio-log-collector'
set volume conf mode 'rw'
set volume conf source '/config/containers/humio-log-collector_etc'
set volume var destination '/var/lib/humio-log-collector'
set volume var mode 'rw'
set volume var source '/config/containers/humio-log-collector_var'

After committing the changes the container should start, and appear in the Fleet management overview screen.