The Problem

I recently set up Loki and Promtail in my Kubernetes cluster, which does a great job and finding any logs pushed to stdout just by using the default service discovery from the helm chart.

However I’m also running an opnsense router, and I wanted to be able to ingest logs from it as well. Of course, I don’t want plaintext transmissions of my logs, nor do I want logs to be ingested from unauthenticated sources.

Since already use a HashiCorp Vault server with a PKI engine enabled, I figured mTLS (mutual TLS) was the way to go. I won’t go into details on the Vault PKI configuration (which could be replaced with self-signed certificates fairly easily) nor will I discuss the opnsense configuration (its easy to add a certificate and CA via the web interface anyway).

Skip to the end to see a complete configuration file.

First Attempt - Directly Adding a Reciever

The configuration guide for syslog in Promtail suggests using a syslog reciever, warning that all messages to it must be in the RFC5424 format. Despite this warning, I initially implemented the reciever directly, as the opnsense syslog-ng sender supported RFC5424 directly.

This worked fine with plaintext, however upon trying to use mTLS (which Promtail can be configured with for syslog) it started to fail - it turns out the encryption support is broadly available in Promtail but not supported for the syslog reciever.

Suddenly, running a separate reciever was making more sense.

Using syslog-ng

I opted to go with syslog-ng for my forwarder since that is what was already in use on opnsense.

Broadly speaking when configuring syslog-ng there are three types of items:

  1. Sources, which look like source name {}
  2. Destinations, which look like destination name {}
  3. Logs, which connect one or more sources and destinations and look like log {}

I wanted two different log connections - forwarding mTLS protected logs from the network onto Promtail and connecting the output of syslog-ng to stdout so that Promtail could consume its errors like any other container.

Sending syslog-ng Messages to stdout

The simplier of the two configurations to understand is sending the syslog-ng messages to stdout, so we’ll look at that first.

First you need to create a source with the internal messages. You use internal() to do this, so to create a source named s_system with internal() messages the configuration looks like the following:

source s_system { 
        internal(); 
};

Next you need a destination - stdout. To create a destination named d_stdout that goes to stdout, do the following:

destination d_stdout {
         pipe("/dev/stdout"); 
};

Finally you need to connect the two with log, like follows:

log {
    source(s_system);
    destination(d_stdout);
};

Forwarding Logs with mTLS

Next up, and considerably more complex, is the actual log forwarder.

First is a source called. s_network which will listen on port 6154 for TLS traffic.

Initially I tried to configure this using the syslog however this had some troubles with the formatting of the messages when using TLS, so I discovered the better option was network

It explicitly configures the certificate, private key and the CA’s authority, allowing it to validate client certificates. Additionally it is configured for IPv4, though IPv6 can also be used.

source s_network {
        network (
                ip-protocol(4)
                port(6514)
                transport("tls")
                tls (
                        cert-file("/tls/cert")
                        key-file("/tls/key")
                        ca-file("/tls/ca")
                )
        );
};

Next is the destination, called d_loki.

This will point at your Promtail instance configured per the syslog in Promtail instructions. Note that this connection is not encrypted, so you should either run both Promtail and the forwarder on the same system and use localhost, or be sure that you are using an encrypted channel. Replace promtail.somedomain with the appropriate address for Promtail.

destination d_loki {
  syslog("promtail.somedomain" transport("tcp") port(1514));
};

With that done, you just need to connect the source and destination.

log {
        source(s_network);
        destination(d_loki);
};

Complete Configuration

This is a relatively simple configuration, and likely will need to be updated in the event that you have many systems sending their logs to the forwader, or simply have a more complex environment. The administration docs may prove helpful in the event that you need something more sophisticated.

The complete configuration is provided below. Also note the @version: 3.37 at the beginnning, this is required to match the syslog-ng version you are deploying.

@version: 3.37

source s_network {
        network (
                ip-protocol(4)
                port(6514)
                transport("tls")
                tls (
                        cert-file("/tls/cert")
                        key-file("/tls/key")
                        ca-file("/tls/ca")
                )
        );
};
 
source s_system { 
        internal(); 
};

##################################################
destination d_loki {
  syslog("promtail.somedomain" transport("tcp") port(1514));
};
 
destination d_stdout {
         pipe("/dev/stdout"); 
};

##################################################
log {
	source(s_system);
	destination(d_stdout);
};

log {
        source(s_network);
        destination(d_loki);
};