Ingesting Linux Authentication Logs With Opentelemetry

This guide configures an OpenTelemetry Collector running in Kubernetes to ingest /var/log/auth.log from a Linux node and forward it to Splunk through the HTTP Event Collector (HEC).

The resulting Splunk events use:

index=monitoring
sourcetype=linux:auth
source=/var/log/auth.log
host=host1.example.com

They are also tagged with the OpenTelemetry resource attribute, but don’t need to be:

k8s.cluster.name=example_cluster

Prerequisites

You need:

• An OpenTelemetry Collector Contrib or Splunk OpenTelemetry Collector image containing the filelog receiver and splunk_hec exporter.
• A Splunk HEC endpoint and token.
• /var/log mounted from the Kubernetes node into the collector container.
• A writable directory for file-reading checkpoints.
• Permission for the collector process to read /var/log/auth.log.

Some Linux distributions store authentication events only in systemd-journald and do not create /var/log/auth.log. Confirm the file exists on the host before deploying this configuration.

I’m using this as part of the splunk-otel-collector helm chart collecting far more than just these logs (k8s and host-based) on the cluster members.

Collector configuration

Add the following configuration to the collector:

extensions:
  file_storage:
    directory: /var/lib/otelcol/file-storage # for checkpoints etc

receivers:
  filelog/auth:
    include:
      - /var/log/auth.log
    include_file_name: false
    include_file_path: true
    # this can/should be tuned, I have it at 15s because some nodes are VERY underpowered
    poll_interval: 1s 
    start_at: end
    storage: file_storage
    resource:
      com.splunk.index: monitoring
      com.splunk.sourcetype: linux:auth
      com.splunk.source: /var/log/auth.log
      host.name: host1.example.com

processors:
  memory_limiter:
    # this can be tuned, I have it at 15s because some nodes are VERY underpowered
    check_interval: 2s
    limit_mib: 256

  resource/auth:
    attributes:
      - action: upsert
        key: k8s.node.name
        value: host1.example.com
      - action: upsert
        key: k8s.cluster.name
        value: example_cluster

  batch: {}

exporters:
  splunk_hec/auth:
    # your splunk instance
    endpoint: https://splunk.example.com:8088
    token: ${env:SPLUNK_PLATFORM_HEC_TOKEN}
    # overwritten by `resource['com.splunk.index']`
    index: main
    # overwritten by `resource['com.splunk.source']`
    source: opentelemetry
    tls:
      insecure_skip_verify: false
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
    sending_queue:
      enabled: true
      queue_size: 1000

service:
  extensions:
    - file_storage

  pipelines:
    logs/auth:
      receivers:
        - filelog/auth
      processors:
        - memory_limiter
        - resource/auth
        - transform/auth_source
        - batch
      exporters:
        - splunk_hec/auth

Replace https://splunk.example.com:8088 with the real Splunk HEC endpoint.

The token must be supplied through the SPLUNK_PLATFORM_HEC_TOKEN environment variable. Do not place the token directly in the collector configuration.

Although the exporter defines default index and source values, the com.splunk.index and com.splunk.source resource attributes override them in the records.

Kubernetes mounts

The collector needs read-only access to the host’s /var/log directory and writable storage for file checkpoints:

spec:
  template:
    spec:
      containers:
        - name: otel-collector
          volumeMounts:
            - name: host-var-log
              mountPath: /var/log
              readOnly: true

            - name: file-storage
              mountPath: /var/lib/otelcol/file-storage

          env:
            - name: SPLUNK_PLATFORM_HEC_TOKEN
              valueFrom:
                secretKeyRef:
                  name: splunk-hec
                  key: token

      volumes:
        - name: host-var-log
          hostPath:
            path: /var/log
            type: Directory

        - name: file-storage
          hostPath:
            path: /var/lib/otelcol/file-storage
            type: DirectoryOrCreate

Run the collector as a DaemonSet when authentication logs must be collected from every Kubernetes node. Each collector instance then reads the files mounted from its own node.

The collector process must have sufficient filesystem permissions to read auth.log. Use the narrowest permissions supported by the host rather than making the log file globally readable.

Reading and checkpoint behaviour

The receiver uses:

start_at: end
storage: file_storage

On its first run, the collector begins with new entries appended after startup instead of importing the entire existing file. It records its position using the file_storage extension so that normal pod restarts do not cause previously processed lines to be ingested again.

Keep the checkpoint directory on persistent node storage. An emptyDir volume loses the checkpoint whenever the pod is recreated.

Resulting fields

An input line remains the event body:

Sep 3 10:42:18 host1 sshd[1234]: Accepted publickey for exampleuser

The resulting OpenTelemetry record contains approximately:

body: "Sep 3 10:42:18 host1 sshd[1234]: Accepted publickey for exampleuser"

attributes:
  log.file.path: /var/log/auth.log

resource:
  com.splunk.index: monitoring
  com.splunk.sourcetype: linux:auth
  com.splunk.source: /var/log/auth.log
  host.name: host1.example.com
  k8s.node.name: host1.example.com
  k8s.cluster.name: example_cluster

Splunk interprets the com.splunk.* attributes as event metadata:

The k8s.cluster.name and k8s.node.name values remain searchable resource fields.

Verification

After deploying the collector, confirm that it is healthy and can see the mounted file:

kubectl exec -n opentelemetry <collector-pod> -- \
  sh -c 'test -r /var/log/auth.log && echo readable'

Check the collector logs for file permission, configuration, checkpoint, TLS, or HEC errors:

kubectl logs -n opentelemetry <collector-pod> \
  --container otel-collector

Finally, search Splunk:

index=monitoring
sourcetype=linux:auth
source="/var/log/auth.log"
host="host1.example.com"

Generate a harmless authentication event, such as a successful SSH login, if a fresh record is needed for verification. Avoid deliberately generating failed authentication attempts on production systems.

Troubleshooting

No events appear

Confirm that:

• /var/log/auth.log exists on the Kubernetes node.
• The file is mounted at the same path inside the collector.
• The collector process can read the file.
• The logs/auth pipeline is enabled.
• The HEC endpoint is reachable and its token can write to the monitoring index.

Old records are not imported

This is expected with start_at: end. Change it to start_at: beginning only when historical ingestion is intentional, as doing so can produce a large volume of duplicate or outdated events if checkpoints are also removed.

Events are duplicated after pod recreation

Confirm that the file_storage directory is backed by persistent node storage. Checkpoint data stored in an ephemeral volume is discarded when the pod is replaced.

The host does not have /var/log/auth.log

The system may store authentication events only in journald. Configure the OpenTelemetry journald receiver instead of creating or scraping an artificial auth.log file.