Monitoring and collecting syslog messages from Unix Domain Socket

Application log is the traditional way to monitor an application/service. On *nix-based system, Syslog is a common but powerful tool for centrally monitoring applications logs. The primary use of syslog is for system management as capturing log data is critical for sysadmin, devOp team, or system analysts, etc. This log data is helpful in case of investigating/troubleshooting problems and maintaining healthy functioning of systems.

Syslog offers a standard log format and a standard alert system with different severity levels to applications in form of a log API. Log daemons such as rsyslog are versatile and flexible with various configuration options that enable different way to interact with the logs: log to file, log to a remote server via network (TCP, UDP sockets), log to local Unix domain socket. Log clients or log analytic applications can collect log data from the log daemon via these interfaces.

Although it is feasible to directly read log messages from the regular syslog output files, it is more preferable to collect log data from the daemon using the socket interface since socket is more suitable for data streaming. TCP/UDP sockets can be used to access log data from the network (TCP/IP). But if the application runs locally on the same machine as the log daemon, Unix domain socket (UDS) may be the best option.

Unix Domain Socket is an inter-process communication mechanism that allows bidirectional data exchange between processes running on the same machine. Thus, UDSs can avoid some checks and operations (like routing); which makes them faster and lighter than IP sockets.

In this post, we will learn how to collect log data from syslog via UDS in C. We will use rsyslog as log daemon in this post.

A use case will be presented at the end of the post.

Configuring rsyslog to log data into an Unix Domain Socket

It is able to configure rsyslog to log data into an Unix Domain Socket by using the omuxsock module. In this case, the log daemon acts as a client socket, it will first check if the socket file exists. If it is the case, the daemon will constantly stream log data to that socket. The daemon will not created this socket file if it is missing.

On Ubuntu system, create a new file /etc/rsyslog.d/30-log-to-socket.conf with the following configuration:

# Load omuxsock module
$ModLoad omuxsock
# Set UDS socket path
$OMUxSockSocket /var/syslog.sock
# Configure template log data in JSON format
template(name="json_syslog"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")       property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"type\":\"syslog_json")
      constant(value="\",\"tag\":\"")           property(name="syslogtag" format="json")
      constant(value="\",\"relayhost\":\"")     property(name="fromhost")
      constant(value="\",\"relayip\":\"")       property(name="fromhost-ip")
      constant(value="\",\"logsource\":\"")     property(name="source")
      constant(value="\",\"hostname\":\"")      property(name="hostname" caseconversion="lower")
      constant(value="\",\"program\":\"")      property(name="programname")
      constant(value="\",\"priority\":\"")      property(name="pri")
      constant(value="\",\"severity\":\"")      property(name="syslogseverity")
      constant(value="\",\"facility\":\"")      property(name="syslogfacility")
      constant(value="\",\"severity_label\":\"")   property(name="syslogseverity-text")
      constant(value="\",\"facility_label\":\"")   property(name="syslogfacility-text")
      constant(value="\",\"message\":\"")       property(name="rawmsg" format="json")
      constant(value="\",\"end_msg\":\"")
      constant(value="\"}\n")
}
# Forward all log messages using the template
*.* :omuxsock:;json_syslog

It is simple, we configure rsyslog to do the following things:

• stream log data to an Unix Domain socket at /var/syslog.sock
• Define a log template in JSON format with different properties such as severity level, program name, message, timestamp, etc.
• Forward all logs to the socket using the defined template

We could define a plain-text template but JSON is more convenient and more consistent as data format. More on rsyslog template definition can be found here.

The log can also be filtered in many different ways, for example we can filter the log by severity level, by replace the last configuration line with the following option:

# Only forward log messages of "notice" priority and higher
*.notice :omuxsock:;json_syslog

Restart the daemon to apply the configuration:

sudo systemctl restart rsyslog

Collect log data in C via Unix Domain Socket

As stated in the previous section, it is the responsibility of the application to create the socket. Note that the omuxsock only support DATAGRAM socket (similar to UDP). This makes sense since the DATAGRAM socket is lighter and faster than the STREAM socket, and since it is a Unix Domain socket, there will be no risk of package lost as DATAGRAM socket is as reliable as STREAM socket.

The following C snippet shows how to create the socket and collect log data via UDS:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <arpa/inet.h>
#include <signal.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/stat.h>
#define MAX_LOG_SIZE 1024u

/** path to the socket file, 
* need to be synchronised 
* with rsyslog configuration */
#define SOCK_PATH "/var/syslog.sock"

static int syslog_sock_fd = -1;

/**
 * Interrupt_handler – so that CTRL + C can be used to
 * exit the program
 **/
void clean_up(int signum)
{
    close(syslog_sock_fd);
    printf("socket connection closed\n");
    exit(signum);
}

int main()
{
    struct sockaddr_un saddr, caddr;

    /*Log buffer*/
    char buffer[MAX_LOG_SIZE] = {'0'};

    int length = sizeof(struct sockaddr_un);
    int error;

    /*Trap SIGINT and SIGTERM signals*/
    signal(SIGINT, clean_up);
    signal(SIGTERM, clean_up);

    /*create the UDS socket */
    if ((syslog_sock_fd = socket(AF_UNIX, SOCK_DGRAM, 0)) < 0)
    {
        printf("unable to create socket\n");
        exit(0);
    }

    /* Remove the socket file if exists */
    remove(SOCK_PATH);

    /*Bind the socket to the socket file*/
    saddr.sun_family = AF_UNIX;
    snprintf(saddr.sun_path, (strlen(SOCK_PATH) + 1), "%%s", SOCK_PATH);

    if (0 != (bind(syslog_sock_fd, (struct sockaddr *)&saddr, sizeof(struct sockaddr_un))))
    {
        printf("bind failed\n");
        clean_up(0) :
    }

    /**
    * Allow every one to write to this file,
    * if security is a thing,
    * the file permission should be handled correctly
    * */
    chmod(SOCK_PATH, 0777);

    /* Receive data from syslog */
    int num_of_bytes;
    printf("Wait for log message\n");
    while (1)
    {
        memset(buffer, 0, sizeof(buffer));
        /*read log data*/
        num_of_bytes = recvfrom(
                syslog_sock_fd, buffer, sizeof(buffer) - 1, 0, (struct sockaddr *)&caddr, &length);
        if (num_of_bytes == -1)
        {
            printf("read from client failed\n");
            clean_up(0) :
        }
        /*simply print it*/
        printf("%%s\n", buffer);
    }
}

The code is straightforward, data read from DATAGRAM Unix domain socket use the same API as traditional UDP socket. The only difference is the way how the socket is created and binded. To create the socket we use AF_UNIX instead of AF_INET. The socket is binded to a local socket file instead of an IP address.

When running, the application creates new socket file and and performs blocking read on the binded socket. On the rsyslog side, the daemon starts streaming log data whenever the socket file becomes available.

Note that, by standard, the log size is limited to 1024 bytes per message, so a buffer of 1024 bytes is enough in most case.

Use case

I started doing the research on how to collect log data from syslog when i want a more convenient way to manage and to monitor my home made server (that hosts this website). This server integrates several services (including some services developed by myself) needed by all of my websites. I wanted to developed an AntOS application that allows me to monitor log from these services directly in the VDE (VIrtual Desktop Environment).

AntOS comes with API and SDK to develop front-end applications that run directly inside the VDE. Data streaming from the server to the application can be done through the Antunnel API which is based on Websocket. The API consists of a front-end API and a back-end plugin that defines a generic communication layer using the websocket protocol. This layer allows to create multiple two-ways communication channels on top of a single websocket connection. Server side applications can create channels, front-end applications can subscribe and listen to theses channels. The Antunnel API take care of dispatching the websocket messages to the correct front-end/back-end applications.

The following diagram shows the basic idea of my web-based log client.

The log client consists of an AntOS application that uses the Antunnel front-end API and a server side application that uses the Antunnel back-end API. The server side application is developed in C and communicates with the rsylog daemon using the method described in this post.

The following GIF shows the application in action.