Set Up Unified Logging Using Fluentd

This applies to: Visual Data Discovery

In Symphony, you can use Fluentd as a logging layer to which you can direct the logs for various components of Symphony. This allows you to customize the log output to meet the needs of your environment.

Symphony leverages Fluentd’s unified logging layer to collect logs via a central API. Fluentd can be configured to aggregate logs to various data sources or outputs. For example, if you are directing all log files from your zoomdata-websocket.log and zoomdata-errors.log to a Fluentd server, you can add one of Fluentd’s plug-ins to write the log files to Elasticsearch to analyze web client errors for your environment.

Unified logging does not replace Symphony's default logging architecture. It only augments that experience with an option for those wanting additional logging control.

By default, unified logging is disabled and needs to be enabled for each microservice you want captured in Fluentd.

If written to a data store compatible with Symphony’s connectors, Symphony can then be used to analyze and visualize your logs. The selected data store must be supported by Symphony's connectors if you plan to view the log files in Symphony dashboards. A list of supported data stores is provided in Data Connector Reference. The selected data store should be available and configured in conjunction with Fluentd to receive logs.

At a high level, the steps to set up Fluentd are as follows:

  1. Set up a database to store the Fluentd log files. Typically, customers store log files in a log capture service such as Elasticsearch or in a database server such as PostgreSQL.

  2. Configure Fluentd logging for your Symphony installation. See Configure Unified Logging Using Fluentd. This includes configuring the Fluentd td-agent.conf file to identify the Symphony microservice log data you want logged.

  3. Enable unified logging and configure the host and port information for your Fluentd server for each microservice log file from which you are extracting data for Fluentd logging. See Enable Unified Logging in Symphony Using Fluentd.