Introduction

While the error log was originally intended for errors only, it has evolved into a general-purpose Datometry Hyper-Q system log, and includes a variety of diagnostic messages. It is usually a good place to look when investigating any problems that arise. 

Log File Naming and Rotation

The error log is rotated when Hyper-Q starts up. This means that there will always be a fresh new log created for any given instance of the Hyper-Q server process; you will never see log messages from two different instances of the server process in the same log file. The log will also be rotated at midnight (as determined based on the local time zone of the server). There is one current, active error log for each individual Hyper-Q VM. At some point in the future we may add support for the use of external log rotation tools as well, though for now the Hyper-Q rotation behavior is automatic and mostly non-configurable.

The precise file naming for the error log varies depends on whether the legacy logger is enabled or not (see the "enable_legacy_logger" option below for more details). If the legacy logger is enabled, then every log file name includes the timestamp from when that file was first opened, and rotation does not actually rename any log files, but rather simply closes the current log file and opens a new one. If the legacy logger is disabled, then the error log file name contains no timestamp when it is opened (i.e. it is just named “error_log.csv”), and the file is renamed to add a timestamp when that file is rotated, with the timestamp being based on the time of rotation.

File names with timestamps are formatted as “base_log_name-YYYY-MM-DDTHH-MM-SS.extension”. Example: “error_log-2021-03-25T23-59-53.csv".

Format

The error log is formatted as CSV (comma separated variable). If the legacy logger is enabled, a column header line will be printed at the top of each log file; if the legacy logger is disabled, the header line will be omitted. Regardless of the presence or absence of a column header, the log file will always be formatted as valid CSV. The error log usually contains one message per line, though occasionally messages may span multiple lines. Note that although multi-line rows are considered valid CSV as per RFC 4180, some tools which are otherwise able to parse CSV files may have problems with this.

The columns are fairly similar to what you might expect to find in a typical non-CSV system log file, and so the plain text is normally fairly readable on its own, even without using any CSV-specific tools to process and/or reformat it. The CSV columns are as follows:

• “Time”: the timestamp of when the event occurred, using the local server timezone and formatted with millisecond-precision timestamps as “YYYY-MM-DD HH:MM:SS.mmm”.
• “Pid”: the internal process ID of the process that published this message. This column allows for filtering out log messages from a specific process, which can be especially useful if one particular process is suspected of having an issue.
• “Session Id”: the integer session identifier for the session that an event belong to. Defaults to “0” if a message is not associated with a session.
• “Client Request Id”: a UUID string identifier for the current client request being handled, or an empty string if the message is not associated with a client request.
• “LSN”: the “logon sequence number” integer ID, generated for bulk jobs that utilize multiple parallel sessions, and usable to tie together different sessions with distinct session IDs that are part of the same job. Defaults to “0” if a message is not associated with a session, or if the associated session does not have an LSN.
• “Module”: the internal source code module that wrote this message.
• “Line”: the line number of the source code line that wrote this message.
• “Message Code”: an integer prefixed with the string “DTM” which identifies what type of message this is. Example “DTM1000”.
• “Severity”: indicates the importance/urgency of the message. Severity values in the error log may either be one of the standard syslog severity levels described in RFC 5424, or a special level of “TRACE” which is for certain messages that are even lower-level than the standard “DEBUG” severity. The system log level setting will dictate whether messages below a certain severity level are included in the logs or not.
• “Message”: the actual text of the log message itself.
• “Query String”: this column is usually unused and contains an empty string, but it can be used to include SQL query text alongside the log message.

Message Codes

For many log messages, the Message Code column will contain a special placeholder value that simply corresponds to the log severity, but certain messages have their own dedicated codes which can make it easier to filter the log files for these messages. Standard INFO messages will use DTM1000, and other severity levels will use codes in the DTM1013-DTM1021 range. There are hundreds of other codes though which will not be explicitly listed here, as the meaning of a specific message code can be clearly seen by looking at the text of the log message where it appears.

Configuration Options

There are a few different options that can be specified which affect the behavior of the error log. All of these settings are optional.

dtm.ini settings

All of the following settings can be specified in the dtm.ini config file, and are specified under the [general_settings] header.

log_dir

Specifies the directory that the log files are written to. NOTE that this also affects other log files, such as the tracer log and system_info log. (There is not currently a way to specify different directories for different types of logs.) Also note that if the -logdir flag is specified on the command line at startup, then that will override this setting.

enable_legacy_logger

If set to “true”, log naming, formatting, and rotation will follow the same behavior as in previous versions of HyperQ. If set to “false”, some of those legacy behaviors will change, as described elsewhere in this document. In addition to these small behavior differences, setting this to “false” will enable a log overload protection system to help maintain system stability in situations where very high load causes large amounts of log messages to be written at once. In these types of situations, the overload protection system may drop messages if it becomes necessary to maintain system stability; if this happens, notice of the dropped messages will be written separately to the error log.

CLI settings

-log

This flag can be used to specify the minimum severity level of log messages that will be displayed. The value of the flag can be set to any of our supported severity levels (the standard syslog levels or “trace”), though note that the severity level on the CLI must be specified in lowercase. Example: -log warning would suppress log messages below the WARNING level.