Using and Managing Trace Files

This topic contains information on tracing and within s-Server including the following:

Introduction to Trace files

SQLstream uses the standard Java logging interface (defined in the package java.util.logging) to trace execution for debugging purposes. Trace files can let you troubleshoot stream functioning during development. Errors logged to the trace log are also available in the global error stream. See the Streaming SQL Reference Guide for more details.

All errors that are logged to the trace log are also available in a system management stream called sys_boot.mgmt.ALL_TRACE. The latest (up to 1000) error messages are also available in a table called sys_boot.mgmt.SAVED_TRACE. You can use these to create visualizations of errors; they can be accessed from SQLline or anywhere that has a JDBC connection to s-Server. See the Global Error Stream for more details.

Trace files and ALL_TRACE / SAVED_TRACE also include Periodic Parser Statistics.

Trace files are useful in troubleshooting streams, because they allow you to see when a stream failed and why.

s-Server has two trace logs. The first is the most useful in troubleshooting streams.

  • Trace.log This is the “master” trace, which includes all trace information for the entire server. It is located at /var/log/sqlstream/Trace.log.0. The trace files are rotated, so the most recent has an extension of .0, and older files are named Trace.log.1, Tracel.log.2, and so on.
  • ApplicationTrace.log This contains only trace information relevant to installed application components (such as plugins, user-defined transforms, and SQL statements). It is located at /var/log/sqlstream/ApplicationTrace.log.0; these files are also rotated.

Configuration Files for Trace Log

To control the depth of the logging and the formatting of the output, Java logging uses external configuration files in the standard Java properties format. You determine the location of these files during installation. By default, SQLstream uses the following logging configuration files:

Component Configuration File Default Trace Log File(s)
SQLstream s-Server /var/log/sqlstream/Trace.properties /var/log/sqlstream//Trace.log/var/log/sqlstream/ApplicationTrace.log
SQLstream s-Studio and other client tools /var/log/sqlstream/ClientTrace.properties /var/log/sqlstream/ClientTrace.log

The SQLstream s-Server configuration controls tracing to three different log files:

  • Trace.log This is the “master” trace, which includes all trace information for the entire server.
  • ApplicationTrace.log This contains only trace information relevant to installed application components (such as plugins, user-defined transforms, and SQL statements).
  • ClientTrace.log This records trace messages generated by client interaction with Streaming Data Protocol or JDBC; that is, all client interactions with s-Server.

You can modify the properties files to change the default logging. Before doing so, you should save a copy of the original, unmodified file.

Note: The installed files are made read-only by default so as to avoid accidental modification.

Configuring Tracing Levels

Because trace files are intended for debugging, by default, very little is logged to these files in normal conditions.

Configure Trace Levels Using Trace.properties

For debugging you can change the default tracing level by editing the Trace.properties or ClientTrace.properties file. To view more logging, open one of the properties files in a text editor and uncomment (remove the “#") lines such as the following (both from ClientTrace.properties):

# com.sqlstream.aspen.jdbc.level=FINE
# com.sqlstream.aspen.sdp.level=FINE

You can also change the tracing level for a particular package or class. For example, suppose you created a set of plugins that you want to debug at a high level of tracing. If the plugins are all part of a package called com.example.plugin, you can add the following to your Trace.properties:

com.example.plugin.level=FINEST

Configure Trace Levels from SQL using SETTRACELEVEL

You may also change the logging level from the SQL command line (or as part of a SQL script) by using the SETTRACELEVEL procedure. This can be more convenient than updating the Trace.properties file and waiting for the new levels to take effect. To repeat the previous example:

CALL SETTRACELEVEL('com.example.plugin.level','FINEST');

See also the topic Error Logging in the Streaming SQL Reference Guide.

Limiting Trace Log Size

The default tracing configurations are intentionally set for production mode, with limits placed on the maximum size of a trace file, and the number of times the file will be rotated, controlled by these properties in Trace.properties::

java.util.logging.FileHandler.limit=10000000
java.util.logging.FileHandler.count=20
  • .limit places a limit on the maximum size of any single trace file
  • .count specifies how many generations of the trace file will be retained. As soon as a trace file is full (reaches the limit size), it will be saved with a “1” appended to the filename, and a new trace file will be started. As the next file fills, each older file is renumbered (from .1 to .2 and so on) and the oldest is deleted.

For testing, it is can be helpful to change these settings to allow more information to be retained in trace files. Just be careful that your tracefiles do not use up all available disk space.

To control the maximum file size and rollover count, change the following properties in the file /var/log/sqlstream/Trace.properties:

Trace.log Property Name Default
File size java.util.logging.FileHandler.limit 10000000 bytes
Rollover count java.util.logging.FileHandler.count 20
ApplicationTrace.log Property Name Default
File size com.sqlstream.aspen.trace.ApplicationLogConfigurator.limit 10000000 bytes
Rollover count com.sqlstream.aspen.trace.ApplicationLogConfigurator.count 20

For SQLstream s-Studio, use the following properties in /var/log/sqlstream/ClientTrace.properties:

ClientTrace.log Property Name Default
File size java.util.logging.FileHandler.limit 10000000 bytes
Rollover count java.util.logging.FileHandler.count 20

If you set file size limit to zero, the file will never roll over, even if a rollover count is set.

Using Different Configuration Files

You can override which logging configuration file SQLstream uses by overriding the value for the property java.util.logging.config.file. To do so, set the property in aspen.custom.properties.

$SQLSTREAM_HOME/aspen.custom.properties.

If this file is not present, you may need to create it or copy it from $SQLSTREAM_HOME/support.

To change the user interface logging configuration file, set the property in

$HOME/.SQLstreamrc.

For example, to use the file /usr/local/MyTrace.properties for user interface logging configuration, add the following line to the .SQLstreamrc file in your home directory:

java.util.logging.config.file=/usr/local/MyTrace.properties

Periodic Parser Statistics Logging

The trace log contains information on how parsers are reading data from external sources such as log files, Kafka, Amazon Kinesis, and AMQP. This is known as Periodic Parser Statistics Logging. These sources are read through the Extensible Common Data Adapter. For more information on parsing data, see the topic Reading Data from Other Sources in the Integrating Guavus SQLstream with Other Systems guide.

You can control how often s-Server logs parser statistics with the following entry in Trace.properties:

com.sqlstream.aspen.namespace.common.input.logPeriod=10

This value determines how often in minutes that s-Server logs parser statistics. By default it is set to every 10 minutes.

For each Extensible Common Data reader open, the trace log lists information on the following:

Statistic Description
Throughput Bytes per second of input processed over the previous period.
Runnable Percent of time that the parser was running (or runnable but waiting for an available core to run in).
Blocked waiting for input Percentage of time that the parser was not runnable because it was waiting for more input to process.
Blocked waiting for downstream writes Percentage of time that the parser was not runnable because of backpressure from downstream processing.

Statistics on runnable and blocked parsers are approximated based on sampling log parser statistics. You can control sampling and reporting by changing the following parameters in Trace.properties:

Parameter Definition
com.sqlstream.aspen.namespace.common.input.logPeriod How often in minutes to log parser statistics. Defaults to 10. Set to 0 to disable logging.
com.sqlstream.aspen.namespace.common.input.probeFrequency How may times per logPeriod should parser statistics be sampled. Defaults to 100.

Example from Trace Log

INFO: Parser for LOCALDB.TEST_READER.P_Pump: throughput=12598532bps, runnable 20%, blocked waiting for input 80%, blocked waiting to write downstream 0%