The splitlogs Binary

Introduction

The /usr/local/cpanel/bin/splitlogs utility allows you to fine-tune your resource and performance trade-offs when creating Apache logs. The splitlogs utility also provides some performance optimizations, including the ability to close file handles without restarting Apache. This greatly improves performance during log and bandwidth processing.

By design, Apache opens and writes to a single log file for each domain. Each virtual host in Apache's configuration file is assigned its own log.

cPanel & WHM systems open and write to 2 log files:

• One log file tracks the domain's resource usage (byte log).
• The other log file tracks how the domain is accessed, and by whom (access log).

cPanel & WHM systems use Apache's piped logs functionality, along with the splitlogs utility, to process Apache log information. In effect, cPanel & WHM is able to reduce Apache's memory consumption by changing how the access and byte logs are written.

cPanel & WHM systems run 2 instances of the splitlogs utility — one to manage byte logs, and another to manage access logs.

The splitlogs utility will extract the domain name and port, and use this information as the space-separated line in the log file. The log file's name is based on this information, and written to either the directory you specify using this utility, or the default location for Apache logs (/usr/local/apache/).

Usage

/usr/local/cpanel/bin/splitlogs --bincheck
/usr/local/cpanel/bin/splitlogs --dir=/logs/apache
/usr/local/cpanel/bin/splitlogs --dir="/logs/apache" --suffix=".log" --sslsuffixi="_ssl.log" --maxopen="20000" --buffer="yes"

Parameters

Program check

Apache configuration options

Performance tweaking options

More about --maxopen

The parameter allows you to configure a resource vs. speed trade-off. The --maxopen parameter allows you to specify the maximum number of log files splitlog can keep open at any one time.

Essentially, this value is the number of files splitlogs can write to at any time without closing and opening additional log files.

The default value for this parameter is 16,000. If the value you choose is too large, the OS will refuse to open a file when it needs to, and the parameter's value will return to the default value of 16,000.

Important considerations:

• When the maximum number of files are opened and splitlogs needs to write to a new file, the oldest file is closed and the new file is opened. This allows the splitlogs utility to write to the log file.
• The splitlogs utility will not close any log files until the value set in --maxopen is exceeded. This means that, as long as the number of log files opened by splitlogs is less than the value set in --maxopen, no action is taken.
• The spllitlogs utility also closes files during log and bandwidth processing, and when Apache restarts.
• The splitlogs utility will only open a log file to write to it when its corresponding domain is accessed. This differs from Apache's default behavior, wherein the number of log files opened is equal to twice the number of virtual hosts configured in the Apache configuration file (/usr/local/apache/conf/httpd.conf). This happens because there are 2 log files for each virtual host (1 access log and 1 byte log).
• Setting this parameter too low will cause the splitlogs binary to open and close files more than necessary, reducing performance.
• Setting this parameter too high requires more system resources and OS file handles, increasing the overall system load. This problem is exacerbated if buffering is enabled.
• Ultimately, you should determine the --maxopen parameter's value based on the current pattern of access to sites on your system.

More about --buffer

This configuration option is another resource vs. speed trade-off. Under most conditions, the splitlogs utility uses buffered writes by default.

Important considerations:

• Enabling buffered writes makes writing to individual log files faster, but requires more memory for each open log file.
  • You will only experience this performance increase if the system has memory to spare and the server's load is light enough to flush bufferers in a timely fashion.
  • If the system is overloaded or does not have extra memory to spare, write buffering will decrease performance overall.
• Disabling buffered writes substantially reduces the amount of memory required by the splitlogs utility.
  • For example, on a system that has an OS buffer of 4,000 file handles and where the --maxopen parameter is set to 1024, 4 MB of memory is used per splitlogs instance.
• If the /var/cpanel/conserve_memory flag file exists, buffered writes are disabled by default. You can override this setting using the splitlogs utility by passing the ‑‑buffer parameter yes (e.g. /usr/local/cpanel/bin/splitlogs ‑‑buffer="yes".
• You may experience some data loss if the splitlogs is terminated ungracefully before the buffers are flushed.

Note: If you decide to modify the --buffer option, monitor the system load and memory usage to determine whether the trade-off is reasonable.

The splitlogs utility's configuration file

When rebuilding Apache's configuration, the cPanel & WHM system will examine the splitlogs utility's configuration file at /var/cpanel/conf/splitlogs.conf. The system uses the contents of the configuration file to set command line options for CustomLog directives in Apache's configuration file (/usr/local/apache/conf/httpd.conf).

The configuration file consists of line-delimited key=value pairs. The key entries are the same as the command line parameters described in the parameters section of this document.

After specifying custom directives in splitlogs.conf, you must rebuild and restart Apache. To do so, you can use the following commands:

/usr/local/cpanel/scripts/rebuildhttpdconf
/etc/init.d/httpd -k graceful

Notes about performance concerns

• The splitlogs utility writes information to its log file whenever it reaches its file handle limit 1,000 times. Ultimately, these log messages should indicate how often the splitlogs utility is opening and closing log files.
  • If you do not encounter these messages in the splitlogs utility's log file, you should reduce the --maxopen parameter's value.
  • If you encounter an unusually high number of these messages in the splitlog utility's log file, you should increase the --maxopen parameter's value. You should determine this value based on the current pattern of access to sites on your system.
• As a part of the regular maintenance of your system, you should examine these values whenever changes in load occur or whenever the number of domains hosted by the system changes. The correct value today may be wrong in the future.