Apparent Discrepancies in Bandwidth Usage Statistics

For WHM version 11.38

Introduction

Often, when users compare bandwidth usage statistics provided by log analysis programs such as AWStats, Analog, Logaholic, and Webalizer to bandwidth statistics provided by cPanel & WHM, they are surprised to find apparent discrepancies.

These apparent discrepancies arise as a result of the way different programs measure bandwidth.

The purpose of this document is to explain the differences in measurement methods. Our goal is to help web hosts and website owners understand how to obtain the most accurate bandwidth report.

Where to find bandwidth data in cPanel & WHM

Bandwidth data in cPanel & WHM comes from 2 sources: data stored in summary files and the data stored in RRDtool databases.

Data stored in summary files

Several cPanel & WHM screens display (mainly numeric) bandwidth usage information stored in summary files. These screens include:

• WHM >> Account Information >> View Bandwidth Usage.
• WHM >> Account Functions >> Limit Bandwidth Usage.
• cPanel >> Stats >> Monthly Bandwidth Transfer.
• cPanel >> Logs >> Bandwidth >> monthly bandwidth usage pie charts.
• cPanel >> Logs >> Bandwidth >> select the domain >> Bandwidth By Day.

All of the information on these pages is generated by the same bandwidth monitoring tool.

In contrast, AWStats, Analog, Logaholic, and Webalizer process bandwidth according to a different method. This is why sometimes it appears that there is no correlation between the data shown on the bandwidth usage information screens and the data that AWStats, Analog, Logaholic, and Webalizer show.

Data stored in RRDtool databases

Most bandwidth graphs are generated from usage information stored in RRDtool databases. These graphs include:

• cPanel >> Logs >> Bandwidth >> Traffic for Today graph.
• cPanel >> Logs >> Bandwidth >> Traffic for Last 7 Days graph.
• cPanel >> Logs >> Bandwidth >> Traffic for Last 12 Months graph.

Data for these bandwidth graphs are calculated using a different method from the summary file data above.

What causes the discrepancies?

Differences in measuring daily bandwidth statistics

The information presented by cPanel & WHM is likely to differ from the day's usage as shown by log processing programs, such as AWStats, Analog, Logaholic, or Webalizer. There are three possible reasons for this:

1. The server administrator can set log processing and bandwidth processing cycles independently. This means that the 2 statistics can be generated hours apart, in which case, they may vary widely.
2. Even if the server administrator attempts to synchronize log and bandwidth processing cycles, the system will always process the statistics at least 8 minutes apart.
3. The statistics are processed such that usage data from 5 minutes before the time of processing, up to the time of processing, is always held until the next processing cycle. Therefore, those statistics would appear in a later report.

In short, daily bandwidth usage numbers are far too volatile for the bandwidth and log processing results to match.

Overall differences in measuring bandwidth

There are further differences between the methods used by log programs and cPanel & WHM to process bandwidth usage.

How the log processing programs measure bandwidth

The log processing programs extract the amount of bytes transferred from the Apache combined access log. The Apache combined access log records incoming data requests. One of the pieces of data recorded is the size of the requested file, measured in bytes.

The size amount of the requested file:

• only shows the full size of the content of the file being requested. It does not reflect the actual number of bytes that were transferred.
• measures the content of the file only. This does not include the size of the headers of the response.
• does not include the size of the request.

In other words, the log processing programs measure bandwidth by showing the size of the data that was requested, not the amount the data that was actually transferred.

How cPanel & WHM measures bandwidth

cPanel & WHM measures bandwidth by combining the amount of bytes of an incoming data request with the outgoing transferred data (i.e., the response to that request). cPanel & WHM obtains this data from the Apache logs.

The amount of bandwidth reported includes:

• the total bandwidth used by the incoming request that solicited the content.
• the total bandwidth used by the complete response: the sum of the size of the content plus the header.

How this affects the results

For most websites, these differences will be relatively small. The headers are a few hundred bytes; the request itself is usually much less. Given reasonably sized output content, this overhead works out to a few percentage points' difference.

There are a certain types of request/response combinations in which these differences begin to make a significant impact on the bandwidth measurement:

1. Large numbers of non-success responses — Because redirects and errors have virtually no content, and are mostly composed of headers, the bandwidth reported by the log processing programs will be much smaller than that reported by cPanel & WHM.
2. Large numbers of very small responses — For transferred content below 1 kilobyte, the log processing programs will show a smaller bandwidth usage amount than that reported by cPanel & WHM. This is because the combined size of the response headers plus the request can often be larger than 1 kilobyte.
3. Large requests cancelled by the user — As stated above, log processing programs measure bandwidth by showing the size of the data requested, not the amount the data that was actually transferred. This means that these programs do not make a distinction between completed and cancelled requests. Therefore, a high number of requests cancelled by the user may cause the amount of bandwidth reported by cPanel & WHM to look substantially smaller than the bandwidth reported by the log processing program. This situation occurs often with sites that serve a large number of video or audio streaming requests.
   • For example, 100 people request a 1GB video, but cancel the request after only 10% of the video has downloaded. cPanel & WHM would report that less than 10 GB of data were transferred. However, because of the way they obtain their information, the log processing programs would report that 100 GB were transferred.

Which report should I use?

You will want to use the more complete bandwidth measurement reported by cPanel & WHM, for example on the cPanel Bandwidth screen, or the WHM View Bandwidth Usage screen. Log analysis programs, while not as accurate in reporting bandwidth, are worth using for information about pageview statistics, referrals, and other information.

You should set bandwidth processing cycles as desired on the Server Configuration >> Statistics Software Configuration screen of WHM, in the Bandwidth Processing Frequency area of the Schedule Configuration screen.

This will allow you to balance server performance with reporting latency and, given those constraints, will provide the most accurate bandwidth usage statistics possible.

For up-to-date usage information, you may process an individual user's statistics at any given moment. You can do this on the Server Configuration >> Statistics Software Configuration screen of WHM, under the Process Statistics for User heading.

Related documentation

• How to Use the cPanel Bandwidth Graphs — This guide will help you use the information provided by the bandwidth graphs displayed in cPanel >> Logs >> Bandwidth.
• View Bandwidth Usage — Documentation for the View Bandwidth Usage feature in WHM.