Site Quality Monitoring — Crawling


Last modified: April 10, 2024

Overview

The Site Quality Monitoring interface (cPanel » Home » Metrics » Site Quality Monitoring) contains tabs that allow you to configure how the feature monitors your project for common issues. This document explains how to use the Crawling tab.

The Crawling tab contains monitoring checks that you can run yourself. These checks, known as crawls, are systemic tests of every monitored URL that you have added for your project.

The Crawling tab contains the following tabs:

Tab Description
Dead links This tab allows you to run a crawl that tests whether the links on the first 500 pages of your project return data.
Missing images and files This tab allows you to run a crawl that checks the first 500 pages of your project for missing or broken JavaScript, CSS, or image files.
Oversize files This tab allows you to run a crawl that checks that the JavaScript, CSS, and image files on the first 500 pages of your project are all smaller than 1 megabyte (MB).

The Dead links tab contains a crawler that checks whether the links on the first 500 pages of your project that it encounters return data. For example, if you add a link to your project that leads to an image you have taken off the Internet, that link will not return data. The crawler will register the link as dead.

Click number URLs to begin a crawl of your project for missing images and files, where number represents the number of pages the feature will crawl.

Important:

This action will not reset the countdown bar.

When you begin a manual crawl for the first time, this tab will update to contain a Past Checks table. This table will contain the following columns for all dead link crawls you have performed on your project:

Column Description
Date The day that you ran the manual crawl.
Found Pages The number of pages that the Site Quality Monitoring feature found on your project. If this number is greater than 0, you may click it to open that past check’s result report. The result report will open.
Searched Pages The number of pages that the Site Quality Monitoring feature crawled on your project.
Status The status of the monitoring crawl.

Click Stop all crawls that are currently running. to stop a manual crawl in progress.

The Site Quality Monitoring feature will email the addresses you have added to the Configure alert emails tab when it finishes the crawl.

Missing images and files

The Missing images and files tab contains a crawler that tests the first 500 pages of your project that it encounters to ensure that their JavaScript, CSS, and image files load. For example, if you try to display an image file that you have deleted on a page on your project, the page will return a failure code to the crawler.

Click number URLs to begin a crawl of your project for missing images and files, where number represents the number of pages the feature will crawl.

Important:

This action will not reset the countdown bar.

When you begin a manual crawl for the first time, this tab will update to contain a Past Checks table. This table will contain the following columns for all missing images and files crawls you have performed on your project:

Column Description
Date The day that you ran the manual crawl.
Found Pages The number of pages that the Site Quality Monitoring feature found on your project. If this number is greater than 0, you may click it to open that past check’s result report. The result report will open.
Searched Pages The number of pages that the Site Quality Monitoring feature crawled on your project.
Status The status of the monitoring crawl.

Click Stop all crawls that are currently running. to stop a manual crawl in progress.

The Site Quality Monitoring feature will email the addresses you have added to the Configure alert emails tab when it finishes the crawl.

Oversize files

The Oversize files tab contains a crawler that tests the first 500 pages of your project that it encounters to ensure that all their JavaScript, CSS, and image files are each smaller than 1 MB. For example, if you add a CSS file that is 3 MB, it will produce a failure.

Click 500 URLs to begin a crawl of the first 500 pages of your project for oversize files.

Important:

This action will not reset the countdown bar.

When you begin a manual crawl for the first time, this tab will update to contain a Past Checks table. This table will contain the following columns for all oversize file crawls you have performed on your project:

Column Description
Date The day that you ran the manual crawl.
Found Pages The number of pages that the Site Quality Monitoring feature found on your project. If this number is greater than 0, you may click it to open that past check’s result report. The result report will open.
Searched Pages The number of pages that the Site Quality Monitoring feature crawled on your project.
Status The status of the monitoring crawl.

Click Stop all crawls that are currently running. to stop a manual crawl in progress.

The Site Quality Monitoring feature will email the addresses you have added to the Configure alert emails tab when it finishes the crawl.

Result report

The result report displays the information from past crawls. This report is divided into two sections, where number refers to the number of pages in each category:

Section Description
Found Page (number) This section contains a list of each page that failed the crawler’s search. The list displays the affected domain, the URL that produced the failure, and HTTP code that the page returned, if any.
Examined Pages (number) This section contains a list of each page that did not fail the crawler’s search. The list displays the URL of each page the crawler checked.

Additional Documentation