Site Quality Monitoring — Crawling
Last modified: October 16, 2024
Looking for this interface?
Your hosting provider can enable or disable this interface in WHM's Feature Manager interface (WHM >> Home >> Packages >> Feature Manager).
Overview
The Crawling tab contains monitoring checks that you can run yourself. These checks, known as crawls, are systemic tests of every monitored URL that you have added for your monitored website.
Dead links
The Dead links tab contains a crawler that checks whether the links on the first 500 pages of your monitored website that it encounters return data. For example, if you add a link to your monitored website that leads to an image you have taken off the Internet, that link will not return data. The crawler will register the link as dead.
Click number URLs to begin a crawl of your monitored website for missing images and files, where number represents the number of pages the feature will crawl.
This action will not reset the countdown bar.
When you begin a manual crawl for the first time, this tab will update to contain a Past Checks table. This table will contain the following columns for all dead link crawls you have performed on your monitored website:
| Column | Description |
|---|---|
| Date | The day that you ran the manual crawl. |
| Found Pages | The number of pages that the Site Quality Monitoring feature found on your monitored website. If this number is greater than 0, you may click it to open that past check’s result report. The result report will open. |
| Searched Pages | The number of pages that the Site Quality Monitoring feature crawled on your monitored website. |
| Status | The status of the monitoring crawl. |
Click Stop all crawls that are currently running. to stop a manual crawl in progress.
The Site Quality Monitoring feature will email the addresses you have added to the Configure alert emails tab when it finishes the crawl.
Missing images and files
The Missing images and files tab contains a crawler that tests the first 500 pages of your monitored website that it encounters to ensure that their JavaScript, CSS, and image files load. For example, if you try to display an image file that you have deleted on a page on your monitored website, the page will return a failure code to the crawler.
Click number URLs to begin a crawl of your monitored website for missing images and files, where number represents the number of pages the feature will crawl.
This action will not reset the countdown bar.
When you begin a manual crawl for the first time, this tab will update to contain a Past Checks table. This table will contain the following columns for all missing images and files crawls you have performed on your monitored website:
| Column | Description |
|---|---|
| Date | The day that you ran the manual crawl. |
| Found Pages | The number of pages that the Site Quality Monitoring feature found on your monitored website. If this number is greater than 0, you may click it to open that past check’s result report. The result report will open. |
| Searched Pages | The number of pages that the Site Quality Monitoring feature crawled on your monitored website. |
| Status | The status of the monitoring crawl. |
Click Stop all crawls that are currently running. to stop a manual crawl in progress.
The Site Quality Monitoring feature will email the addresses you have added to the Configure alert emails tab when it finishes the crawl.
Oversize files
The Oversize files tab contains a crawler that tests the first 500 pages of your monitored website that it encounters to ensure that all their JavaScript, CSS, and image files are each smaller than 1 MB. For example, if you add a CSS file that is 3 MB, it will produce a failure.
Click 500 URLs to begin a crawl of the first 500 pages of your monitored website for oversize files.
This action will not reset the countdown bar.
When you begin a manual crawl for the first time, this tab will update to contain a Past Checks table. This table will contain the following columns for all oversize file crawls you have performed on your monitored website:
| Column | Description |
|---|---|
| Date | The day that you ran the manual crawl. |
| Found Pages | The number of pages that the Site Quality Monitoring feature found on your monitored website. If this number is greater than 0, you may click it to open that past check’s result report. The result report will open. |
| Searched Pages | The number of pages that the Site Quality Monitoring feature crawled on your monitored website. |
| Status | The status of the monitoring crawl. |
Click Stop all crawls that are currently running. to stop a manual crawl in progress.
The Site Quality Monitoring feature will email the addresses you have added to the Configure alert emails tab when it finishes the crawl.
Result report
The result report displays the information from past crawls. This report is divided into two sections, where number refers to the number of pages in each category:
| Section | Description |
|---|---|
Found Page (number) |
This section contains a list of each page that failed the crawler’s search. The list displays the affected domain, the URL that produced the failure, and HTTP code that the page returned, if any. |
Examined Pages (number) |
This section contains a list of each page that did not fail the crawler’s search. The list displays the URL of each page the crawler checked. |