Determine Your System Status


Last modified: November 4, 2021

Overview

Hosting providers and system administrators can use this document to determine whether their systems have been root-level compromised. We also recommend that you read our Tips to Make Your Server More Secure documentation.

Important:
We strongly recommend that you consult a qualified security specialist and/or your hosting provider if you believe your server has been compromised.

Rootkits

Rootkits allow a malicious user to gain undetected administrative access to the server. Every rootkit has at least two purposes:

  • Hides the attacker.
  • Grants access to the attacker.

Rootkits are difficult but not impossible to detect. Some rootkits (such as Ebury, NCOM, Shellbot, and XorDDoS) affect system libraries. These libraries run on the operating-system level, so they will (or should) have been installed via the yum or rpm commands.

Note:

These commands do not work on servers running the Ubuntu® operating system.

Check system libraries

If a library file is not owned by a package, this may indicate a possible root compromise. You can check whether a system library has been installed by a package with this command:

rpm -qf /path/to/libraryfile

For example, run the following command to check the package that owns the /lib64/libpw5.so file:

rpm -qf /lib64/libpw5.so

If the output shows that the library is not owned by any package, this may indicate a root compromise.

Check Size and Key ID of packages

If the output shows that the library is owned by a package, you should check the Size and Key ID values of the package that owns the library file and compare them to the Size and Key ID values of the same package on a known clean server. If the Size and Key ID values do not match, this may indicate a compromise. Use this command to check the Size and Key ID values of a package:

rpm -qil <package-name> | egrep 'Size|Key ID'
Note:

You must compare package versions on the same operating system. For example, you cannot compare packages between servers running the CentOS and CloudLinux™ operating system.

Check size of system library

You should also check the size of the library in question. On clean servers, the library file either does not exist or its size is 25 Kb or smaller. On servers compromised by a rootkit, the file’s size is larger than 25 Kb. You can check the size of the library file with this command:

ls -lh /path/to/libraryfile

Check timestamp on library

You can get an idea of when a rootkit was installed via the Change timestamp from the stat command’s output. For example, to check the timestamp on the /lib64/libpw5.so library file, run the following command:

stat /lib64/libpw5.so

Then check the Change line of the output:

1
2
3
4
5
6
7
8
  File: '/lib64/libpw5.so'
  Size: 16        	Blocks: 0          IO Block: 4096   symbolic link
Device: fc01h/64513d	Inode: 5072132     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-10-28 23:39:26.107888218 +0000
Modify: 2021-09-08 22:00:57.707609700 +0000
Change: 2021-09-08 22:00:57.707609700 +0000
 Birth: -

Check hash of library on VirusTotal

Run the sha256 hash of the library file through the VirusTotal website to check it against pre-existing scans of the same file.

For example, to generate the sha256 hash for the /lib64/libpw5.so file, run the following command:

sha256sum /lib64/libpw5.so

And make note of the sha256sum value in its output:

sha256sum: 970b49c16eebd558ac8984643f3763e76a52c9be4118f9e5830b8f5c406414fc

Then navigate to the VirusTotal website, using the following URL where 970b49c16eebd558ac8984643f3763e76a52c9be4118f9e5830b8f5c406414fc represents the sha256sum value:

https://www.virustotal.com/en/file/970b49c16eebd558ac8984643f3763e76a52c9be4118f9e5830b8f5c406414fc/analysis/

The website’s output displays a list of antivirus systems that have detected this file as a compromise.

Important:
  • Replace 970b49c16eebd558ac8984643f3763e76a52c9be4118f9e5830b8f5c406414fc with the actual hash from the sha256sum command.
  • The VirusTotal website does not contain results from every potentially compromised file. However, if your server is reported as root-compromised and the report includes a link to virustotal.com, there is a very strong chance that the server is indeed root-compromised.

Run the CSI script

cPanel Security Investigator (CSI) is a script that provides a variety of functions to assist with the investigation of both root- and user-level compromises. By default, it scans for rootkits or root-level compromises, but it can also perform user-level scans.

To perform a root-level scan, run the following command:

/usr/local/cpanel/3rdparty/bin/perl <(curl -s https://raw.githubusercontent.com/cPanelTechs/CSI/master/csi.pl)

Additional Documentation