Determine Your System's Status

This document was created on February 25, 2013.

Overview

Regretfully, cPanel support department has experienced a security issue. Two types of compromises have been detected. One, which involves compromised RPMs in the OpenSSH binaries. The second type, involves libkeyutils. In both cases, files contained within the directories or binaries were "trojaned." We highly encourage system administrators to read this document to determine the status of their system. If you experience any issues while you perform these commands, please contact Tech Support for assistance.

Compromised RPMs

We have determined that some systems were compromised with "trojaned" OpenSSH binaries. The OpenSSH binaries appears to contain the Ebury trojan.
note Note: The network traffic created by the "trojaned" OpenSSH binaries and the libkeyutils are similar.

In regards to CentOS and RedHat systems, we have determined that the sshd, ssh, ssh-keygen, and ssh-askpass binaries all appear to contain trojan code. This code is used to collect authentication credentials for both inbound and outbound connections. We believe that the SSH keys generated by these binaries were also captured.

OpenSSH RPMs Trojans Characteristics

The most distinctive features of the "trojaned" OpenSSH RPMs are:

The RPMs contain 3-digit release numbers to prevent updates from overwriting them.

The following example indicates a desired output:

[user@host ~]$ rpm -qa | grep -i ssh
[...]
openssh-clients-5.3p1-81.el6_3.x86_64
openssh-server-5.3p1-81.el6_3.x86_64
openssh-5.3p1-81.el6_3.x86_64

The following 3 examples indicate a compromised system:

  • In the following example, 730 is the release number:
    openssh-askpass-4.3p2-730.el5_67.3

  • In the following example, 721 is the release number:
    [user@host ~]$ rpm -qa | grep -i ssh
    [...]
    openssh-4.3p2-721.el5_61.65
    openssh-clients-4.3p2-721.el5_61.65
    openssh-server-4.3p2-721.el5_61.65

  • In the following example, 209 is the release number:
    openssh-5.3p1-209.el6_10.41.x86_64
    openssh-clients-5.3p1-209.el6_10.41.x86_64
    openssh-server-5.3p1-209.el6_10.41.x86_64

The change logs for RPMs do not contain any entries for the installed release.

jd@goat:~/$ rpm -q --changelog openssh-5.3p1-209.el6 | head -10
* Thu Aug 12 2010 Jan F. Chadima <jchadima@redhat.com> - 5.5p1-20
- set correct socket name length in abstract socket (#621691)

The RPMs are not signed.

The following output shows an empty signature:

jd@goat:~/$ rpm -q -i openssh-clients
Name        : openssh-clients
...
Signature   : (none)
...

The system YUM logs have no record of the RPMs being installed.

note Note: On a typical installation of cPanel & WHM, there are several RPMs that are not signed and do not contain complete change logs. Besides OpenSSH, an unsigned RPM does not indicate that a system was compromised.

Important information about your server's architecture

Before you run these commands, note that you will receive different information depending on the architecture of your server.

note 32-bit systems use lib. 64-bit systems use /lib64. /lib64 may also have a lib directory that contains 32-bit libraries for compatibility purposes. If you are not sure of your server's architecture, you can use the arch command:

[user@host ~]$ arch

An output of x86_64 means you use a 64-bit server. An output of i386, i686, or something similar means you use a 32-bit server.

How to Check Your System for the trojan in libkeyutils

We recommend that you run each of the following commands to fully determine the status of your server. Even if you receive the desired output, we recommend that you still run the other commands.

Command 1

You should verify the keyutils-libs package.

Ideally, a change did not take place in your system without your consent. If a change did not take place, you should receive a blank output.

The following output indicates your system was not comprised:

[user@host ~]$ rpm -V keyutils-libs
[user@host ~]$ 

However, the following output indicates something changed. In this case, you should investigate further:

 [user@host ~]$ rpm -V keyutils-libs
....L...    /lib64/libkeyutils.so.1 

Command 2

You should verify which file is linked to the libkeyutils.so.1 file.

Run the following command to see which file is linked to:

[user@host]$ ls -l /lib64/libkeyutils.so.1

The following example shows the desired output:

lrwxrwxrwx 1 root root 18 Feb 20 12:15 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3*

If the /lib64/libkeyutils.so.1 file points to a file with one of the following names, then the server may be compromised:

  • libkeyutils.so.1.9
  • libkeyutils.so.1.3.2
  • libkeyutils-1.2.so.2

Command 3

You should check the strings for all libkeyutils.* libraries for items that relate to networking.

The default libkeyutils.so.1.3 file should not contain the following strings:

  • connect
  • socket
  • inet_ntoa
  • gethostbyname

The following output indicates that you server was not compromised:

[user@host ~]$ strings /lib64/libkeyutils.so.1.3 | egrep 'connect|socket|inet_ntoa|gethostbyname'
[user@host ~]$

The following output indicates that your server may be compromised:

[user@host ~]$ strings /lib64/libkeyutils.so.1.9 | egrep 'connect|socket|inet_ntoa|gethostbyname'
gethostbyname
socket
inet_ntoa
connect

Command 4

You should check to see if any sshd processes currently use shared memory segments.

Run the following command:

[root@host ~]# ipcs -mp

The following output indicates that there are no programs that use shared memory segments:

[root@host ~]# ipcs -mp

------ Shared Memory Creator/Last-op --------
shmid      owner      cpid       lpid      

The following output indicates that there are programs that use shared memory segments:

------ Shared Memory Creator/Last-op --------
shmid      owner      cpid       lpid
1769472    root       1975       1975
2129921    root       2931       2940
1736706    root       1965       1965
2162691    root       2931       2940
2195460    root       2931       2940
2228229    postgres   4011       6813

If there are program that use shared memory segments, you can use the ps command to check if any of the items in the cpid and lpid columns correspond to the sshd processes:

[user@host ~]$ ps aux | grep 1975
root     1975  0.0  0.0  64080  1172 ?        Ss   Feb17
0:00 /usr/sbin/sshd

note Note: While is it not typical of sshd to use shared memory segments, you should investigate further to see if this is an indication of a compromise.

Command 5

You should use a network sniffer, such as tcpdump, to monitor outbound UDP port 53 traffic while you log into the shell.

Run the following command:

[root@host ~]# tcpdump -Annvvs 1500 -i any udp and dst port 53

It is normal to see DNS traffic between your server and your local resolvers from /etc/resolv.conf (and possibly to other servers). However, it is not normal to see traffic that contains odd, alphanumeric strings, followed by the IP address that you logged in from.

The following output indicates a compromise.
note Note: The asterisks indicate important items to notice.

07:54:48.233159 IP (tos 0x0, ttl  64, id 31281, offset 0, flags [DF],
proto: UDP (17), length: 91) **1.2.3.4.43089** > **72.156.139.154.53**:
[bad udp cksum d7a9!]  4619+ A?
**6196g8f43a4facd3561de4gec736fb.5.5.5.5**. (63)

The example of the packet above shows that the cPanel server at 1.2.3.4 sends a UDP packet on port 53 to the host at 72.156.139.154. You should notice the following false query from the example above:

6196g8f43a4facd3561de4gec736fb.5.5.5.5

The string 6196g8f43a4facd3561de4gec736fb is an encoded password, and 5.5.5.5 is the IP address that was used to log into the cPanel server. This indicates that the malicious host at 72.156.139.154 has learned the following information:

  • The IP address that was used to log into the server, and
  • The password that was used to log in

note Note: You may see 2 of these packets in quick succession: one packet contains the username used to log in, while the other packet contains the password. The content of both packets will be extremely similar.

Command 6

You should verify that each library that sshd is linked against belongs to a known package.

You can use the ldd command to see which libraries sshd links against:

[user@host ~]$ ldd /usr/sbin/sshd
   linux-vdso.so.1 =>  (0x00007fffb5dfd000)
   libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00002aaabc550000)
   libwrap.so.0 => /lib64/libwrap.so.0 (0x00002aaabc753000)
[...]

The items to the right of the arrow are the libraries that sshd links to. In this example, the libraries are:

  • libfipscheck.so.1
  • libwrap.so.0

You should verify that these libraries belong to a valid package. Run the following command with the libraries included:

[user@host ~]$ rpm -qf /lib64/libfipscheck.so.1
fipscheck-lib-1.2.0-7.el6.x86_64

[user@host ~]$ rpm -qf /lib64/libwrap.so.0
tcp_wrappers-libs-7.6-57.el6.x86_64

Topic revision: r8 - 27 Feb 2013 - 17:36:48 - Main.JenniferDoubrava