How a server gets pwnt

– Before I begin –

Be warned - a lame wanna-be-sys-admin story with potentially cringy content is to follow. Read at your own risk.

While I do realize that the text below covers the very basics of the matter I am delving into and that a lot of the professionals here are way over this, some of the unexperienced enthusiasts might find the information provided here useful. In no way do I approve the misuse of this article for personal gains by breaking the hacker ethics (“no theft, vandalism, or breach of confidentiality”). I would be more than happy, however, for you to utilize the information on your own website or server. Remember: Be aware, connect with care.

– Background –

In my day-to-day work at a British web hosting corporation, there isn’t really much interesting going on - normally all the hassle is about changing passwords, installing content management systems (mostly Wordpress) on clients' behalf, explaining the exact same thing over and over again (did I ever tell you the definition of insanity?) and covering up the mess caused by the Infra- and Sysops overnight. Occasionally, there are some breached websites (once again, Wordpress, I’m looking at you), but normally no large-scale operations take place.

To give you a better idea of the hierarchy in the company, there are five different departments involved in most of the cases. At the very beginning, there are the phone support and live chat teams which handle the first wave of customers and inquiries. Then there am I and a handful of colleagues at the second tier who handle escalations and technical queries that do not require messing with server configuration. If there’s something wrong with the servers, our system administrators join the party and finally there is the infrastructure team which has physical access to the data center and performs the most complicated server-related tasks. As for the hosting environment deployed on the server, it is quite standard: CentOS, Apache, MySQL, PHP, CloudLinux with CageFS for selectable PHP versions (we switched to MultiPHP on newer servers) and a cPanel / WHM bundle.

Last Monday I came across a rather frustrated customer - a reseller of ours - who blamed our servers for not being secure enough and claimed that all accounts under his reseller, a total of roughly 90 accounts, were compromised. At first, I thought that was the regular blabbering of a client who tried to make a point and get away with a period of free hosting or some other compensation. A quick look into the accounts however proved me wrong - the majority of the websites displayed symptoms of MySQL injections - what was most peculiar, though, was that all websites (all of them built on Wordpress) had the exact same content injected - some base-64-encoded gibberish. The websites, however, were not completely defaced and only had the additional encoded content displayed in their title. This could only happen if:

a) Somebody had root MySQL access;
b) All websites offered the same vulnerabilities (in Wordpress this normally means that they are using the same outdated or exposed plugins and themes).

I ruled the former out since root access to the server is only possible after obtaining the password from a system available only through a VPN. Furthermore, the root password was only visible for our system administrators and higher tiers of support, which means that just a selected few of people could access the password. The latter was equally impossible due to the simple fact that the odds of more than 50 websites, each serving different purposes and being maintained by different people, to be built the exact same way were quite slim. This brought me to:

c) Something mean is going on.

– It’s time to panic –

Moments after reaching the aforementioned conclusion I found myself smashing keys as I was rushing to inform the system administrators via our internal messaging system. I hoped that they were already aware of the situation or, since they had root access to the server, could do something about it. Briefly, this is what followed:

Me: There is a bunch of compromised websites on server X - all share the same symptoms;
Sysop: Are they running the same CMS?
Me: Yeah, Wordpress
Sysop: There is your reason ;p
Me (after a brief pause): Isn’t there anything we could do about it?
Sysop: Nah, it’s up to the customers to keep their websites up to date.

After this reply, I realized that I should get my hands dirty myself and practice some typing (sudo all the way). With my limited server access, I had to do things the clumsy way - so I listed the websites on the server and started opening them one by one in order to find out anything that could give away how the infection occurred. Out of the blue, there it was - a website which didn’t have encoded content injected but was defaced altogether and displayed some typical political-based crackers' slogans and related images. I changed the shell to this user because of my own limited access, navigated to the public_html folder and listed the content of the directory. It mostly consisted of the Wordpress core files and directories (e.g. index.php, wp-config.php, wp-admin, etc.) with the exception of a single directory whose name was simply a bunch of keys pressed together. After entering it, I listed the content once again and was quickly swarmed by the output. In desperate need of more information, I listed the content again, this time using ls -lah. The following appeared on screen:

lrwxrwxrwx 1 thisuser thisuser 5 *date* justuser-WordPress.txt -> /home/justuser/public_html/wp-config.php lrwxrwxrwx 1 thisuser thisuser 5 *date* justuser-Magento.txt -> /home/justuser/public_html/app/etc/local.xml lrwxrwxrwx 1 thisuser thisuser 5 *date* justuser-Joomla.txt -> /home/justuser/public_html/configuration.php lrwxrwxrwx 1 thisuser thisuser 5 *date* anotheruser-WordPress.txt -> /home/anotheruser/public_html/wp-config.php (...)

For those who are not familiar what the above means, the attacker managed to gain access to either the cPanel account or the website and created symbolic links (shortcuts) to potentially existing configuration files of a couple of pre-defined content management systems, relative to the public_html direcotry. This means that almost anyone who had an installation of Wordpress, Joomla, Magento or a handful of other CMSs directly within the public_html directories of their accounts got owned. Almost - more on this later. In order to mitigate the immediate issue and prevent further damage from being inflicted, I ran a script that is present on all of our servers that allows the quick suspension of an account and I also made sure that there aren’t any .htaccess files that would allow the attacker to regain access to the account:

find . -name .htaccess -exec mv {} .htaccess.disabled \;

Of course, that’s not the best way to go about this, but back then I hadn’t much time to think about it. With the attacker intercepted for the moment being, I turned my attention to how this actually happened.

– The philosophy behind –

The first step towards achieving this is, of course, gaining access to the first account. The following two possibilities are plausible:

a) A compromised Wordpress instance to which a PHP-based shell was uploaded.

Wordpress is known as a wide-spread and common content management system because of three main reasons - it is easy to maintain, easy to customize and easy to configure. There is a great deal of websites running on Wordpress - in fact, every fourth website on the Internet has been built on Wordpress. There are many plugins and themes that could be used to extend the base functionality of the CMS and improve its design with minimum web skills required. These extensions, however, come from many different authors and developers across the world and do not pass any sort of quality tests before being submitted to the market. This potentially results in bad code being added to your website which also opens up a lot of vulnerabilities. Discovering these is in fact a piece of cake, given that the website owner doesn’t care much about security. There is the WPScan tool which could list vulnerable themes on plugins on a poorly secured Wordpress installation within minutes or even seconds. The exploitation of these vulnerabilities then only requires half a brain and some of these vulnerabilities allow SQL injections which, in turn, allow a shell environment to be uploaded which gives full access to the attacker.

b) Malware on the customer’s device. Not going to go into detail here - I assume that you are already aware of keyloggers and malware in general.

The second step is to find out all usernames that are present on the server. This is not hard at all - it is in fact done by simply previewing the content of a file:

cat /etc/passwd

This file is specific to cPanel / WHM and basically contains all users on the server and their encrypted passwords. This file is available to anyone on the server because it is used for authentication - a combination of user and password are required at login, the user’s row is found with /etc/passwd, then the password entered by the user is encrypted with the same algorithm as the one used to encrypt the password in /etc/passwd and then the two strings are compared - if they match, the user is granted access by cPanel / WHM. Unfortunately, securing the file is not possible as this would break the authentication mechanism.

Therefore, all it takes is to upload a shell to an account that you had compromised (or directly access its shell given that you know the password) and build a script that takes the usernames from the /etc/passwd file, then creates symbolic links / shortcuts to potential configuration files within the public_html folder of the account. The script should only be a couple of lines long - I haven’t had the time to try writing my own yet, but I will post a reply here with the script as soon as I have one.

It wasn’t long until we were also contacted by a couple of customers who were unable to log in to their cPanel accounts. Upon investigating the access logs of the server, we found out that a single IP address had been logging in to individual accounts and changing their passwords before proceeding to the next account. While we are yet not certain how the attacker acquired the cPanel passwords, we have reasons to believe that a lot of customers use the same cPanel password as the database user password, which could be found within the content management system’s configuration file to which the attacker already had access.

As I explained earlier, however, not every user on the server was affected - tech-savvy customers who are aware of the potential security threats lurking on the Internet apply extra measures to secure their data, such as writing their own code, using paid plugins and themes, moving the core files of their content management systems to a sub-directory within public_html and changing the permissions of their files and especially the configuration file among other practices. As for the permissions, for Linux and similar operating systems, the permissions I could recommend you (although this may vary depending on the CMS that you are using) are 400 - this means that only you and the website would be able to read the file (and nothing else) while nobody else would be granted any kind of access to the file.

– Conclusion –

While this article concerns web hosting, the moral of the story applies for the entire Web. By using the Internet, you become exposed and you should take all necessary measures to protect not only yourself, but other people on the Internet as well.

There is really much more to be said regarding each of the aforementioned topics, but my intentions are to share experience as opposed to write a book. If anyone is interested in further information, you could reply to this topic or message me with specific questions.

– tl;dr –

If you have a basic website and intend to keep it secure:

[list]
[] Use complex passwords;
[] Keep your devices malware-free;
[] Double-check the origin and reliability of any pieces of code that you deploy on your website;
[] Keep any third-party applications (content management systems, extensions, etc.) up to date;
[] Apply secure permissions;
[] If using a CMS, take the time to read through its documentation and apply extra security measures. For example: Wordpress, Magento, Joomla or the official website of the CMS that you are using.
[/list]

Trying to break into your own website in order to ensure that the security mechanisms that you have applied are sufficient for your needs is not simply an option, but also a recommendation.