By now most of you know that we were part of a mass hack yesterday afternoon approximately 4:20PM. While I was driving home from work, Dianna told me the forum was down. I said I will send an email to our host if I can't get to the control panel when I get home. Once I got here, she also showed me one of the messages that someone had got when trying to connect. When I saw that I had a bad feeling about the whole mess. Anyway I sent an email off to our host and got a response within a few minutes. Here is what the response said:
Your site was hacked and the compromise may have gotten into the server. We are doing an investigation at this time. Once we are satisfied that there is no threat to our systems we will open the server to the public network again.
After what Dianna had shown me I wasn't surprised. I was sure that they didn't get in through our account though as I keep the forum up to date and we (the SMF team) address any known vulnerabilities in the software rather quickly. Anyway there wasn't much I could do but wait till they reconnected the server to the internet so I could see how much damage was done.
About 10 minutes later I got this email from our host:
This is a mass-notification to let everyone know that Pegasus has been disconnected from the internet momentarily. An account on the server was hacked, and this allowed the attack access to certain system utilities. These utilities were exploited to change some "index" files to simply say "Hacked by...".
We are right now making sure the server is clean.
No personal information was compromised, as it is NOT stored on the production servers. All custom files appear to be in tact, and no data was lost other than the index files.
We estimate the server to be back up within an hour or so.
We are also preparing a new system, with a fresh installation of UNIX and all utilities. We are going to move the accounts to the new server AFTER inspecting every single file. This will be done late at night and we will be re-using the same IP addresses so it will not cause any inconveniences to you, our client.
UPDATE: Current indications are that there is a vulnarability in Tinyportal, a well-used SMF MOD that allowed access... Index pages with only this text: Hacked by Ma3sTr0~Dz are symptomatic of this hacker.
Great!
Shortly after that email I got this one:
Pegasus is back up.
That is to say, it is connected to the network, and Apache, and MySQL processes are running.
cPanel is currently not running, but FTP is.
If your site is displaying "by Ma3sTr0-Dz", you will need to re-upload your "index.php", or "index.html", or "index.htm" (whichever you were using). No other files were harmed, and no database content, etc. was removed.
Again: NO FILES other than the INDEX files were changed (caps added for emphasis)
For those running phpBB, SMF, etc. it is best to download a fresh index.php file. If you require help with this, please reply to this email to raise a ticket and we will gladly assist you.
We have found the cause of the attack. Our servers are "lockboxed", that is, each account can not go into any other account's files. However, we have one exception, and that is through the accelerator systems (APC, eaccelerator) which scripts like SMF use to work quicker. These accelerators compile PHP scripts and store them in /tmp. When one of the scripts is compromised, it is also compiled to /tmp and from there can be run on all accounts. We have disabled accelerators to prevent this.
We also found the account that was compromised - using an outdated SMF and Tinyportal installation. Tinyportal was used to upload the initial rogue payload. Because of this, we are asking all clients to update their sites as soon as possible. If you are using Tinyportal, we encourage you to find an alternative, or at the very least upgrade it to the latest version.
Right now we are preparing a server to use in place of Pegasus ( or rather, new hardware to use Pegasus with). This should be ready by nightfall and everything should be back to normal at the latest by tomorrow morning. Note, that this does not mean there wil be downtime during this entire period, simply that services such as cPanel should be back by morning at the latest.
As an added bonus, the new system will have 12GB of RAM, and 8 Woodcrest 3.0Ghz CPUs (roughly a 4x performance increase).
We will send out another mass-mailing to all clients on Pegasus once we are ready to place clients on the new hardware.
An outdated SMF and TinyPortal? That's why we always insist that everyone update to the most recent release of the software. Some folks have modified the files so much that they refuse to do it and this is what can happen. Oh and if you hadn't figured it out our host names all the servers.
We are not using TinyPortal!
Now is when my work began. At last count I had to replace approximately 165 files. In most cases the hacker just replaced every file that began with index.* with the hacked file. He also replaced every file that began with main*.* and home*.* with a hacked file. For some strange reason he also deleted files that began with log*.*. Maybe that was to cover his tracks!
While I was trying to get the files replaced everyone was trying to logon and in some cases did even though lots of the buttons were gone. That created 1358 pages of errors in the log. Once I finally got to where I could log in and get to the admin panel, I placed the forum in maintenance mode and continued to replace files. It's a good thing I had a backup to work with.
This is the email that came in about 2:30AM this morning from our host:
Most accounts are now back to normal. We have a few left that still require manual intervention which we are working on.
Because of this, we have pushed out the move of the data from the old Pegasus to the new one, to tomorrow night starting at 10:00 PM CDT.
This will allow everyone to warn their memberships of what's going on. We do not anticipate any downtime, but there may be some flakiness / slow loading going on while we are in the process.
We are right now capturing a snapshot of all accounts, which will be stored off-server, and in fact, out of datacenter.
We have changed all system passwords, and blocked all exterior access to the old Pegasus as a precaution. We will continually monitor it until such time as everyone is safely on the new hardware setup.
For those of you that don't know, the new Pegasus' specs:
4 x Intel Xeon-Woodcrest 5148-DualCore-LV [2.33GHz]
12GB ECC FB-DIMM DDR2
1TB RAID10 array
1Tbit/s network uplink
And all the usual goodies
As usual, if you notice anything out of the ordinary, please let us know.
As a security precaution cPanel has been turned off until tomorrow evening. If you require access to any cPanel functions, feel free to raise a ticket and we'd be happy to assist you. Once on the new system, full cPanel access, along with a few new features, will be available immediately.
We'd also like to take this time to thank everyone for their patience. It's been a pleasure working with you all today - as we know these kinds of things can get on one's nerves.
So that is just about it. Tonight after 10:00 EST we should be moved to the new hardware. If you experience some times after that where you can't login, it is because the move is taking place.
I can't say anything but good words about our host. If any of you ever decide to put up a website you should definitely consider MonteCarloHosting. They are the best!
