Here is the final update on my issues with Viruses and my Hosting company!
Virus issue has been resolved! Here is the latest news!
After a few days of us going crazy we have solved the recent issues. Here's what we figured out...
This was a 0 day cpanel exploit. Anyone in the world running cpanel could have been exploited.
They actually did the cpanel exploit about a month ago which explains what we thought at the time to be "bad cpanel updates." We thought this because sites weren't loading in IE and the fix was just changing a line in cpanel. At this point in time viruses weren't loading as far as we knew or heard so there was nothing to suggest anything different then a bad cpanel update.
We believe whoever did this was perfecting what they were about to launch and waiting for the right moment. They chose a few days ago to launch it in full force to exploit Microsoft's newly announced vml exploit. They used the exploit in cpanel to distribute trojans / viruses to target the vml exploit.
Here's what cpanel said once we showed them the exploit....
Quote:
Originally Posted by Cpanel
"This has been confirmed and patched. Running /scripts/upcp will fix the vulnerability in all builds."
We had our own patch we ran before the release of cpanel's, and as soon cpanel provided an update we ran their patch as well.
We had a few problems to figure out......
1. How was it happening and where was it coming from?
We could easily fix the problem but every time we did in minutes to an hour later it would come back. After hours of looking how this was happening we made little / no progress. We reached out to the web hosting community for help and soon had everyone helping us.
To name a few....
ThePlanet's security team, Layeredtech's security team, idefense.com, verisign, our best inhouse administrators and gurus, some server admin companies, and a customer of our's named Brad who helped build the architecture of paypal and ebay.com.
Brad had some contacts in symantec, trend, and Mcafee that he was able to contact on our behalf. We had everybody working on this. Our CTO Dave finally figured out what was causing it and cleaned it up at which point it has not happened since.
2. What was exploited and how?
We might have cleaned it up to fix the problem, but without knowing how they were exploiting our boxes they could easily do it again and again. One of our best admins Tim Greer solved this mystery today when he came across a cpanel root exploit that nobody knew about. He tested it and it was soon proven this cpanel root exploit is how the hackers had the power to do the redirects. As soon as we knew the function of cpanel that was being exploited we had help with the creation of a bandaid patch that was applied immediately.
At the same time this was going on I got on the phone calling everybody in the industry to get cpanel involved. I was able to reach cpanel's operations manager Dave who quickly came up with a patch that has now been released to the public. At this point we ran upcp which will prevent our boxes from being exploited this way again.
3. Where are we at now?
A lot of people that use Internet Explorer got viruses and will need to run a virus scan. I'd appreciate if some affected people could post the best way for scanning and removal. We do not have any evidence of anyone's passwords or personal information being taken, but to be safe it would not hurt for everyone to update their passwords to something complicated.
Hostgator's boxes have all been cleaned and the cpanel patch has been applied to avoid this from happening again. All other hosting companies that haven't applied this patch are going to get it installed automatically tonight. Many of them will remain exploited until they clean their boxes as we did.
The person or group that did this is very intelligent, and obviously knows how to plan a big attack. While we are protected from this threat we cannot predict what's to come for hostgator and the industry. Nobody can. No server is 100% secure. There could be a new 0 day exploit around the corner that takes out the entire internet. Anything is possible. We will continue to stay on top of security and do our best to provide the best possible hosting experience.
I realize our staff gave a lot of people wrong information. The truth is we weren't really sure what the problem was for a while, and we were actively working on it. There was a lot of trial and error involved. Had we simply just turned everyone off to avoid prorogation of the virus we would not have been able to peform the trial and error needed to trace it down to cpanel being exploited.
Our staff was responding to tickets saying they couldn't reproduce some of the issues, this was likely because we would fix it and respond back to the ticket at which point the time you read it the problem was back again.
4. Why were we targeted and by who?
We were most likely targeted due to our size and solid reputation. Since this exploit could have worked on anyone running cpanel it had nothing to do with how secure we were. We suspect it was done by someone in china as we have a small piece of information supporting this. I wish I could provide more specific information related to the exploit, but the less people know about it the less likely it will popup in a different variation.
We apologize to everyone for this issue, and if there was anything we could have done to avoid it or solve it quicker we would have done it. I would like to thank everyone that helped us in this crisis. I'm sure the situation could have been handled better, however we did the best that we would with the man power we had available to take calls, chats, and tickets around the clock. A tech usually takes 4 chats at a time while this was going on many had 20+
Thank you for understanding. This was devasating to us as well as anyone that had a website affected. We will do our best to help everyone recover from this.
__________________
DJSO Admin
|
Linear Mode
