Restoring a Hacked Linux Server

This is a great post at MDLog. Marius goes into great detail on how you should respond when you know a linux server has been compromised.

It’s an interesting read if you’ve ever had the opportunity to be the administrator of a linux server. Probably interesting if you haven’t had that opportunity. None of my systems have ever been hacked, and I admin about 5 linux servers both at home and work. I’ve always sort of had a plan of attack though in the event it did happen. And, I’m happy to say, it would look very similar to Marius’s post.


Dreamhost and World War III

The Dreamhost outage yesterday is now resolved. Four separate issues all came to light at roughly the same time, taking a good portion of Dreamhosts services offline. I personally assumed the Dreamhost outage was due to the poor connectivity between Level3 and Internap lastnight. See this image:
Level3 to Internap

It doesn’t sound like the Dreamhost issues had anything to do with their connectivity providers, but I’m glad they’re back up and running. I wonder who Dreamhost gets their connections from, probably a few different providers.

Is the Dreamhost outage the cause for high search rankings of “World War III” on Technorati? No, but it is pretty amusing that Dreamhost is one of the top technorati searches. It somehow doesn’t fit in with the rest of the top searches: “World War III”, “Lebanon”, “Israel”, and of course, “Bush”. Where’s “Hezbollah”?? heh.


Dreamhost Issues

This site hasn’t been accessible for a while this morning. Dreamhost appears to be having some issues with a fileserver that keeps crashing. All seems to be good now, but it sounds like they’ve been through this before. The got the box up once and thought it was stable, until it crashed again a short time later:

Once we got the machine up, fixed and serving files, everything seemed like it was ok, so we went back to making sure all content, data and websites were working normally.

Right about then, it crashed again! This time, however, it came back up correctly, so it didn’t take as long as it had previously.

See the Dreamhost Status Blog and the Unofficial Dreamhost Blog for details. Hopefully this is the end of the issues for now, but who knows.


What’s Wrong With OpenDNS?

OpenDNS is surely going to prove to be a useful tool for those not intimately familiar with the internet. OpenDNS, provides some unique functionality compared with other DNS servers in that it detects typos and prevents phishing. For example, say you type http://www.longren.og into your browser. That URL obviously doesn’t exist, notice the .og at the end? OpenDNS will recognize the typo and will redirect the user to

Smart huh? Yes, but it could have it’s drawbacks. This post highlights what could be a potential security risk in OpenDNS. It has to deal with intrusion detection systems (IDS) not realizing which URL is actually being requested. That post uses the mod_speling apache httpd module as an example.

If I send a request for indexh.tml, mod_speling detects the mistake and will serve back index.html. The problem is any security products like an IDS/IPS won’t have this intelligence to try and “fix” the request before they analyze it. The IDS/IPS simply sees and logs a request for indexh.tml Modspelling, like this feature in OpenDNS, allows an attacker to side step the attack signatures on a IDS/IPS to exploit a site because the web server will “fix” the attack once it reaches its target.

I disagree with the logic behind the authors claims. Why? Simply because I have a feeling OpenDNS was built with that taken into consideration. I’m betting there’s some sort of database internally that lets every piece of the network know exactly what is being served when a typo is detected. Everything from the IDS boxes to the DNS servers themselves. Maybe I totally missed the point of what that post was trying to get across.

Another thing OpenDNS should work on ASAP is transparency. I’d really like to know the false positive rate on phishing sites. How many legitimate sites get flagged as a phishing site? A publicly available reporting system would also be nice. Something to show DNS changes in particular would be nice for helping to maintain the integrity of the database.

But, I’m sure these questions will be answered in the near future, after all, today is the company’s first day with exposure to the “public”. There’s already mention of a new feature on the most recent post at the OpenDNS blog.

One important feature which is not yet available, but will be soon, is self-service control over the DNS settings. Ryan’s article, understandably, doesn’t mention this capability, since it’s not yet live.

The point? We’re going to put more control in your hands, so if you want to turn off features like typo correction or phishing prevention, you’ll be able to. Account management is the top priority now, to help demonstrate the power of control over your DNS. We think transparency and control will show you (not just tell) that we’re making the right choices.

Ryan’s article is of course the article that was in Wired this morning. See, they’re already taking steps to provide more transparency, hopefully it will continue.

Harper Reed is also a bit skiddish with OpenDNS still, like me. I think OpenDNS has great intentions though, so I’m not too worried. Founder of OpenDNS, David Ulevitch, already has a pretty outstanding reputation in the internet community, probably due mostly to the success of EveryDNS. OpenDNS is out to do good on the internet, just like EveryDNS. That doesn’t mean they can’t do harm, as we saw with Blue Security.

I’m pretty sold on OpenDNS overall. I put their DNS servers in my DHCP server config tonight after I got home from work. And the Nevada office as well as a couple servers in Ankeny are using OpenDNS now too.