Dreamhost Has A Slow Network

The issues I spoke about in Poor Site Performance probably aren’t related to any content within this site, as I had previously thought. It’s more likely related to the shabby state of Dreamhosts network.

I am now blaming the Dreamhost network for the poor performance of this site for the following reasons.

1. Intermittence:
This one is simple. Sometimes single posts will load in a snap. Other times they take up to 30 seconds to display. Same with the index page, archives, about, and contact pages. The contact and about pages are relatively void of content. There’s really no reason for slowness or lag when loading those pages, there’s not much info to display there.

2. Other Pages: Other pages at the longren.org domain also load slow at times. Take Mint for example. The Mint dashboard fails to load or loads very slowly when this blog is also responding very slowly. It takes up to a minute to refresh some very basic peppers from within Mint.

3. MySQL Is Usually OK: If I’m not mistaken, Dreamhost has it’s MySQL databases on separate servers. In other words, a machine that serves http requests doesn’t server MySQL data at the same time, another entirely separate box would handle the MySQL responses. My thought here is that Dreamhost might have their MySQL serving machines on a different network entirely, one that might not be having issues the rest of their network is experiencing. I could be way off here though, entirely speculation.

The Dreamhost Status site has been unavailable for most of the day. There’s normally a blog hosted there. Currently, there’s simply a few paragraphs of text explaining the current situation:

Network downtime Monday Night (09/11)
Monday night, we will upgrading our core networking equipment, which will result in some downtime of all services lasting approximately 30-45 minutes.
We’re expecting this to put an end to the network problems that were created by the power outages about a month ago..
– Sep. 8, 2006 3:30 p.m.

Oops! Temporary Status Site
We had a little goof and this will be our status site until the other machine can be resuscitated.
– Sep. 8, 2006 11:40 a.m.

So, as you can see, they’re relating this to the downtime experienced a month or so back when a generator caught fire. Hopefully the work they do on this coming Monday will fix the issues we’ve been having. It’s getting real old, real quick. Couldn’t happen at a worse time too, this site is starting to grow by great leaps and bounds. Hopefully growth won’t be affected by this poor performance.

UPDATE: This site was unreachable for about 12 hours lastnight/this morning. Here’s some new items that have been posted to dreamhoststatus.com:

Wilmington Failed Over
Wilmingtons second network card, which carries internet traffic, decided to give up the ghost. The server is on shiny new hardware now and websites are already starting to serve again. Our apologies for the downtime.
– Sep. 9, 2006 1:24 p.m.

Web Control Panel Issue Resolved
The network issues causing slowness for the web control panel for the web control panel have now been resolved. The network overall is more responsive and we will continue to make minor changes to keep everything working as well as possible. The major maintenance on Monday night will be essentially a complete replacement and upgrade of one of our core routers. In the meantime, we have been re-routing as much of our network traffic as possible through our other core router to improve overall performance.
– Sep. 9, 2006 2:48 a.m.

Web Control Panel Slowness
The network issues are also causing problems with the Web Control Panel for several users. We are looking into this and hope to have everything restored shortly.
– Sep. 8, 2006 8:56 p.m.

Network Problems Today
We have been experiencing Network Problems today. These problems are the same problems that have actually been happening since we first reported problems with our network. Unfortunately these problems have gotten worse today and are causing a majority of the downtime and slowness issues you are reporting today. These problems, and our attempts at fixing them, have been an ongoing effort. The maintenance Monday Night will be a big step towards resolving these problems completely. We are currently working on the network and all servers having problems and hope to improve the situation soon. Sorry about the downtime, we hope to have this all resolved soon.
– Sep. 8, 2006 5:45 p.m.

Dreamhost: Generators Caught Fire

OK, Dreamhost has had two outages due to power outages this week. Lots and lots of people are mad. Luckily this is the only site I host with Dreamhost, and this site doesn’t really matter so it doesn’t really upset me.

However, there are lots of people who try to run a business with Dreamhost. They either use Dreamhost to serve their company website or they resell web hosting. In either case, having two large periods of unexpected downtime in two weeks will have a negative impact on business.

On the Dreamhost Status blog, there was mention of a generator catching fire. The mention of the fire is no longer on the Dreamhost Status blog, but I copied the entire text of the post and quoted it at the Slackware Blog.

There has apparently been another power outage at the datacenter. One of the generators caught fire and they were taken offline. We’re waiting for more information and we’ll post here as soon as we hear anything.

I posted some stuff about the outage on the Slackware Blog, including some links to some comments made by some upset customers:

Lots of customers are pissed off, as they probably should be. Those links are to customers comments on the most recent post at the DreamHost Blog. I’m lucky enough that the only sites I have there are personal and don’t really matter. I feel for the people trying to run a business with Dreamhost. I had problems at work with Bluehost, but nothing like this.

One customer thinks Dreamhost should demand compensation from the datacenter in LA for the recent downtime:

I think Dreamhost should look at demanding compensation from the datacentre – and passing that onto affected customers.

A datacentre which says it provides “one of the most sophisticated, redundant power systems in all of Los Angeles” should never have this sort of problems. Sure – I can understand one power outage disaster (accidents happen), but TWO in the space of a week is just unacceptable.

Sounds like a good idea to me. There’s lots of interesting comments at the most recent post on the Dreamhost Blog. Most of the comments are pretty negative and a fair number suggest they’re going to drop Dreamhost and go elsewhere.

Dreamhost and World War III

The Dreamhost outage yesterday is now resolved. Four separate issues all came to light at roughly the same time, taking a good portion of Dreamhosts services offline. I personally assumed the Dreamhost outage was due to the poor connectivity between Level3 and Internap lastnight. See this image:
Level3 to Internap

It doesn’t sound like the Dreamhost issues had anything to do with their connectivity providers, but I’m glad they’re back up and running. I wonder who Dreamhost gets their connections from, probably a few different providers.

Is the Dreamhost outage the cause for high search rankings of “World War III” on Technorati? No, but it is pretty amusing that Dreamhost is one of the top technorati searches. It somehow doesn’t fit in with the rest of the top searches: “World War III”, “Lebanon”, “Israel”, and of course, “Bush”. Where’s “Hezbollah”?? heh.

Dreamhost Issues

This site hasn’t been accessible for a while this morning. Dreamhost appears to be having some issues with a fileserver that keeps crashing. All seems to be good now, but it sounds like they’ve been through this before. The got the box up once and thought it was stable, until it crashed again a short time later:

Once we got the machine up, fixed and serving files, everything seemed like it was ok, so we went back to making sure all content, data and websites were working normally.

Right about then, it crashed again! This time, however, it came back up correctly, so it didn’t take as long as it had previously.

See the Dreamhost Status Blog and the Unofficial Dreamhost Blog for details. Hopefully this is the end of the issues for now, but who knows.

Gmail For Your Domain

Hosted Gmail, or “Gmail for your domain”, is sweet. It provides the ability to use the Gmail interface with your domains email. So, I go to my hosted Gmail and I can send/receive email at the longren.org domain.

I had to change my MX records at DreamHost over to Google’s, naturally. I didn’t include a priority when switching them over though, stupid. I guess I was expecting a seperate form field for those. Didn’t cause much of a delay though.
dreamhostMX

I really like the idea of “Gmail for your domain.” It serves it’s purpose very well. I could go for a few more features in the administration area. The “Advanced Tools” in the picture is really a “tool” for editing your accounts in bulk. Under “Domain Settings” you’ll have the opportunity to set a custom logo, edit the sign-in page, and enable/disable chat. The only thing you can currently edit on the sign-in page is the color of the username/password boxes. Click for a bigger picture.
hostedGmail_lastTime
I expect a whole slew of new customizable settings as the beta progresses. I believe the chat feature was just added today, although standard Gmail has had that for some time now. I’d really like to be able to edit the title used on pages. It seems to just use the domain as the title right now.

I noticed they’ve got an “Account Plan” indicator on the administration dashboard. Google will probably start charging for this service once it’s out of beta, which could be in like 15 years. I mean, Gmail itself is still in beta. They might even charge per user, I’ve got “Up to 200 users for free.” I’d actually expect a pricing schema very similar to that seen today in web hosting. But probably cheaper.

Hopefully that screenshot isn’t against the TOS or anything. I’m not gonna bother reading that 50 page psycho-babble though. Maybe I’ll post more screenshots later.

On a somewhat related note, Garett Rogers (at ZDNet blogs) is speculating about “Google Music“. It’s not news though, just hardcore speculation.

Information gathered at the Digital Hollywood pre-show suggests Google is holding an invitation only event for twenty top music industry executives this Thursday. This may suggest that Google is moving forward plans to compete with Apple and others in the downloadable music space.

The topic of this meeting isn’t known, but we can speculate that Google may be showing an early stage product that is in the making or trying to form partnerships like they have with CBS.

UPDATE: Science Addiction has a nice review of Gmail for your domain, along with screenshots. Site may be slow as it got slashdotted earlier today, seems to be recovering now.