Dreamhost is recovering from yet another network outage. Apparently a switch had issues and they had to take a few servers off that switch. Supposedly the servers were moved over to another temporary switch so they’d have network access, but I haven’t had access to longren.org for at least 2 hours now.
We have moved the servers in the affected rack to knew switches in other racks temporarily while we get a new switch in place. We are working on recovery efforts now on servers which may be down. The list of affected servers is:
Arrow, mel, caesar, herod, alondra, overland, rossmore, oxnards, cerritos, selma, nala, demeter, jarvis. This is a mix of 3 MySQL servers and the rest webservers.
Longren.org is hosted on oxnard unfortunately, that box seems to have a lot of issues, even when the Dreamhost network is functioning properly. They’ve now managed to replace the problematic switch and are working on bringing servers back online.
Sorry about no timestamp before, it’s 12:36 Pacific time (-0800? I never could keep track of daylight savings time.)
We have completely replaced the switch and are working on getting the servers back online. All networking cables are back in their regularly scheduled switch ports.
I always get a kick out of the angry comments that show up on the Dreamhost Status blog whenever a fairly widespread problem occurs. Of course, Dreamhost hasn’t had great reliability in the last few months. I feel bad for those trying to run a business with Dreamhost.
UPDATE: Apparently there’s still trouble. Now Dreamhost is saying the servers affected are still behaving abnormally. This box, oxnard, doesn’t appear to be affected though, for once! They’re still updating that same post at the Dreamhost Status blog:
Wierdness is afoot. The servers are still exhibiting the same problems, we have even moved two of them into our new datacenter and it’s the same deal. It’s only specific servers in half of one of our racks.
UPDATE 2 @ 8:52 PM CST: Dreamhost now says all their problems are over. Which they do appear to be. I believe the K2 site is hosted at Dreamhost, I noticed it was down for a while but has since been revived. Here’s the update from the Dreamhost Status blog:
As of 1700 Pacific time (GMT-0800 or so, 5PM), we think the issues with this rack are behind us. We moved some machines to new hardware and this seems to have fixed the problems. We will be looking in to this more on monday, and doing some extensive testing on that hardware to make sure it was the root cause of the problem.
Hopefully this was just a minor fluke and we don’t see any more problems. Like I said before, Dreamhost has been less than stable these last couple months. Honestly though, I have noticed this site loads much quicker since Dreamhost fought with their network last time. Still, there’s a lot of recent posts categorized as “system outages” at the Dreamhost Status blog.