My horrible Dreamhost experience
Jan. 24th, 2008 10:36 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
gizaAs promised, here is a blog entry on Dreamhost.
I'm not much of a writer, so instead I'm going to include all of my trouble tickets that I had with them, and let the nature of the tickets (and their numbers) speak for themselves. I've removed a lot of the verbage from the tickets and instead boiled them down to the essence of each message. So you won't see the troubleshooting steps that I took, traceroutes, etc. I did make sure to be verbose when I filed each ticket, because I didn't want to give Dreamhost any excuse to try and blame me for the problem.
Now, I know I've done a lot of griping about Dreamhost here, so I'm going to put these all behind some LJ cut tags so you don't have to read through all of them if you don't want to.
Message from you (Sep 13th, 2007 - 12:55:21 / #2067677)
The load average on our webserver is 17.29
Reply from DreamHost (Sep 13th, 2007 - 14:15:33 / #13567581)
(Roberto V.)
"I am sorry for the inconvenience that this may have caused you."
Message from you (Sep 14th, 2007 - 14:10:43 / #2069613)
The load average on our webserver is 10.69
Reply from DreamHost (Sep 14th, 2007 - 14:37:34 / #13584394)
(Chih)
"I'm terribly sorry about this"
Message from you (Sep 26th, 2007 - 11:39:17 / #2088819)
Attempting to load any page times out. I cannot SSH in either.
Reply from DreamHost (Sep 26th, 2007 - 15:22:46 / #13768683)
("This was an announcement message sent due to a large support incident.")
"Sorry about the problems that you noticed today with downtime slowness and the overall inability to access and use your services, including mail and webmail. We were working on upgrading our network and unfortunately we hit a problem during the process. We were working on the backup router making it as powerful as our main router to help with some of the network issues we have been experiencing lately. Unfortunately this did affect our whole network."
Message from you (Sep 27th, 2007 - 08:24:40 / #2090334)
The load average is 73
Reply from DreamHost (Sep 27th, 2007 - 08:58:50 / #13779439)
(Mike J)
"I'm extremely sorry about this. What is happening right now is that the cluster you're on is having issues with a file server that hosts most of the user accounts. We have an admin running optimizations right now, and
more disks are going to be added to the cluster very shortly. This is causing high disk read/write times, and thus most processes are waiting on Disk IO and the load is going up on most of the machines on the cluster. This should be resolved within the next hour or two."
Message from you (Sep 29th, 2007 - 11:49:04 / #2094107)
HTTP times out, SSH times out after I enter in my password.
Dreamhost failed to respond to this ticket.
Message from you (Oct 14th, 2007 - 19:56:35 / #2118883)
The load average is 114.
Reply from DreamHost (Oct 14th, 2007 - 20:37:06 / #14057858)
(Christian)
"I'm _really_ sorry about the erratic load averages that you've been seeing as of late. The reason is that there is a storage filer on the cluster that is currently experiencing problems, and this is causing issues for customers that have their files stored on it."
Message from you (Oct 16th, 2007 - 06:44:11 / #2121477)
The load average is 647!
Reply from DreamHost (Oct 16th, 2007 - 07:49:21 / #14084138)
(Javier)
"Sorry about that, The server you are on had some issues which caused it to be Powercycled (restarted) your service is now back up and running."
[Wow. I'm sure glad they explained what "powercycled" means. I would have been completely lost without that tidbit!]
Reply from DreamHost (Oct 16th, 2007 - 12:35:57 / #14086622)
(Jaiver, in a reply to my reply asking if the load average was mostly from waiting on the fileserver)
"You'll find that went the load gets above 200 that the load is usually a NFS issue."
[So then a load average of 195 is perfectly normal? Thanks for clearing that up!]
Message from you (Dec 18th, 2007 - 08:21:03 / #2216338)
The load average is 691.
Reply from DreamHost (Dec 18th, 2007 - 09:31:41 / #15106777)
(Kacy)
Unfortunately, we encountered a network filer issue that brought the load on many servers to an extremely high level. However, our admins have cleared up that situation and the high loads are down now.
Message from you (Dec 20th, 2007 - 12:16:11 / #2220150)
The webserver is not running. Connections to port 80 and 443 are being refused.
Reply from DreamHost (Dec 20th, 2007 - 14:18:35 / #15142867)
(Message sent out to many people, not signed)
"We are terribly sorry for the site outages that have occured today"
[Long story short, they screwed up an upgrade to Apache]
Message from you (Jan 2nd, 2008 - 08:22:19 / #2233833)
Load average is 319
Reply from DreamHost (Jan 2nd, 2008 - 08:28:19 / #15309133)
(Scott)
"Please accept our apologies. We are currently addressing a load issue on your server."
[A "load issue"? You guys have a load subscription!]
Message from you (Jan 9th, 2008 - 08:13:48 / #2245041)
Why does this keep happening, anyway?
Reply from DreamHost (Jan 9th, 2008 - 14:42:50 / #15421930)
(Glen)
"It keeps happening since the offload process is so slow and can take weeks on a single file server due to amount of data that needs to be redundantly moved off of it via a 100Mb connection only."
[Weeks to move all of their data? And no GigE? Are you kidding me?]
Message from you (Jan 21st, 2008 - 06:34:05 / #2276307)
Page loads take as long as 60 seconds, but the load average is very reasonable today. We've been getting lots of HTTP 500 errors. I am also seeing lots of messages from FastCGI in our error.log file.
Reply from DreamHost (Jan 21st, 2008 - 17:44:58 / #15907985)
(Chris)
"I'm sorry for the errors you were experiencing with your site.
I did some checking and it appears that you are hitting our memory limit on shared hosts. I would recommend installing some kind of caching so the php engine doesn't need to work as hard every time a page of yours is
pulled up."
[Uh, I don't know how a cache is going to consume less memory but hey, I'll give it a try.
I looked through their Wiki for instructions on how to build and install a custom PHP CGI and the APC cache, then did so. Also note that they didn't bother to give me any actual numbers -- they just said "too much memory" without actually quantifying it.]
Message from you (Jan 22nd, 2008 - 06:39:30 / #2277922)
I'm still seeing HTTP 500 errors and watching my error.log fill up, despite getting APC working. What's up with that?
Reply from DreamHost (Jan 22nd, 2008 - 11:07:31 / #15923211)
(Greg)
"Everything on your site is loading very quickly for me and I'm not seeing any 500 or 404 errors. Maybe your caching system is working now?
The only other thing we can suggest is for you to move to a PS so you don't have the memory constraints of a shared server."
[By this point, I had just enough of Dreamhost, so I moved the site to another webhost. I guess Greg didn't notice. I love the attempt to try and upsell me, too.]
So, there we have it. A total of 11 separate issues in less than 5 months. This does not include other times I saw the load skyrocket but failed to report it because I was in the middle of something else. Taken separately, each of these incidents are something that can happen at even the best webhost. But, as established here, Dreamhost is nowhere near the "best". In fact, I'd say that they were one of the worst companies that I have ever done business with. To have this many issues in such a short time is simply amazing. To have a webserver throw a 404 on a valid URL is simply crazy. I have never seen that happen in Apache before. It defies logic.
Looking at their webpage now, they promise 500 Gigs of disk space and 5 TB of bandwidth for only $5.95 a month. Yet the reality here is that you will never ever come close to these limits -- chances are you will hit their memory limit (whatever THAT is, they never told me nor did they disclose it on their site) long before then. Does this sort of business practice qualify as fraud? I don't know. But is certainly is misleading to advertise numbers like that.
Dreamhost is pissing off customers left and right and cannot expect to keep up this pattern forever. This is evident just by reading the comments to posts on Dreamhoststatus.com. Heck, check out the comments on the entry about their billing system SNAFU if you want to see some anger. There are some seriously pissed off people on there. Sooner or later this bad service is going to come back to haunt them, and then things are really going to get interesting. Who knows what will happen? Massive layoffs? A scandal? Bankruptcy? Government investigation? No matter what, it will be interesting to watch!
I'm not much of a writer, so instead I'm going to include all of my trouble tickets that I had with them, and let the nature of the tickets (and their numbers) speak for themselves. I've removed a lot of the verbage from the tickets and instead boiled them down to the essence of each message. So you won't see the troubleshooting steps that I took, traceroutes, etc. I did make sure to be verbose when I filed each ticket, because I didn't want to give Dreamhost any excuse to try and blame me for the problem.
Now, I know I've done a lot of griping about Dreamhost here, so I'm going to put these all behind some LJ cut tags so you don't have to read through all of them if you don't want to.
Message from you (Sep 13th, 2007 - 12:55:21 / #2067677)
The load average on our webserver is 17.29
Reply from DreamHost (Sep 13th, 2007 - 14:15:33 / #13567581)
(Roberto V.)
"I am sorry for the inconvenience that this may have caused you."
Message from you (Sep 14th, 2007 - 14:10:43 / #2069613)
The load average on our webserver is 10.69
Reply from DreamHost (Sep 14th, 2007 - 14:37:34 / #13584394)
(Chih)
"I'm terribly sorry about this"
Message from you (Sep 26th, 2007 - 11:39:17 / #2088819)
Attempting to load any page times out. I cannot SSH in either.
Reply from DreamHost (Sep 26th, 2007 - 15:22:46 / #13768683)
("This was an announcement message sent due to a large support incident.")
"Sorry about the problems that you noticed today with downtime slowness and the overall inability to access and use your services, including mail and webmail. We were working on upgrading our network and unfortunately we hit a problem during the process. We were working on the backup router making it as powerful as our main router to help with some of the network issues we have been experiencing lately. Unfortunately this did affect our whole network."
Message from you (Sep 27th, 2007 - 08:24:40 / #2090334)
The load average is 73
Reply from DreamHost (Sep 27th, 2007 - 08:58:50 / #13779439)
(Mike J)
"I'm extremely sorry about this. What is happening right now is that the cluster you're on is having issues with a file server that hosts most of the user accounts. We have an admin running optimizations right now, and
more disks are going to be added to the cluster very shortly. This is causing high disk read/write times, and thus most processes are waiting on Disk IO and the load is going up on most of the machines on the cluster. This should be resolved within the next hour or two."
Message from you (Sep 29th, 2007 - 11:49:04 / #2094107)
HTTP times out, SSH times out after I enter in my password.
Dreamhost failed to respond to this ticket.
Message from you (Oct 14th, 2007 - 19:56:35 / #2118883)
The load average is 114.
Reply from DreamHost (Oct 14th, 2007 - 20:37:06 / #14057858)
(Christian)
"I'm _really_ sorry about the erratic load averages that you've been seeing as of late. The reason is that there is a storage filer on the cluster that is currently experiencing problems, and this is causing issues for customers that have their files stored on it."
Message from you (Oct 16th, 2007 - 06:44:11 / #2121477)
The load average is 647!
Reply from DreamHost (Oct 16th, 2007 - 07:49:21 / #14084138)
(Javier)
"Sorry about that, The server you are on had some issues which caused it to be Powercycled (restarted) your service is now back up and running."
[Wow. I'm sure glad they explained what "powercycled" means. I would have been completely lost without that tidbit!]
Reply from DreamHost (Oct 16th, 2007 - 12:35:57 / #14086622)
(Jaiver, in a reply to my reply asking if the load average was mostly from waiting on the fileserver)
"You'll find that went the load gets above 200 that the load is usually a NFS issue."
[So then a load average of 195 is perfectly normal? Thanks for clearing that up!]
Message from you (Dec 18th, 2007 - 08:21:03 / #2216338)
The load average is 691.
Reply from DreamHost (Dec 18th, 2007 - 09:31:41 / #15106777)
(Kacy)
Unfortunately, we encountered a network filer issue that brought the load on many servers to an extremely high level. However, our admins have cleared up that situation and the high loads are down now.
Message from you (Dec 20th, 2007 - 12:16:11 / #2220150)
The webserver is not running. Connections to port 80 and 443 are being refused.
Reply from DreamHost (Dec 20th, 2007 - 14:18:35 / #15142867)
(Message sent out to many people, not signed)
"We are terribly sorry for the site outages that have occured today"
[Long story short, they screwed up an upgrade to Apache]
Message from you (Jan 2nd, 2008 - 08:22:19 / #2233833)
Load average is 319
Reply from DreamHost (Jan 2nd, 2008 - 08:28:19 / #15309133)
(Scott)
"Please accept our apologies. We are currently addressing a load issue on your server."
[A "load issue"? You guys have a load subscription!]
Message from you (Jan 9th, 2008 - 08:13:48 / #2245041)
Why does this keep happening, anyway?
Reply from DreamHost (Jan 9th, 2008 - 14:42:50 / #15421930)
(Glen)
"It keeps happening since the offload process is so slow and can take weeks on a single file server due to amount of data that needs to be redundantly moved off of it via a 100Mb connection only."
[Weeks to move all of their data? And no GigE? Are you kidding me?]
Message from you (Jan 21st, 2008 - 06:34:05 / #2276307)
Page loads take as long as 60 seconds, but the load average is very reasonable today. We've been getting lots of HTTP 500 errors. I am also seeing lots of messages from FastCGI in our error.log file.
Reply from DreamHost (Jan 21st, 2008 - 17:44:58 / #15907985)
(Chris)
"I'm sorry for the errors you were experiencing with your site.
I did some checking and it appears that you are hitting our memory limit on shared hosts. I would recommend installing some kind of caching so the php engine doesn't need to work as hard every time a page of yours is
pulled up."
[Uh, I don't know how a cache is going to consume less memory but hey, I'll give it a try.
I looked through their Wiki for instructions on how to build and install a custom PHP CGI and the APC cache, then did so. Also note that they didn't bother to give me any actual numbers -- they just said "too much memory" without actually quantifying it.]
Message from you (Jan 22nd, 2008 - 06:39:30 / #2277922)
I'm still seeing HTTP 500 errors and watching my error.log fill up, despite getting APC working. What's up with that?
Reply from DreamHost (Jan 22nd, 2008 - 11:07:31 / #15923211)
(Greg)
"Everything on your site is loading very quickly for me and I'm not seeing any 500 or 404 errors. Maybe your caching system is working now?
The only other thing we can suggest is for you to move to a PS so you don't have the memory constraints of a shared server."
[By this point, I had just enough of Dreamhost, so I moved the site to another webhost. I guess Greg didn't notice. I love the attempt to try and upsell me, too.]
So, there we have it. A total of 11 separate issues in less than 5 months. This does not include other times I saw the load skyrocket but failed to report it because I was in the middle of something else. Taken separately, each of these incidents are something that can happen at even the best webhost. But, as established here, Dreamhost is nowhere near the "best". In fact, I'd say that they were one of the worst companies that I have ever done business with. To have this many issues in such a short time is simply amazing. To have a webserver throw a 404 on a valid URL is simply crazy. I have never seen that happen in Apache before. It defies logic.
Looking at their webpage now, they promise 500 Gigs of disk space and 5 TB of bandwidth for only $5.95 a month. Yet the reality here is that you will never ever come close to these limits -- chances are you will hit their memory limit (whatever THAT is, they never told me nor did they disclose it on their site) long before then. Does this sort of business practice qualify as fraud? I don't know. But is certainly is misleading to advertise numbers like that.
Dreamhost is pissing off customers left and right and cannot expect to keep up this pattern forever. This is evident just by reading the comments to posts on Dreamhoststatus.com. Heck, check out the comments on the entry about their billing system SNAFU if you want to see some anger. There are some seriously pissed off people on there. Sooner or later this bad service is going to come back to haunt them, and then things are really going to get interesting. Who knows what will happen? Massive layoffs? A scandal? Bankruptcy? Government investigation? No matter what, it will be interesting to watch!
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
(no subject)
Date: 2008-01-25 04:49 am (UTC)32 servers, total cost? $130k.. but you'll get a freaking LOT of servers run on them. and not whimpy xeons either. I need to send that excel spreadsheet I did home.
With big web servers you need high speed data links... I think I'm liking supermicro. I can't wait to play with the IPMI stuff.
(no subject)
Date: 2008-01-25 05:10 pm (UTC)(no subject)
Date: 2008-01-25 08:28 pm (UTC)(no subject)
Date: 2008-01-26 03:43 am (UTC)(no subject)
Date: 2008-01-27 10:14 pm (UTC)If you're having issues with the service (or lack thereof) that they provide, you could probably get out of the contract for failure to provide service.
(no subject)
Date: 2008-01-28 12:22 am (UTC)(no subject)
Date: 2008-01-28 01:51 am (UTC)I wanted to give them the benefit of the doubt.