System Failure

Scott Hardie | September 13, 2007
Remember the nice things I said about my webhost a couple of weeks ago? Forget 'em.

Last night was a productive time on the site. All five sections of the site got new content, and several players scored big in RB. When I went to bed at 2am on September 13, the site was fine.

I woke up this morning to certain PHP errors that I recognized right away. Like a previous webhost had done, my new one had upgraded my version of PHP overnight without warning me, causing certain functions to break all over the site. It's nice of them to keep their software semi-current, but how about some warning next time? At first, I thought it was no big deal, since I was able to make the needed adjustments to my code quickly and get the site to appear again.

That's when I realized it was worse. No new goo. TC was a day behind. None of September 12's concerts ever happened. The site had lost an entire day's worth of activity.

The system runs a complete backup every night at 2am est, which is midnight at the webhost's facility. When they upgraded to PHP5, they must have reinstalled the databases from the 2am 9/12 archive, destroying all of the data acquired since then. This may have been the result of an accident or it may have been planned, but either way, I haven't heard jack squat from them to explain it. Some advance warning that they were going to modify the system would have given me a chance to run my own backups.

Guess who's going to change webhosts again?

I have to go to work now, so I'll do what I can tonight to restore some of it. Amy will get a special declaration that she unlocked the win-five-concerts-in-one-day achievement. Mike will get his free R8 card back. I'll put up two goos tonight. In the meantime, we'll have to live with the missing day because we don't have a choice; every backup I have is from 2am September 12.

I will not apologize for the inconvenience because I did not cause it, but you better believe I will take steps to prevent this from ever happening again at any webhost.

Denise Sawicki | September 13, 2007
Wow that sucks... Hey I didn't deserve my victory against Darrell yesterday anyhow I guess. Maybe that makes me feel better, I don't know... Oh I guess I have to go back and try to do damage control on the various stupid things I said before the crash, once again.. Anyway... best of luck in finding a web host that won't pull this kind of stunt on you!

EDIT: yes I agree it is not your fault and you did a good job of getting it back to normal fast! Sorry to forget to say the important things that I do in fact mean and go babbling on about stupid stuff...

Eric Wallhagen | September 13, 2007
Dude, that sucks. Definitely not gonna hold it against you. I mean it's awsome that you back it up on a daily basis so only one day was lost. *phew* Keep up the good work on it, and good luck finding a webhost that won't screw you over again.

Lori Lancaster | September 13, 2007
[hidden by request]

Justin Woods | September 13, 2007
Sorry to hear that Scott, good job on backups though!!! I am probably one of the biggest people it affected as far as yesterday but I am thinking in a good and bad way.... I was hoping that I can at least keep my win for each trade rule for my achievements, if not I'll break down and do it again, no big deal... Thanks for haven't the site so quickly too...


(Edit) never mind that Scott I won my Each concert today!!!

Mike Rothstein | September 13, 2007
Sorry for your frustration, good luck finding a better host.

Aaron Shurtleff | September 14, 2007
I wonder if anyone got their card before 2 am then lost it.

It was probably for me! :P

Scott Hardie | September 14, 2007
I did. It was an R10.

Thanks for the kind words, folks.

Aaron Shurtleff | September 14, 2007
Damn! That sucks! :(

It's actually pretty cool that you got everything taken care of so quickly!! Good luck with the hunt!!

Scott Hardie | September 14, 2007
Thanks. At least I'll be able to put some other issues with this host to rest, like the sending-email-from-the-server problems.

Amy Austin | September 14, 2007
This blows.

Lori: KMA.

Scott Hardie | September 25, 2007
Sorry to have to take the site down. I don't know what happened to the database, but starting around 5:45pm it started failing every third process or so. After I cleaned out the hundreds of error notices that had accumulated, I tried some adjustments and it didn't get any better, so I figured I'd wait for the host to do something (after I submitted a ticket). It seems to be resolved now.

Scott Hardie | September 25, 2007
...Perhaps not. It's still happening. A lot. And I don't have time to deal with this. :-(

I don't want to shut the site down again for the night, so I'm leaving it running with errors. I have a help ticket open; hopefully they can reboot the db server or something. Those of you who stay on for a while, good luck with it.

Erik Bates | September 25, 2007
[hidden by request]

Scott Hardie | September 25, 2007
Nope, ditched them because of constant outages, along with lesser problems. I still have some time left on my paid account there, so I'm considering going back temporarily if the new webhost can't resolve this database problem.

Lori Lancaster | September 25, 2007
[hidden by request]

Scott Hardie | September 26, 2007
After some complete outages this afternoon (606 error emails in three hours), it seems to be fine now. I'm glad it's back to normal, and I'll thank them, but their tech support didn't earn my favor by telling me a clear lie and then telling me to "talk to [my] software developer" about how to fix it.

On the other hand, it's only supposed to show *me* those error messages that you quoted, Lori. Other users are merely supposed to get the "unable to load, Scott alerted" part. That is totally something wrong with my code but I don't know what.

Lori Lancaster | September 26, 2007
[hidden by request]

Scott Hardie | September 26, 2007
Yeah, I realized what was happening. The code that determined whether the user was me came after most of the db errors, so it didn't know better. I think it's fixed, but please let me know if you spot another exposed query.

Good news on the technical front with all of these issues, but I can't say more until this weekend.