We’re Back! Plus Postmortem

We are back finally! Glad to have the site back up and running.

So what happened? Digital Ocean was doing maintenance on the host of this virtual server this site was hosted on. Something happened and severely corrupt the filesystem. A couple days were spent attempting to recover data. While this site was backed up, other things were not.

Why did it take so long to get back? The original server hosted the web code, the database, and the mail for the company and everything was hand set up. It was planned to improve this infrastructure, but of course something happened before that could be accomplished. The decision was made to focus on doing the new infrastructure instead of losing time resetting the original config.

In hindsight, knowing that it would take this long,  a temporary site should have been set up to at least handle interested parties. Lesson learned!

In any case, the site is now running on four separate virtual serves (web, database, caching, and mail). All the set up is scripted out, and in the case of one server going down, its a quick 15 minutes to bring a new one back up. The site will also be faster due to the use of Redis as a caching server.

Note that the mail server portion is still being configured, and we’ll post an update when that’s complete. For now, please contact us via our Twitter page.