ATTENTION: Data Loss

Okay, unfortunately due to some bad process we lost about 5 days of data on our website. Here's what happened...

1.) On 12/16, a snapshot of our site/database was taken so it could be moved to another server by our host.
2.) Since then, we have been logging into & posting on the old copy of the site on the original server.
3.) The old copy of the site was retired on 12/24 around midnight PST, rendering the site inaccessible as the new location hadn't propogated to DNS resolvers yet.
4.) Now that the new location is becoming available, we're able to get to our site but it appears as it was on 12/16 when the snapshot was taken.


I have good news and bad news. The good news is that there is a backup from around Friday 12/19 that we 'might' be able to load in. We'll check on this during the day Friday if we can, and if successful this gets some of our recent posts and data back. The bad news is we will still be missing a fair bit and that stuff is totally gone.

Over the next few days I will try to see whose accounts and applications no longer exist, and ask that they're re-created. It's okay to keep any recreated applications brief since I know this is an inconvenience.



Also for any wondering, this incident may lead to us changing hosts. I have some resources I will contact in the morning. Here was my initial message to our current host once I had enough pieces of the puzzle to figure out what took place...

-----------------------

So wait. Are you telling me that you're taking snapshots of the server/site/db on the day of migration, but allowing users access to the legacy site following the snapshot while not capturing any data changes in the overlapping post-migration to legacy shutdown period??

I'm not a network engineer, but as an IT Systems Admin this has now crossed into my field. I have very big issue with the process that just took place and the potential impact it creates, as we are now an example of it. You really should be cutting off access to the legacy site and/or creating a redirect at the point of migration, or at least generating a backup of the old site on the day the server is retired to ensure that there is no potential for data loss.

Yes, I received a notice that server migrations were happening. There was no indication however that we still might be hitting the legacy server/site nor any statement that we should refrain from using the site and/or action may be required on our part to ensure the safety of data. The resulting incident was the illusion of a seamless transition (until yesterday), yet in truth we working on the legacy site for the past week which is now gone.

The concerns with DNS are minor and we can easily have users hit the site via the address provided in the interim, but the data loss I can't let slide. That's preventable at several points and shouldn't have ended this way.

This ticket can be closed. There's nothing left to be done on your side.

-----------------------

Jemy

I am personally attempting to restore our data from Friday's backup. Please stand by.

Jemy

UPDATE:  The backups could not be loaded in a traditional manner. I am reviewing ALL of the SQL tables and directly adding in the missing rows between Tuesday 12/16 and Friday 12/19 when the two snapshots were taken.

It's going to take me a while, but you'll eventually see things catch up to mid-day Friday 12/19. At that point I'll list out what I know is missing from the past week and make a list for us to work on.

Jemy

I'm starting up the list of what is missing and unrecoverable... if I've forgotten anything you might recall, feel free to chime in.


No Need to Replace
- Merry Christmas/Happy Holidays thread by Carfanatic
- Shout with link to Heavensward details revealed during Tokyo fanfest (I really would like this though... I may start a thread later or add to the old one)


Needs to be Recreated
- Velius' application
- Ridus' user account & application
- Thav Onion Updates

Jemy

I'm starting up the list of what is missing and unrecoverable... if I've forgotten anything you might recall, feel free to chime in.


No Need to Replace
- Merry Christmas/Happy Holidays thread by Carfanatic
- Shout with link to Heavensward details revealed during Tokyo fanfest (I really would like this though... I may start a thread later or add to the old one)


Needs to be Recreated
- Velius' application
- Ridus' user account & application
- Thav Onion Updates

I think that about covers it. I believe Raven was the last one to request an onion and I was before him. I got mine and I think he did also. I will update that onion post and also add back my shouts.

Tomas

Thanks a ton Tomas. I should be online a bit this weekend so I'll reach out to Velius and Ridul about getting their applications remade.

No news on if/when we're changing hosts yet, but I'm also looking more into our back-end features and found at least one better method of backing up our data. Also, we may take this opportunity to push them about upgrading to SMF 2.0 again.

Jemy

Well I for one think the site could use a fresh coat of paint so to speak lol. An upgrade to the forums would be nice.

Tomas

We were told by our hosts a while back that they would be upgrading everyone on SMF 1.X versions to 2.X which should have given us some nice new features and a more up-to-date look.  Unfortunately, years have passed since then.  Another reason we are rather disappointed. 

While we are definitely looking at alternatives, here's a list of a few Pros and Cons with staying with the the same company. 
Pros (Stay)

• We have been with them forever - loyalty.
• Troubleshooting is always the same very responsive guy
• Small company - we get more customized support
• Monthly price is right
• Allow us to use a website name we own, not purchased through them.
• We will likely loose a lot of (or all of) our data from these message boards, including our apps.
• Many of the guild making websites don't allow the level of customization we have here.  We have have imported a lot of custom things to this site - allowing us to turn our single game site into a gaming community.
• Cost and time involved in setting up a new website may be prohibitive
• Website has always been secure.  Due to our setup, we haven't had any known 'hacks.'

Cons (Leave)

• The troubleshooting guy still has good response time, but the issues have gotten more complex and we are finding the responses aren't what we expect.
• Small company - when they have issues it can take longer to find a fix
• Broken backup feature in the website's admin console.  They seem either unaware of this or didn't communicate it to us.
• Twice they have caused outages to the sire due to changes in the IP.  This last time their processes caused us to lose data.
• Still haven't upgraded to SMF 2.x despite previous conversations of an upgrade.
• SMF 1.x new features are minimal and many mods are note being updated to keep up with the most recent versions.
• Potential new website features - maybe mobile options?
• Website could use a fresh coat of paint; new look/feel

I hope to use this list to create some requirements if/when we do switch.
Let me know if there is anything you'd like added. 

Grlphantom (Fantome)

I was just curious what apps might we lose and about how much is the cost?

Tomas

I was a little vague with that - I meant applications to the guild; all out info on the members.  Depending on the system, we would potentially lose all of our previous data.

Cost varies widely - some sites are free, but those sites are VERY basic and have ads.  We are actually provided with some impressive customization options (within the confines on SMF 1.x).  In looking at other sites we'd likely be paying $30+ a month for similar access to customization.  We pay a fraction of that now, but we also pay separately for our website name and Vent.  Some websites now offer bundles where you can get a Vent or Mumble Server included.  That being said, if we have a good base website we might not need the customization level we currently enjoy. 

If anyone has any great guild websites they know of, let me know and I'll look into who they are hosted with.

Grlphantom (Fantome)

Oh ok, I thought you meant like apps for the site. I will do some more digging, I've only skimmed several sites and what they offer. I'll post here if I find something I think is worth looking into.

Tomas

Just to add another nugget to this topic - I submitted a ticket to our host a few minutes ago about upgrading to SMF 2.0.x. Honestly as far as I can tell I might be able to attempt the upgrade on my own, and even hinted to them that I can/might.

As Fant said though, we actually get a good price now ($87/year for the hosting and $20/year for the domain) and have good customization options on top of being ad-free - it just requires some effort on our part when it comes to themes/etc to create the appearance we want. Despite my frustration over the data loss, I was happy at the fact that I could take the SQL data from Fant's backup, get to our tables on the back end, and recreate the missing rows. The more I look at the tools in the cPanel, the more it seems we have nearly full control of our domain (minus hardware/networking of course).

I think one thing we should consider is exactly what types of things we'd like to improve with our site? If it's just the themes & layouts then that may not warrant changing hosts, but if it's more about customized windows (ex: the iFrame code I still can't get to work for the Ventrilo panel) and mods and such, then we might need to start with figuring out what platform fits our vision (SMF 2.0, etc) and then find out who provides it.

Jemy

For some reason I didn't get an email notice, but Justin (Ejeet Founder/CEO - usually the one who responds to me) replied like 20 mins after I put in that ticket on 1/5 where I asked if we can do a SMF 2.x upgrade. Here's the full response:


------------------------------------------------------------

Hello,

Your web hosting environment can host 2.0 no problem at all you will
retain all post/forums/boards all core smf data and while data from
thrid party addons will remain in the database they may not always be
compatible with 2.0 or may use a different method of displaying data,
for example you may have to re-create your tinyportal content like
blocks and pages, thats if you even use tinyportal many now use portamx
or simpleportal but there are for certain tinyportal users still its
just the main developer stopped creating it and caused alot of forks and
so on.

In any case you can upgrade this at anytime on your own its your web
space to do what you deem fit within legal bounds.

We can always do the upgrade for you just let us know and we will
schedule this event for you.

------------------------------------------------------------

Jemy

That sounds good. Ball is in our court. Only concern I have is the site is so slow for me, I would hate for something to make it slower. Hard to predict I know before hand.

Tomas

Yes, tough since we don't know if it is the path to the server, the server itself, or the data/tables causing the bottleneck.

While we have no control over the first two factors, I do know that we have some plugins and features that we don't use. For example, we have a whole dkp system (it's some variation of ravenlegion.net/dkploot ) that we have never used. One thing I'm thinking is that regardless of if we move to 2.0, we can run thru everything and clean out/remove things that aren't being used. Also it's possibly that TinyPortal is inefficient and as Justin hinted we can shift to portamx or simpleportal or whatever we want.

I normally have good load times, so I'm hopeful that the hardware that's hosting our site is okay and we can have a cleaner/faster site with the 2.0 an upgrade and some review to how we're presenting data. Plus he's given me the specs before and they are strong servers, so even though I'm sure it's hosting numerous sites it shouldn't have a problem handling it. Regardless, I can poke around and get some diagnostic info to make certain. I'll post what I can find.

Jemy